Appearance
H3C RoCE 交换机配置
H3C RoCE 交换机管理、VLAN、端口拆分、PFC／ECN 及 MXRDMA 配置实例。
常用的基本操作
分配网络管理口和账户
进入交换机命令行
使用命令
sys
进入特权模式
使用命令
sysname MUXI
更改设备名为：MUXI
使用命令
bash
ssh server enable开启ssh登录
使用命令
bash
ssh user创建 SSH 用户，并指定 SSH 用户的服务类型和认证方式
undo ssh user 命令用来删除 SSH 用户
配置密码
配置管理口IP地址
退出保存配置
exit
write
在别的网段的机器上面ssh登录，验证配置是否成功
查看信息
进入特权模式
sys
端口连接状态
display interface brief
查看当前配置
display current-configuration
端口拆分和恢复
400G 拆分成 200G
interface 400GE1/0/1 //单个拆分
interface range 400GE1/0/1 to 400GE1/0/35 //范围拆分(只支持拆分，不支持范围恢复)
using twohundredgige
400G 拆分成 200G后恢复400G
using fourhundredgige //恢复只能一个一个操作，或者自己写脚本批量
拆分后如果网口没有UP，尝试下面命令使能端口
undo shutdown
丢包查看
先确认要查看的网卡对应网口，服务器侧通过"lldpcli show neigh"命令来查看网卡连接的交换机端口
查看网络是否丢包和ECN触发次数1
display packet-drop interface HGE1/1/16
重置丢包信息：
reset packet-drop interface HGE1/1/16
reset counters interface HGE1/1/16
查看网络PFC触发次数
display priority-flow-control interface
vlan配置
新增vlan
上图ip address后面填写的是网关地址，也就是yaml文件中的via字段后面跟的ip地址
下图是某次配置的截图：
ip address执行完毕后，需要执行undo shutdown命令
display this的结果如图：
添加端口到vlan
端口使用lldpcli show neigh命令查看，下图框起来的就是port后需要跟的端口号
根据该图，执行如下命令即可将此两个端口加到vlan中，如下图
添加完毕后，使用下图命令查看vlan 2的内容
最后sudo netplan apply或者在最开始执行这个命令也行
show_gids不连续
执行完4.2.2节后，show_gids发现gid不连续？是因为在sudo netplan apply的时候，某个网卡正在被使用，某个GID正在被占用，所以你在执行sudo netplan apply的时候，在原先GID的基础上递增。
如何解决？只需要关闭掉正在使用这些网卡的应用，再次执行suod netplan apply即可
100Gbps交换机的推荐配置
假设使用优先级 5 来承载分布式存储的 RoCEv2 流量。在交换机里，各个面板端 口需要针对优先级 5 使能 PFC 流控，命令行如下。
bash
sys //进入特权模式
#全局配置
buffer egress cell queue 5 shared ratio 100 //共享缓冲区 出口队列五可以使用率100%
buffer egress cell queue 6 shared ratio 100
buffer apply
qos wred queue table QOS-EGRESS-100G-PORT
queue 5 low-limit 1000 high-limit 2000 discard-probability 20
queue 5 weighting-constant 0
queue 5 ecn
priority-flow-control poolID 0 headroom 130000 //配置为允许的最大值
priority-flow-control deadlock cos 5 interval 10
priority-flow-control deadlock precision high
#配置端口HundredGigE1/0/1到interface HundredGigE1/0/35
interface range 100GE1/0/1 to 100GE1/0/35
qos wred apply QOS-EGRESS-100G-PORT
qos trust dscp
priority-flow-control enable
priority-flow-control no-drop dot1p 5
priority-flow-control dot1p 5 headroom 491
priority-flow-control dot1p 5 ingress-buffer dynamic 5
priority-flow-control dot1p 5 ingress-threshold-offset 12
priority-flow-control deadlock enable
#保存配置
exit
save400Gbps交换机的推荐配置
假设使用优先级 5 来承载分布式存储的 RoCEv2 流量。在交换机里，各个面板端口需要针对优先级 5 使能 PFC 流控，命令行如下。
bash
sys //进入特权模式
buffer egress cell queue 5 shared ratio 100 //共享缓冲区 出口队列五可以使用率100%
buffer egress cell queue 6 shared ratio 100
buffer apply
qos wred queue table QOS-EGRESS-400G-PORT
queue 5 drop-level 0 low-limit 2100 high-limit 5000 discard-probability 20
queue 5 drop-level 1 low-limit 2100 high-limit 5000 discard-probability 20
queue 5 drop-level 2 low-limit 2100 high-limit 5000 discard-probability 20
queue 5 weighting-constant 0
queue 5 ecn
priority-flow-control poolid 0 headroom 64000
priority-flow-control deadlock cos 5 interval 10
priority-flow-control deadlock precision high
#配置端口FourHundredGigE1/0/1到interface FourHundredGigE1/0/35,通过dis interface b 查看可用端口
interface range 400GE1/0/1 to 400GE1/0/35
qos trust dscp
qos wred apply QOS-EGRESS-400G-PORT
priority-flow-control enable
priority-flow-control no-drop dot1p 5 pause-threshold ratio 5 pause-threshold-offset 12 reserved-buffer 16
priority-flow-control deadlock enable
exit
saveMXRDMA网卡交换机配置实例
N300和C600机器连接交换机的方式不同
N300是MXRDMA网卡连接200G端口的交换机H3C-S9825，128口，
C600是将两个MXRDMA网卡通过转接器合并到400G的线接入400G双端口（每个端口200G）的交换机H3C-S9857。
N300 MXRDMA 交换机配置
目前针对型号H3C-S9825-128B。
交换机上创建vlan 2：
根据服务器上IB网卡个数为vlan 2分配IP，如服务器上有16个网卡，则分配16个IP地址。
将网卡连接的16个交换机端口加入到vlan 2
# 进入vlan 2配置
interface vlan 2
# 进入特权模式
system-view
# 创建vlan 2
vlan 2
#端口接入vlan 2
port 200GE1/0/1
port 200GE1/0/2
port 200GE1/0/3
port 200GE1/0/4
port 200GE1/0/5
port 200GE1/0/6
port 200GE1/0/7
port 200GE1/0/8
port 200GE1/0/9
port 200GE1/0/10
port 200GE1/0/11
port 200GE1/0/12
port 200GE1/0/13
port 200GE1/0/14
port 200GE1/0/15
port 200GE1/0/16
#配置vlan的网关
interface vlan 2
undo ipv6 forwarding
ip address 11.22.100.1 255.255.255.0
ip address 11.22.101.1 255.255.255.0 sub
ip address 11.22.102.1 255.255.255.0 sub
ip address 11.22.103.1 255.255.255.0 sub
ip address 11.22.104.1 255.255.255.0 sub
ip address 11.22.105.1 255.255.255.0 sub
ip address 11.22.106.1 255.255.255.0 sub
ip address 11.22.107.1 255.255.255.0 sub
ip address 11.22.108.1 255.255.255.0 sub
ip address 11.22.109.1 255.255.255.0 sub
ip address 11.22.100.1 255.255.255.0 sub
ip address 11.22.111.1 255.255.255.0 sub
ip address 11.22.112.1 255.255.255.0 sub
ip address 11.22.113.1 255.255.255.0 sub
ip address 11.22.114.1 255.255.255.0 sub
ip address 11.22.115.1 255.255.255.0 sub
# 退出vlan配置
exit
# 保存vlan配置
write交换机上连接服务器网卡的端口配置，所有的端口配置QoS。
# 进入特权模式
system-view
# 配置端口1-16，注意根据实际情况配置实际的端口号
interface range TwoHundredGigE 1/0/1 to TwoHundredGigE 1/0/16
# 进行配置
port link-mode bridge
port access vlan 2
link-delay down 0
link-delay up 2
# 为队列5配置PFC
priority-flow-control enable
priority-flow-control no-drop dot1p 5 pause-threshold ratio 12 headroom 2000
priority-flow-control deadlock enable
flow-interval 5
# 配置ecn dscp
qos trust dscp
qos wfq byte-count
qos wfq ef group 1 byte-count 96
qos wfq cs6 group sp
qos wfq cs7 group sp
qos wred queue 5 drop-level 0 low-limit 3500 high-limit 5000 discard-probability 40
qos wred queue 5 drop-level 1 low-limit 3500 high-limit 5000 discard-probability 40
qos wred queue 5 drop-level 2 low-limit 3500 high-limit 5000 discard-probability 40
qos wred queue 5 ecn
qos wred queue 5 weighting-constant 0C600 MXRDMA 交换机配置
交换机上创建vlan 2：
根据服务器上IB网卡个数为vlan 2分配IP，如服务器上有16个网卡，则分配16个IP地址。
将网卡连接的16个交换机端口加入到vlan 2
H3C-S9827。
port link-mode bridge
port access vlan 2
priority-flow-control enable
priority-flow-control no-drop dot1p 5 pause-threshold ratio 5
priority-flow-control deadlock enable
qos trust dscp
qos wfq byte-count
qos wfq ef group 1 byte-count 96
qos wfq cs6 group sp
qos wfq cs7 group sp
qos wred queue 5 drop-level 0 low-1imit 6000 high-1imit 12000 discard-probability 40
qos wred queue 5 drop-level 1 low-limit 6000 high-limit 12000 discard-probability 40
qos wred queue 5 drop-level 2 low-limit 6000 high-limit 12000 discard-probability 40
qos wred queue5 ecn
qos wred queue 5 weighting-constant 0H3C-S9827-128DH
port link-mode bridge
port access vlan 2
priority-flow-control enable
priority-flow-control no-drop dot1p 5 pause-threshold ratio 6 headroom 1200
priority-flow-control deadlock enable
qos trust dscp
qos wfq byte-count
qos wfq ef group 1 byte-count 96
qos wfq cs6 group sp
qos wfq cs7 group sp
qos wred queue 5 drop-level 0 low-1imit 3500 high-1imit 8000 discard-probability 40
qos wred queue 5 drop-level 1 low-limit 3500 high-limit 8000 discard-probability 40
qos wred queue 5 drop-level 2 low-limit 3500 high-limit 8000 discard-probability 40
qos wred queue5 ecn
qos wred queue 5 weighting-constant 0
星融元CX864-E
ssh登录交换机
普通视图只能查看信息，配置需要config视图（config视图查看信息需要do show xxx）
vlan配置
!
vlan range 100-300
!
interface vlan 100
ip address 11.22.100.1/24
ip address 11.22.101.1/24
ip address 11.22.102.1/24
ip address 11.22.103.1/24
ip address 11.22.104.1/24
ip address 11.22.105.1/24
ip address 11.22.106.1/24
ip address 11.22.107.1/24
ip address 11.22.108.1/24
ip address 11.22.109.1/24
ip address 11.22.110.1/24
ip address 11.22.111.1/24
ip address 11.22.112.1/24
ip address 11.22.113.1/24
ip address 11.22.114.1/24
ip address 11.22.115.1/24
exit
!接口配置
!
interface ethernet 0/0
service-policy roce_lossless_40m_low_balance_200g
breakout 4x200G[100G]
fec rs
mtu 9216
speed 200000
switchport access vlan 100
exit
!接口绑定RoCE
!
qos roce roce_lossless_40m_low_balance
bind interface range 0/1:1-0/64:4
exit
!RoCE配置（星融元内置模板，请谨慎改动）
!
buffer-profile egress_lossless_profile
mode lossless static 22798336 size 0
exit
!
buffer-profile egress_lossy_profile
mode lossy dynamic 2 size 1518
exit
!
buffer-profile ingress_lossless_profile
mode lossless dynamic 1 size 1518 xon-offset 9408 xoff 135520
exit
!
buffer-profile ingress_lossy_profile
mode lossy static 22798336 size 0
exit
!
buffer-profile pg_lossless_10000_40m_profile
mode lossless dynamic 1 size 1518 xon-offset 9408 xon 0 xoff 16928
exit
!
buffer-profile pg_lossless_25000_40m_profile
mode lossless dynamic 1 size 1518 xon-offset 9408 xon 0 xoff 16928
exit
!
buffer-profile pg_lossless_50000_40m_profile
mode lossless dynamic 1 size 1518 xon-offset 9408 xon 0 xoff 23392
exit
!
buffer-profile pg_lossless_100000_40m_profile
mode lossless dynamic 1 size 1518 xon-offset 9408 xon 0 xoff 38816
exit
!
buffer-profile pg_lossless_200000_40m_profile
mode lossless dynamic 1 size 1518 xon-offset 9408 xon 0 xoff 71904
exit
!
buffer-profile pg_lossless_400000_40m_profile
mode lossless dynamic 1 size 1518 xon-offset 9408 xon 0 xoff 135520
exit
!
buffer-profile pg_lossless_800000_40m_profile
mode lossless dynamic 1 size 1518 xon-offset 9408 xon 0 xoff 205520
exit
!
buffer-profile roce_lossless_40m_low_balance_200g_profile
mode lossless dynamic 1 size 1518 xon-offset 9408 xoff 71904
exit
!
wred default_drop
mode drop gmin 1048576 gmax 2097152 gprobability 10 ymin 1048576 ymax 2097152 yprobability 10 rmin 1048576 rmax 2097152 rprobability 10
exit
!
wred default_ecn
mode ecn gmin 1048576 gmax 2097152 gprobability 10 ymin 1048576 ymax 2097152 yprobability 10 rmin 1048576 rmax 2097152 rprobability 10
exit
!
wred roce_lossless_40m_low_balance_200g_ecn
mode ecn gmin 1768622 gmax 8926720 gprobability 50
exit
!
priority-flow-control enable 1 5
!
class-map roce_lossless_40m_low_balance_200g_class_map
match cos 1 5
exit
!
policy-map roce_lossless_40m_low_balance_200g
class roce_lossless_40m_low_balance_200g_class_map
priority-group-buffer roce_lossless_40m_low_balance_200g_profile
wred roce_lossless_40m_low_balance_200g_ecn
exit
queue-scheduler priority queue 6
queue-scheduler priority queue 7
set cos dscp diffserv roce_lossless_diffserv_map
exit
!