Bandwidth pinched through VxLAN tunnel

lctn · Thu Apr 14, 2022 10:13 pm

ROS: 7.1.3
Devices: CCR1036-8G-2S

I had a production VxLAN tunnel running with 2 CCR1036-8G-2S Cloud Core Routers. The tunnel was for a k12 school district and I would often get complaints about web traffic being painfully slow. If left alone the issue would resolve itself but repeat several times throughout the day. I finally had to pull the Mikrotiks and put the district back on our Aruba VxLAN tunnel, which resolved the issue for them. My 1st choice is to use the Mikrotiks but I need to resolve the issue before implementing again.

In duplicating the setup, with 2 cloud routers directly connected, I discovered the following:
1. With a VxLAN configuration, a laptop pulled a 3Gb ISO down at 19MBs
2. With the same configuration a laptop and PC pulled the same file down at 11MBs, each
3. Removing VXLAN and configuring the devices with only L2 VLANs, the laptop pulled the file down at 34MBs.
4. The laptop and PC pulled the file down at the same time at 34MBs, each.

Obviously, there is a major degrade in performance when using VxLAN, especially when more than 1 client is downloading a file.

I have contacted support concerning this but have not received instruction on how to remedy the problem yet, outside of a recommendation to change the mss to 1300, but that rule did not have any hits, regardless of the interface used. Ideas??

Attaching all configs:

VxLAN:
Master:

# apr/13/2022 12:29:37 by RouterOS 7.1.3 # software id = IHD4-H4MR # # model = CCR1036-8G-2S+ /interface bridge add ingress-filtering=no name=BRIDGE-VxLAN-VNI-102 vlan-filtering=yes /interface ethernet set [ find default-name=ether1 ] l2mtu=8000 set [ find default-name=ether2 ] l2mtu=8000 set [ find default-name=ether4 ] l2mtu=8000 /interface vxlan add group=224.0.0.1 interface=ether1 mtu=1400 name=vxlan-vni-102 port=8473 \ vni=102 /interface vlan add interface=ether1 name=vlan703 vlan-id=703 /interface lte apn set [ find default=yes ] ip-type=ipv4 /interface wireless security-profiles set [ find default=yes ] supplicant-identity=MikroTik /ip pool add name=dhcp_pool0 ranges=172.169.0.10-172.169.0.30 /ip dhcp-server add address-pool=dhcp_pool0 name=dhcp1 /port set 0 name=serial0 set 1 name=serial1 /interface bridge port add bridge=BRIDGE-VxLAN-VNI-102 interface=ether2 add bridge=BRIDGE-VxLAN-VNI-102 interface=vxlan-vni-102 /ip settings set max-neighbor-entries=8192 /ipv6 settings set disable-ipv6=yes max-neighbor-entries=8192 /interface bridge vlan add bridge=BRIDGE-VxLAN-VNI-102 tagged=vxlan-vni-102,ether2 vlan-ids=703 add bridge=BRIDGE-VxLAN-VNI-102 tagged=vxlan-vni-102,ether2 vlan-ids=704 /interface vxlan vteps add interface=vxlan-vni-102 port=8572 remote-ip=172.169.0.2 /ip address add address=172.169.0.1/24 interface=ether1 network=172.169.0.0 /ip dhcp-client add interface=ether8 /ip dhcp-server network add address=172.169.0.0/24 gateway=172.169.0.1 /ip firewall mangle add action=change-mss chain=forward log=yes new-mss=1300 out-interface=\ BRIDGE-VxLAN-VNI-102 passthrough=yes protocol=tcp tcp-flags=syn tcp-mss=\ 1301-65535 /system clock set time-zone-name=America/Chicago /system identity set name=VxLAN-Master

Client:

# apr/14/2022 10:20:18 by RouterOS 7.1.3 # software id = MPIL-B0WN # # model = CCR1036-8G-2S+ /interface bridge add ingress-filtering=no name=BRIDGE-VxLAN-VNI-102 vlan-filtering=yes /interface ethernet set [ find default-name=ether1 ] l2mtu=8000 set [ find default-name=ether2 ] l2mtu=8000 set [ find default-name=ether4 ] l2mtu=8000 /interface vxlan add group=224.0.0.1 interface=ether1 mtu=1400 name=vxlan-vni-102 port=8473 \ vni=102 /interface wireless security-profiles set [ find default=yes ] supplicant-identity=MikroTik /ip pool add name=dhcp_pool0 ranges=172.168.0.10-172.168.0.30 /ip dhcp-server add address-pool=dhcp_pool0 name=dhcp1 /port set 0 name=serial0 set 1 name=serial1 /interface bridge port add bridge=BRIDGE-VxLAN-VNI-102 interface=vxlan-vni-102 add bridge=BRIDGE-VxLAN-VNI-102 interface=ether2 add bridge=BRIDGE-VxLAN-VNI-102 interface=ether4 pvid=703 /interface bridge vlan add bridge=BRIDGE-VxLAN-VNI-102 tagged=vxlan-vni-102,ether2 vlan-ids=703 /ip address add address=172.169.0.2/24 interface=ether1 network=172.169.0.0 /ip dhcp-client add interface=ether8 /ip firewall mangle add action=change-mss chain=forward new-mss=1300 out-interface=\ BRIDGE-VxLAN-VNI-102 protocol=tcp tcp-flags=syn tcp-mss=1301-65535 /ip route add disabled=no dst-address=0.0.0.0/0 gateway=172.168.0.1 routing-table=main \ suppress-hw-offload=no add disabled=no dst-address=0.0.0.0/0 gateway=172.169.0.1 routing-table=main \ suppress-hw-offload=no /system clock set time-zone-name=America/Chicago /system identity set name=LCTN-Rm424 /tool sniffer set filter-interface=ether1

L2 VLAN Only:
Master:

# jan/02/1970 00:54:26 by RouterOS 7.1.3 # software id = IHD4-H4MR # # model = CCR1036-8G-2S+ /interface bridge add ingress-filtering=no name=B703 vlan-filtering=yes /interface wireless security-profiles set [ find default=yes ] supplicant-identity=MikroTik /port set 0 name=serial0 set 1 name=serial1 /interface bridge port add bridge=B703 interface=ether1 add bridge=B703 interface=ether2 add bridge=B703 interface=ether3 pvid=703 add bridge=B703 interface=ether4 pvid=703 /interface bridge vlan add bridge=B703 tagged=ether1,ether2 vlan-ids=703 add bridge=B703 tagged=ether1,ether2 vlan-ids=704 /ip address add address=172.169.0.1/24 interface=ether1 network=172.169.0.0 add address=10.7.3.99/24 interface=ether2 network=10.7.3.0

Client:

# 4月/ 14/2022 13:40:15 Roul雷竞技terOS 7.1.3 #软件e id = MPIL-B0WN # # model = CCR1036-8G-2S+ /interface bridge add ingress-filtering=no name=B703 vlan-filtering=yes /interface wireless security-profiles set [ find default=yes ] supplicant-identity=MikroTik /port set 0 name=serial0 set 1 name=serial1 /interface bridge port add bridge=B703 interface=ether1 add bridge=B703 interface=ether3 pvid=703 add bridge=B703 interface=ether4 pvid=703 /interface bridge vlan add bridge=B703 tagged=ether1 untagged=ether3 vlan-ids=703 /ip address add address=172.169.0.2/24 interface=ether1 network=172.169.0.0 add address=10.7.3.100/24 interface=B703 network=10.7.3.0 /ip dhcp-client add interface=ether8 /system clock set time-zone-name=America/Chicago

smyers119 · Fri Apr 15, 2022 7:59 am

Until you hear back about the speed problem are you interested in alternative solutions to your problem? There is plenty of other ways to extend your layer 2 network that may not take as much of a speed deficit as vxlan, assuming there's nothing you can do about that.

Fri Apr 15, 2022 12:46 pm

The tunnel was for a k12 school district

Why then are you using an interior slice ofOath/Yahoo/AOL’s IP space？除非你的K12的学校,毫无原因地拥有by the parent company of Yahoo, this choice is likely to break access to Yahoo properties.

4. The laptop and PC pulled the file down at the same time at 34MBs, each.

Please use Mbit/sec. I’ll never understand why browsers use bytes when the entire rest of the networking world uses bits.

What’s the size of your upstream pipe? You show at least 544 Mbit/sec, but how large is that compared to the claimed maximum?

a recommendation to change the mss to 1300, but that rule did not have any hits

Since that mangle rule you have…

/ip firewall mangle add action=change-mss chain=forward log=yes new-mss=1300 out-interface=\ BRIDGE-VxLAN-VNI-102 passthrough=yes protocol=tcp tcp-flags=syn tcp-mss=\ 1301-65535

…is up at the top, there should be no way to skip it unless you've misspecified it.

My guess is that your out-interface rule is wrong. If we temporarily set aside my inference above that your LAN IP space is badly-chosen, I think you want to replace it with something like dst-address=!172.168.0.0/15, since you want this rule to affect only traffic forwarded to the Internet, not in-LAN traffic. It doesn't matter which interface it's goingout, it matters which IP it's goingto.

如果你决定你需要指定的国际米兰face name, I believe they're case-sensitive in RouterOS filters. You've got a mix of all-lowercase names and mixed-case names in this config.

Drop the log=yes bit. That's needlessly expensive here: logging is expensive in general, and you've got it on a rule that should be hit by every TCP SYN packet, of which there are often dozens per HTTP hit. For your purposes, simply watching the packet counter in WinBox should suffice to tell you when it's working.

/interface vxlan add group=224.0.0.1

The224.0.0.0/24 rangeis for well-known services, registered with IANA. The ".1" address in that range is a particularly bad choice, meaning "all multicast hosts".

You should be using something in 239.0.0.0/8 that you've pre-cleared as unused locally.

lctn · Sat Apr 16, 2022 4:36 am

Thank you for the replies..

Just a reminder: The Aruba VxLAN tunnel does not have any of the same limitations and runs across the same WAN for 18 other school districts.

My design is as follows (10 Gb WAN)

School district edge switch > Mikrotik Cloud Router (VxLAN) > Cisco 3400 WAN switch > Cisco 3600 WAN Switch > Mikrotik Cloud Router > Fortigate Vdom> Internet

@smyers119

I would be very interested in learning of other options that have better throughput.

@sweetcandysp

I left things as is so all things could be considered when responding.

@tangent

My IPs came from something similar I found on a tutorial. None of the IPs shown ever get beyond our 10Gb WAN

There are no other rules on the devices. The mss is all I have. If it should be configured differently, I would love to know which of the listed interfaces should be used. In my troubleshooting, I believe I tried every interface that was not a slave interface.

I am the WAN admin, the UDP addresses are available. I actually had started with 239.x.x.x but had 2 more testing tunnels on the original equipment and wanted to be sure I was not causing an issue between them. In my current test, I have a single tunnel on two isolated Mikrotik routers. The results are the same, regardless.

Sat Apr 16, 2022 6:56 am

The Aruba VxLAN tunnel does not have any of the same limitations

Part of the problem may be that you're trying to treat MikroTik as Aruba.

There's a lot of foundational stuff you can port from one vendor to another, but take that too far and you'll run into trouble no matter which direction you're going. At some point, you have to take each vendor's offering on its own merits.

My IPs came from something similar

Yeah, it looks like you saw something done with a 172.16.x.y through 172.31.x.y scheme and thought you could make up anything in the 172 space.You can't. Same as how you're making up multicast IPs without knowing what you're doing and running into trouble with that, too.

None of the IPs shown ever get beyond our 10Gb WAN

Not the point: if anyone on your LAN tries to go to any Oath Holdings property that happens to resolve via DNS to an IP in your bogarted range, your routers will say, "Oh hey, that's a local IP! Let me direct you!"

If you're lucky, no local machine will be at the referenced IP, and you'll just get an apparent DNS failure.

If you'reunlucky, it'll resolve to something with a server running on it and your user will start griping about how they went to frobozz.yahoo.com and got the district SAN's management web app instead.

If you'rereally unlucky, the same thing will happen, but it'll affect only one critical element on the page out of 100 that work just fine, with the result that you won't figure out why it's failing until you spend two hours with the browser developer tools, a copy of the Necronomicon, and a ouija board to summon the ghost of Postel to serve as your spririt guide into the bowels of someone else's overcomplicated enterprisey web app.

If it should be configured differently, I would love to know which of the listed interfaces should be used.

I told you: you don't want an interface name, you want a direction of some kind.

Look at your bridge configuration: you've got the L2 MTU set to 8000. Why would you do that and then clamp MSS to 1300? A wise network admin wouldn't, because that'd be silly. Just because the TCP SYN transits the LAN doesn't mean it must be clamped. Where it's going affects what MSS you need.

Instead of clamping MSS to the worst case and making everything suffer, a better config might use three different MSS values:

1. In-LAN destinations, where you leave the MTU/MSS unclamped, so your jumbo packet config (MTU=8000) can take effect.

2. Cross-district destinations where you have to take the VXLAN overhead into account. Something like 1450 might suffice here.

3. Internet destinations where you also have to take into account another bottleneck down the line, where a smaller MSS still might be needed.

It's all conditional. You have to know your network.

I am the WAN admin, the UDP addresses are available.

If you're trying to justify 224.0.0.1, it doesn't matter that it's "unused" on your LAN. It's wrong, period. It's a type of broadcast address, so you can expect trouble if you try to use it for VXWAN broadcast signalling. That's broadcast over broadcast, see?

我和239. x.x实际上已经开始。x,但2莫re testing tunnels on the original equipment and wanted to be sure I was not causing an issue between them.

In multicast space, there's nothing like a subnet.† We use CIDR notation to talk about blocks of multicast IPs, but 239.0.1.2 is no closer to 239.0.1.3 than to 239.253.254.255. You can have two services right next to each other in this space and they won't create a conflict.

—◆—◆—◆—

† Okay, okay, there's the1:32 Ethernet to IP chunking, but the stack takes care of that. If you want to be paranoid, add 32 to each IP to be extra-sure they don't conflict, even down at the L2 level.

lctn · Sat Apr 16, 2022 5:32 pm

Thanks for the feedback. I will make some changes next week and report my findings.

lctn · Tue Apr 26, 2022 7:18 pm

I followed the recommendations but really could not get VxLAN to perform any differently than it did with my posted configs. I finally moved to an EOIP solution which performs quite a bit better with the configs below. I will have 300+ users getting their Internet across 20 VLANs through this tunnel. Is there anything else I can modify to inure the best throughput possible?

Master

# apr/26/2022 10:56:02 by RouterOS 7.1.3 # software id = IHD4-H4MR # # model = CCR1036-8G-2S+ /interface bridge add name=B703 add name=B704 add ingress-filtering=no name=trunk vlan-filtering=yes /interface eoip add loop-protect=off mac-address=02:97:91:21:41:0E mtu=1500 name=eoip-tunnel1 \ remote-address=10.1.1.2 tunnel-id=7 /interface vlan add interface=trunk name=vlan703 use-service-tag=yes vlan-id=703 add interface=ether2 name=vlan703-acc vlan-id=703 add interface=trunk name=vlan704 use-service-tag=yes vlan-id=704 add interface=ether2 name=vlan704-acc vlan-id=704 /interface wireless security-profiles set [ find default=yes ] supplicant-identity=MikroTik /port set 0 name=serial0 set 1 name=serial1 /interface bridge port add bridge=trunk interface=eoip-tunnel1 add bridge=trunk interface=ether2 add bridge=B703 interface=vlan703 add bridge=B703 interface=ether3 add bridge=B703 interface=vlan703-acc add bridge=B704 interface=vlan704 add bridge=B704 interface=vlan704-acc add bridge=B704 interface=ether4 /interface bridge vlan add bridge=trunk tagged=trunk,ether2 vlan-ids=703,704 /ip address add address=10.1.1.1/27 interface=ether1 network=10.1.1.0 /ip dhcp-client add interface=ether8 /system clock set time-zone-name=America/Chicago /system identity set name=Master-EOIP

Client

# apr/26/2022 10:55:55 by RouterOS 7.1.3 # software id = MPIL-B0WN # # model = CCR1036-8G-2S+ /interface bridge add name=B703 protocol-mode=none add name=B704 protocol-mode=none add ingress-filtering=no name=trunk protocol-mode=none vlan-filtering=yes /interface eoip add loop-protect=off mac-address=FE:35:A9:BE:03:2F mtu=1500 name=eoip-tunnel1 \ remote-address=10.1.1.1 tunnel-id=7 /interface vlan add interface=trunk name=vlan703 use-service-tag=yes vlan-id=703 add interface=ether2 name=vlan703-acc vlan-id=703 add interface=trunk name=vlan704 use-service-tag=yes vlan-id=704 add interface=ether2 name=vlan704-acc vlan-id=704 /interface wireless security-profiles set [ find default=yes ] supplicant-identity=MikroTik /port set 0 name=serial0 set 1 name=serial1 /snmp community add addresses=::/0 name=lctnmrtg /interface bridge port add bridge=trunk interface=eoip-tunnel1 add bridge=trunk interface=ether2 add bridge=B703 interface=vlan703 add bridge=B703 interface=ether3 add bridge=B703 interface=vlan703-acc add bridge=B704 interface=vlan704 add bridge=B704 interface=vlan704-acc add bridge=B704 interface=ether4 /interface bridge vlan add bridge=trunk tagged=trunk,ether2 vlan-ids=703,704 /ip address add address=10.1.1.2/27 interface=ether1 network=10.1.1.0 /system identity set name=Client-EOIP

rextended · Tue Apr 26, 2022 7:26 pm

Remove serial numbers from export, everytime...

lctn · Tue Apr 26, 2022 7:35 pm

Thank you!

rextended · Tue Apr 26, 2022 7:38 pm

Ehm... also from your other posts...
search.php?keywords=%22serial+number%22&author=lctn

lctn · Tue Apr 26, 2022 7:52 pm

Got it. Thanks

mada3k · Tue Apr 26, 2022 9:52 pm

Smells like MTU issues. What the the maximum allowed MTU on the line between the CCR'es?

lctn · Tue Apr 26, 2022 10:31 pm

If you are referring to the VxLAN config, we finally settled on 1400 (tried 1450 too) on the tunnel interface and 1500 for all other interfaces. In my test environment, the two Mikrotiks were directly connected to each other, via ether1

Bandwidth pinched through VxLAN tunnel

Bandwidth pinched through VxLAN tunnel

Re: Bandwidth pinched through VxLAN tunnel

Re: Bandwidth pinched through VxLAN tunnel

Re: Bandwidth pinched through VxLAN tunnel

Re: Bandwidth pinched through VxLAN tunnel

Re: Bandwidth pinched through VxLAN tunnel

Re: Bandwidth pinched through VxLAN tunnel

Re: Bandwidth pinched through VxLAN tunnel

Re: Bandwidth pinched through VxLAN tunnel

Re: Bandwidth pinched through VxLAN tunnel

Re: Bandwidth pinched through VxLAN tunnel

Re: Bandwidth pinched through VxLAN tunnel

Re: Bandwidth pinched through VxLAN tunnel

Who is online