Hi Guys,

Ive recently upgraded one of my CCR1036-2S+ to version 7 which is being used as a router for my clients. Im performing traffic policy on it using simple queues (Majority CIR Connections) for each customer. Approx. have 350-400 simple queues on it. Ive tested fq_codel for simple queues to counter bufferbloat on 4 simple queues and it has worked as intended but Im a bit confused to implement it for all the simple queues just because Im not sure how will the CCR's CPU behave if fq_codel algo is selected in all simple queues. Can anyone share his/her experience of using this algo for such a large number of simple queues and the impact it has on the CPU of the CCR? There are no PPPoE sessions on this CCR. Just SVIs for each customer and in simple queues the target is set to the each customer's IP. Will using this algo for such a no. of queues increase the CPU utilization/choke any core during peak traffic (max. 2 Gbps in total)?

I would just implement it and go measure, and be ready to roll back. Pound it flat with artificial traffic at 4am?

(I am one of the authors of fq_codel and cake, but I do not have enough data either, on how well this stuff scales on given bits of mikrotik hardware). In most cases it is the per customer shaper that dominates the cpu by a factor of 9, and the underlying queue type be it a fifo, sfq, fq_codel or cake, adds only a tiny bit (cake is about 2.5 times as slow as fq_codel but does more, and again, it is the shaping cost that dominates). If you are low on memory, for folk running at less than a gbit, you can use memlimit 8mbyte rather than the default 32Mbyte.

On x86 gear we have individual implementations of cake scaling to 10Gbit/core. On 500mhz single core mips, cake scales to only about 80Mbit. your mileage will be somewhere between those. :/

See also.

viewtopic.php?t=179307

Thank you for the detailed explanation. You are right! I will implement and check what impact it has on the CPU. I read your post and my my, what a gem of information it is. Ill test it out and share results as well. I have 15.2 GB memory free on the CCR so I think wont be needing to change memlimit. Ill share my findings in a few days.

I do hope a table of performance metrics appears for more mikrotik gear. Sometimes I just hope they will put the fq_codel or cake algorithms more directly into their PCQ implementation. I am blogging more and more, and hoped that someday that insanely long thread on cake for mikrotik got boiled down into a compact usage guide. I look forward to hearing about your results!

https://blog.cerowrt.org/post/has a few pieces on flent and such...

Thank you for the detailed explanation. You are right! I will implement and check what impact it has on the CPU. I read your post and my my, what a gem of information it is. Ill test it out and share results as well. I have 15.2 GB memory free on the CCR so I think wont be needing to change memlimit. Ill share my findings in a few days.

I am really intensely curious as to how well this went? Did you try it, your router melt down, and lose your life to a lynch mob of unhappy users? Or... ?

我recently added cake as an interface queue on one of my CCR routers that tops out at about 700Mbps of traffic. I left the bandwidth limit off as a test, and I haven't noticed any change and I haven't had any user complaints. I set it up using the dual srchost and dsthost options. The CPU usage on this CCR barely ever breaks 0% even with cake running. I have only 3 raw firewall rules since it's just a router, so there's not much extra taxing the CPU. I'm curious to know if cake will actually do anything without the bandwidth limit. This router is fed by wireless so I'm not sure the bandwidth limit would help all that much anyway since there is likely to be less bandwidth when there are issues and that's when I'd hope to see cake help keep things running well. So now I'm just waiting to see how it goes the next time we have a bad storm and the link drops modulations.

Generally, queues need to get full before they become effective. If there's no pressure on the traffic, all the packets get thrown out there as fast as possible (FIFO). The bottleneck becomes the final delivery mechanism, i.e. loop to the house or WiFi to the end-user device.

The only place where I've seen QOS/COS useful without the queue being filled was on Cisco 3550's feeding Motorola (now Cambium) Canopy AP's. When we deployed VoIP, we were seeing (hearing) poor audio quality. Employing COS rules on the switch to get the DSCP-tagged VoIP packets out first, in addition to Canopy's QoS rules, allowed the VoIP packets to always get priority, regardless of how lightly loaded the AP/SM were.

Now (showing my ignorance), if Cake/fq-codel do reorder high-priority packets regardless of queue pressure, then I can see leveraging those queues on interfaces without shapers. In my simple WiFi speed testing at home, however, I don't see any results/benefits until I start maxing out the radio's capacity and apply a shaper that matches.

优先考虑这个词并不适用于fq_codel或cake (in besteffort mode). A better word would be "sparsity", the relevant paper on this is here:https://ieeexplore.ieee.org/stamp/stamp ... er=8469111- but in a nutshell it translates to flows having an arrival rate of less than the departure rate of all the other flows, go out first. This is a natural fit for voip (20ms interval), in particular, but applies to many other forms of req/response traffic. Only flows that build a queue are (slightly) deprioritized, interleaved over all the other flows.

As for the effectiveness of the algorithms on less than full bandwidth, voip is very subject to very hard to see in the aggregate tiny bursts of latency - a 60ms delay or loss is audible, and 100ms, very much so. If you are not measuring your link at a nyquist frequency (half the samplerate, e.g. for voip that would 10ms), and instead looking at 5 minute averages, everything might seem fine, but the voice quality still be poor due to microbursts elsewhere in the network. So in general, always apply FQ in some form, AND classification if you can do it, even if your link never appears to saturate over a 5 minute interval - and someday, if you can, find a way to sample via nyquist's methods!

"you will see things, you people won't believe...
锯齿飞过一个门……”

We are effectively sampling these days using ebpf at 10ms and I hope the technology makes it into more devices such as mikrotiks. Here's a demo (click run test, then select one of the slower sites)

https://payne.taht.net/

A much better metric for voip would be something like Glitches Per Minute - where someday, if we could get calls and videoconferences down to just glitches per hour, it would be a better world.

I am, btw, fairly certain that more than a few > 1gbit ISP uplinks to the internet are behaving this badly:

https://blog.cerowrt.org/post/juniper/

I loaded just a few queues on my CCR1036. Initially they were all set as Cake with 600M to 2G for each queue, with a total of 6 or 7 queues.

Today, as the total utilization approached 2Gbps, the CPU load jumped to 100%. This is with all packets being tagged and assigned to one of the half-dozen queues. I changed the main queue to fq-codel and the utilization drops to 18-20%.

I'll have to try this on an ARM router (CCR2116) to see if it can handle the Cake work better than the 1036.

Update:

我还把一些规则在一个RB4011 7.4.1运行,initially with Cake shaping all traffic going out to about six AP's, each on their own VLAN on the same ethernet port to the AP switch. Each queue was Cake with Internet RTT, Ethernet overhead, Ack-filtering, and diffserv4 (all else defaults). The queues themselves rarely hit 50-70% (only one ever hit 90%). CPU utilization was around 20%.

Today I got a call from a customer who complained that the last four days he'd been seeing packet loss, but hadn't made any changes on his end. I ran mtr to trace the path to his radio and found that I was losing packets at the router itself. After a few minutes of tinkering with the backhaul to rule it out, I disabled the queues on the router. I saw an immediate improvement, both to the router and to his radio. He likewise saw improvements in whatever tools he was using to test from his PC.

I changed the queue to fq-codel, and while the CPU utilization was lower, I still saw packet loss to/through the router, just not as much. As of now, I've disabled all queueing on the RB4011, turned fasttrack back on, and we're humming along at 6% load with 200-300Mbps continuous traffic.

Update update:

为1036 and the 4011, I disabled the shaper queues and created basic queues for the interfaces. The CPU load was causing more problems than perceivable benefits on both routers.

Even with basic settings for Cake on the 4011, I still saw packet loss increase. With fq_codel on the interfaces, it's working as expected.

On the 1036, Cake besteffort on the two SFP+ interfaces pushing 1Gbps has the CPU at 2-3%. With fq_codel on the interfaces, it's at 0%. No perceivable losses either way.

I'll have to try this on an ARM router (CCR2116) to see if it can handle the Cake work better than the 1036.

CCR2116 is DRAMATICALLY better than the tile platform. Tile is very slow for general purpose compute. Having a ton of crappy CPUs is ok... The ARMv8 cores in the CCR2116 must be like 10x faster per core for general purpose compute. I can get over 8Gbps one way on a single cake shaper between two CCR2116 with onboard mikrotik speed test.

Packet loss is not a particularly good metric to use against a cake or fq_codel instance, as it uses packet loss to control congestion if RFC3168 is not enabled by the endpoints. In exchange for decreased latency, you get more packet loss. So in both fq_codel and cake you should have seen an increase in packet loss, and an improvement in network latency. Did the network "feel" better? Did videoconferencing and voip work better? are better questions to ask.

Now, if cake is actually mis-behaving due to cpu load or otherwise, and randomly inducing packet loss rather than intelligently dropping flows, then you are right to disable it.

What I try to do in a circumstance like this is to get a packet capture of a few test flows, and look at them via wireshark. It could very well be that your queues and offloads and/or applications are behaving better in this case with the fast path enabled, or there is a bug related to the offload + cake, or that the customer finds the range of latencies they currently experience acceptable.

Packet loss is not a particularly good metric to use against a cake or fq_codel instance, as it uses packet loss to control congestion if RFC3168 is not enabled by the endpoints. In exchange for decreased latency, you get more packet loss. So in both fq_codel and cake you should have seen an increase in packet loss, and an improvement in network latency. Did the network "feel" better? Did videoconferencing and voip work better? are better questions to ask.

Now, if cake is actually mis-behaving due to cpu load or otherwise, and randomly inducing packet loss rather than intelligently dropping flows, then you are right to disable it.

What I try to do in a circumstance like this is to get a packet capture of a few test flows, and look at them via wireshark. It could very well be that your queues and offloads and/or applications are behaving better in this case with the fast path enabled, or there is a bug related to the offload + cake, or that the customer finds the range of latencies they currently experience acceptable.

Dave, I see dramatically less packet loss with cake. Cake may be dropping packets as part of traffic control, but by managing the congestion other flows drop a lot fewer.

But back on topic, that CCR1036 has too slow single core performance IMO to do large shapers.

To the OP, as long as it's individual shapers and NO top level shaper, then CCR1036 will spread the shapers out on separate CPUs. If you do a top level shaper, everything in that shaper tree will be trapped on a single core.

i agree about limited single core performance on Tile Architecture, but in the test realized bysirbryanhe replicated the situation in a rb4011 which has much better single core performance (it has OoO A15 CPU) at a rate normal for that router doing shapping 200-300mbps

i agree about limited single core performance on Tile Architecture, but in the test realized bysirbryanhe replicated the situation in a rb4011 which has much better single core performance (it has OoO A15 CPU) at a rate normal for that router doing shapping 200-300mbps

Rb4011 isn’t dramatically faster. It’s a 32bit arm v7.

That said, I have rb4011 running up to about 650Mbps aggregate with cake. I can get that reliably. I use them for backup backhaul links so I have fq-codel or cake handling the constriction.

To the customer, the feel was horrible, as he was no longer able to reliably play his games. He spent the previous four days testing things with friends, updating drivers, twiddling with settings on PC, etc. to no avail. Latency was all over the place and just in my ping tests we were losing one or two every 10-15 seconds. MTR reported 6-8% packet loss to the router and 4-5% to the radio.

I've reconfigured the queue setup on the 4011 and will report back. After some more digging, I think I "overconfigured" it initially, burdening the CPU with unnecessary tasks.

I've reconfigured the queue setup on the 4011 and will report back. After some more digging, I think I "overconfigured" it initially, burdening the CPU with unnecessary tasks.

Feel fry to post a sanitized config.

also note, you cannot use fasttrack with shaping. I've seen it a few times that someone has the fasttrack firewall rule and shaping and it's not behaving as desired, only shaping things that have hit other firewall rules and so on.

Here's what I've whittled it down to:
There's only one firewall rule presently, which is to drop any traffic destined for the public IP addresses assigned to the router from non-internal IP's.

The queues show the same amount of traffic whether or not fasttrack is enabled. Since these queues aren't tied to "global", I imagine that fasttrack should work. Either way, it's not impacting the CPU much at this point as traffic is just under 100Mbps right now.

(To clarify, this particular config is working with zero packet loss, so far. So the problem I was seeing was likely due to mangling everything coming into the router.)

I don't like how queue trees behave when explicit directionality isn't set via a packet mark.

Am I extrapolating that you have a vlan per customer from this tree?

If it were me, I would separate this out to:
mangle add packet mark LTU192DL where dst is vlan4001
mangle add packet mark LTU192UL where src is vlan4001

Then 2 queue trees for the separate DL and UL.

Also, I'd do diffserv4 instead of besteffort.

Here's an example of how I build these. This is a CCR2004 so I can get away with a bit more than on a 4011 but principals are the same. This is a very effective model. I don't use vlans per so this is IP matched
match ip and mark UL and DL packets separately via mangle
Add a queue tree with NO markers to identify network structure.
add individual queue with packet mark and shaped speed.

currently doing fq_codel for 'groups' and cake for individuals.

thank you for filling in here. I note that in a properly functioning fq_codel setup, you should see drops or marks. Not seeing them is bothersome.

to clarify here for the readers, the fq_codel and cake in routeros <=7.8 does not show drops or queued packets. I have a ticket open with mikrotik support on this that is marked 'pending'. Until that is resolve, you wont know if packets are being dropped.

I don't know if you also requested RFC3168 marks be logged also? A surprising amount of packets are being marked by fq_codel/cake etc, today, being signaled end to end by at least some (apple?s?) devices requesting ECN support. RFC3168 ecn is a way to do congestion control without dropping packets, and is baked into most OSes today, accepted by most TCP servers today, but rarely initiated by clients. I kind of suspect some embedded clients are negotiating it given just how much of it I have seen of late - or mismarking it - can't tell without logging it!

RFC3168 ECN is enabled by default in fq_codel and cake.

It's a one liner to enable in windows and linux transports.

https://www.bufferbloat.net/projects/ce ... nable_ECN/

Apple uses a variety of heuristics as to when they enable it, nowadays.

I always thought that the ECN support in fq_codel would make it easier for those with tough SLA requirements regarding both limiting drops AND latency (how?), to suggest further deployment of ECN enabled transports to their clients and servers.

Here's an example of how I build these. This is a CCR2004 so I can get away with a bit more than on a 4011 but principals are the same. This is a very effective model. I don't use vlans per so this is IP matched
match ip and mark UL and DL packets separately via mangle
Add a queue tree with NO markers to identify network structure.
add individual queue with packet mark and shaped speed.

currently doing fq_codel for 'groups' and cake for individuals.

Screenshot 2023-03-01 at 5.04.18 PM.png

do You use FastTrack with this setup ?

Here's an example of how I build these. This is a CCR2004 so I can get away with a bit more than on a 4011 but principals are the same. This is a very effective model. I don't use vlans per so this is IP matched
match ip and mark UL and DL packets separately via mangle
Add a queue tree with NO markers to identify network structure.
add individual queue with packet mark and shaped speed.

currently doing fq_codel for 'groups' and cake for individuals.

Screenshot 2023-03-01 at 5.04.18 PM.png

do You use FastTrack with this setup ?

No. you cannot. fasttrack bypasses queues.

No. you cannot. fasttrack bypasses queues.

Actually it works very well for me but obviously only for upload and with Queue Tree

No. you cannot. fasttrack bypasses queues.

Actually it works very well for me but obviously only for upload and with Queue Tree

Only because you are interface routing. To match by IP or packet marks etc you have to hit the firewall, which fasttrack bypasses.

that's why I'm asking

as soon as I add rules before fasttrack:to use those for simple rules, the general speed for that network drops from 800-900mbps to max 400

just thinking how to make it work in best way.
I have a NAS device for which I would like to limit upload during daytime, but give full speed over night.
with rules like above, simple queue is working fine, but would have to disable them for night to give full speed. plus during daytime it's cutting download due to skipping fasttrack..

I don't like how queue trees behave when explicit directionality isn't set via a packet mark.

Am I extrapolating that you have a vlan per customer from this tree?

Each of those VLANs goes to an AP. I have 10-25 customers per AP. The queueing is on the VLAN's egress.

My hope was to take just enough of the edge off before traffic hits the AP as opposed to letting the AP's max out.

So far, so good.

that's why I'm asking

as soon as I add rules before fasttrack:

Code:Select all

3 ;;; cust: guests download chain=forward action=accept connection-state=established,related dst-address=192.168.100.0/24 log=no log-prefix="" 4 ;;; cust: guests upload chain=forward action=accept connection-state=established,related src-address=192.168.100.0/24 log=no log-prefix=""

to use those for simple rules, the general speed for that network drops from 800-900mbps to max 400

just thinking how to make it work in best way.
I have a NAS device for which I would like to limit upload during daytime, but give full speed over night.
with rules like above, simple queue is working fine, but would have to disable them for night to give full speed. plus during daytime it's cutting download due to skipping fasttrack..

Then you need faster hardware. rb4011 can handle about a gig of cake queues as long as it's not doing NAT, if you NAT on the same box that's more like 600Mbps.

You could modify your fast track rules to allow est/rel for the rest of the hosts. ie, dst-address!=nas.ip.ad.ress in your rules.

Or look at the nas and see if it has a built in upload scheduler.

Each of those VLANs goes to an AP. I have 10-25 customers per AP. The queueing is on the VLAN's egress.

My hope was to take just enough of the edge off before traffic hits the AP as opposed to letting the AP's max out.

So far, so good.

you just loose flexability because you can only really shape 'downloads' to the individual AP. What if your uploads get overwhelmed? You don't have an interface shaper that will handle that if you have more than 1 AP.

you just loose flexability because you can only really shape 'downloads' to the individual AP. What if your uploads get overwhelmed? You don't have an interface shaper that will handle that if you have more than 1 AP.

到目前为止,还没有一个问题。在将来point I'll load queueing + shapers onto customers' IP's, or better yet onto their routers on the outbound interface.

you just loose flexability because you can only really shape 'downloads' to the individual AP. What if your uploads get overwhelmed? You don't have an interface shaper that will handle that if you have more than 1 AP.

到目前为止,还没有一个问题。在将来point I'll load queueing + shapers onto customers' IP's, or better yet onto their routers on the outbound interface.

Ideally you shape downloads on the head end with a big shaper tree, so you don't transport packets you will end up dropping. And you shape uploads twice, once at the customer for plan enforcement, and once at or right above the AP to keep in from getting overwhelmed. maybe also at each backhaul if you have constraints.

I would just implement it and go measure, and be ready to roll back. Pound it flat with artificial traffic at 4am?

(I am one of the authors of fq_codel and cake, but I do not have enough data either, on how well this stuff scales on given bits of mikrotik hardware). In most cases it is the per customer shaper that dominates the cpu by a factor of 9, and the underlying queue type be it a fifo, sfq, fq_codel or cake, adds only a tiny bit (cake is about 2.5 times as slow as fq_codel but does more, and again, it is the shaping cost that dominates). If you are low on memory, for folk running at less than a gbit, you can use memlimit 8mbyte rather than the default 32Mbyte.

On x86 gear we have individual implementations of cake scaling to 10Gbit/core. On 500mhz single core mips, cake scales to only about 80Mbit. your mileage will be somewhere between those. :/

See also.

viewtopic.php?t=179307

Hiya.. that certainly looks like an easy comment, but when u are running on a production hardware mikrotik with over 1000 pppoe-sessions.. it does not seem the smartest thing to do.. beta-testing on a production hardware with lots of pppos-sessions runnig..

Hiya.. that certainly looks like an easy comment, but when u are running on a production hardware mikrotik with over 1000 pppoe-sessions.. it does not seem the smartest thing to do.. beta-testing on a production hardware with lots of pppos-sessions runnig..

Certainly is, that's why you need to bench test. Further, I find that ipq4018 devices aren't stable on cake when you overrun the queue. easy to replicate by doing a bandwidth test at 2x the shaper, the device will crash. That's probably because not many people are doing this across all hardware types.

I'm running 1G queue trees 8 levels deep ( the limit with the current kernel builds ) and all cake on ccr2004 and ccr2116 AND rb4011 though it's right at the edge on the rb4011 and I've had to clock it up to 1.8Ghz. Stable.

Thank you for the detailed explanation. You are right! I will implement and check what impact it has on the CPU. I read your post and my my, what a gem of information it is. Ill test it out and share results as well. I have 15.2 GB memory free on the CCR so I think wont be needing to change memlimit. Ill share my findings in a few days.

I am really intensely curious as to how well this went? Did you try it, your router melt down, and lose your life to a lynch mob of unhappy users? Or... ?

Hi. So at last i got the time to test it out. I did implement FQ_Codel using simple queue for a customer of mine on the CCR with a capping of 800 Mbps and to my disappointment, the CCR's CPU got clocked at 100%. In addition, the throughput for the customer also dropped to half of what Ive set in simple queue. Ive not tested it out on customers with less than 100 Mbps circuits yet. But it seems CCR cant handle it on high capacity circuits.

Hi. So at last i got the time to test it out. I did implement FQ_Codel using simple queue for a customer of mine on the CCR with a capping of 800 Mbps and to my disappointment, the CCR's CPU got clocked at 100%. In addition, the throughput for the customer also dropped to half of what Ive set in simple queue. Ive not tested it out on customers with less than 100 Mbps circuits yet. But it seems CCR cant handle it on high capacity circuits.

I assume that this is on your 1036 right? Those TILE CPUs just can't cut it. CCR2116 is THE mikrotik shaping box to get...

What is the situation with BQL driver support on MikroTik ROSv7.8/7.9?