Tuning for high throughput low latency

[twhittock]

Hi.

I've been experimenting with NORM, and found it's really excellent at providing reliable multicast communications.

My use case matches the stream type of connection, for high data rate, low latency applications. I found that defaults in the NORM system really aren't conductive to this sort of use case. For example, the default socket baud rate is 64000 bits/s.

Are there other parameters users should tune to improve the performance of low latency high data rate streaming use cases?

[bebopagogo]

There are a number of things that can be set or tweaked to get higher performance and lower latency. What sort of realm of throughput are you looking for? We have (with a lot of coaxing) gotten to as much as 6-8 Gbps on 10 Gbps connections in some testing. At those higher rates, setting larger socket buffer sizes and other factors help. And those rates, the FEC parameters are zeroed since computation of the FEC becomes a bottleneck. But NORM can work as an efficient ARQ protocol even without FEC enabled. FEC does maintain reliable delivery efficiency up to larger scale group sizes, but there are use cases where this isn't necessary and NORM can work without it. Another thing related to ARQ operation that can impact throughput is NORM buffer sizes, the default timer-based flow control the sender observes, and the RTT time among your sender(s) and receiver(s). Following one of the examples to do explicit ACK-based flow control (see normStreamer, normMsgr, or normCast examples in norm/examples) can help here rather than just following the timer-based flow control discipline.

Similarly, another factor in NORM's ARQ protocol are some "backoff" timers used to suppress extraneous, duplicative NACK feedback as multicast group sizes grow. These "backoff" timeouts add latency to the ARQ process and latency can be reduced by zeroing these (NormSetBackoffFactor(sessionHandle, 0.0)) at the cost of losing the feedback suppression but that isn't crucial for some use cases with modest group sizes (or unicast).

If high assurance data delivery for streaming purposes is sufficient without absolutely guaranteed reliable delivery, another way that NORM can be used is to configure "auto parity" that transmits some FEC "repair" packets at the end of each logical packet coding block without waiting for any receivers to NACK. This adds transmission overhead, of course, but can provide lower latency delivery than the ARQ. You can use "auto parity" in conjunction with NACking or run silent receivers that do not NACK but rely upon the "auto parity" alone for high assurance delivery. With this type of operation you can set smaller FEC block sizes to manage latency and used the "maxDelay" parameter of the NormSetSilentReceiver() call to limit how many partially-received FEC blocks a receiver will buffer before releasing the data to the application. The reason it buffers by default is that in a system with a mix of NACKing and silent receivers, the silent receivers can also often benefit from repair packet transmission elicited by the NACKing receivers.

If you provide some more details on your use case, I may be able to help guide you on how to use some of these options and other tunable features. Yes - the default parameters are pretty antiquated (64 kbps, etc) and perhaps some more modern parameters (e.g., enabling TCP-friendly congestion control) should be used as default. In the ZeroMQ binding for NORM, I do something like that as opposed to the examples, etc.

[bebopagogo]

By setting the transmission rate (via NormSetTxRate()) so high you may be be causing packet loss by sending more packets than the 1 Gbps link can handle. I would suggest setting a NORM tx rate maybe double your application load rate so there is enough rate to allow for some amount of repair transmissions as needed. For example, try using NormSetTxRate(m_session, 250.0e+06) for 250 Mbps or 300 Mbps ... this will keep NORM in bounds of your 1 Gbps link rate

[twhittock]

Sadly even going down to a transmit rate of 100e6 still kept the outbound network adapter transmitting at 10Mb/s, looking at the Windows Resource Monitor. When receiving the messages, the disjoint messages come in at the same frequency, no change.

[twhittock]

I'm wondering if I must use some watermarking functionality to achieve a higher data rate? I am not sure how it would be used, but currently I'm simply writing 150k chunks and calling a passive flush with an EOM marker. Is there something special about the actual stream write we need to do to achieve performance?

[twhittock]

Hi again.

It turned out that everything in my setup was being rate limited to 10Mbit/s because my switch decided to apply flow control to the multicast traffic. Establishing a point-to-point connection, or turning off flow control both allowed the data to flow smoothly at high rates, with a reasonable latency overhead from the NORM system as compared to raw multicast packets. After some research, it turns out this is quite a common problem with consumer switches. It depends on which devices are plugged into the switch, regardless of whether they have joined the multicast group or not.

NORM's response to the rate limited traffic was a bit interesting - I think this is just the nature of simultaneously demanding reliability and throughput, but the end result was neither - approx. 50% of the rate-limited bandwidth was being used to reissue dropped packets due to the send demands exceeding the link capabilities, meaning I got less reliability and less throughput as a result.

I'm letting you know because I thought it might be interesting - I totally understand how it could happen, and it's a bit of a pathological case, but if you're interested in resolving this sort of usage scenario within the protocol in future, I wonder if it could be tuned to 'prefer reliability' or 'prefer throughput' in the face of limited bandwidth.

In the case of raw multicast UDP sockets, the sender application is "notified" of the rate limiting being imposed, due to the send method blocking execution. I feel that perhaps my use of NormStreamSetPushEnable caused my application to ignore this sort of feedback which could be present in the library. The reason I felt it was appropriate to enable this was because to my reading of the documentation, I thought it would only discard messages which had already been sent.

I hope this was useful to you - thank you again for the help with configuring NORM to reach my required performance.

[bebopagogo]

Yes - NormStreamSetPushEnable() lets your application favor newer messages/data it has to send over older messages in sitting in the NORM tx buffers that haven't been sent or necessarily reliably delivered. The "push" overrides the default timer-based flow control I mentioned too. So it can discard data that hasn't been sent at all if your link rate is much lower than what your application is trying to write to the NormStream.

NORM fixed rate operation is intended for cases where you have the actual throughput configured available on the network path. When you don't have known or dedicated bandwidth, that is of course where the congestion control options can be useful. For media streaming, it's good to have a fixed allocation. I have dabbled with some media apps where we dynamically adjust video (or audio) encoding dynamically in response to congestion control feedback from NORM's rate-based mechanism.

By the way, the "normStreamer", "normMsgr", and "normCast" examples are useful for benchmarking performance. They can be used in tandem with our MGEN message generation tool for test purposes. The "streamer" supports both "byte" streaming and the "message" streaming NORM supports and lets you experiment with different congestion control techniques, FEC parameters, etc.

Thank you for sharing that info about the switches and multicast. That is valuable information.