Application Notes: Lessons for adaptive programs with the CM

In this document, we describe some lessons and ideas we've learned in creating or adapting applications which use the Congestion Manager. Don't take these as written in stone, but simply as suggestions and pitfalls you'll want to avoid in your own code. As we continue to receive feedback from developers about the CM, we hope this list will grow increasingly helpful.

  1. Event loops with existing applications:

    When modifying an existing application, you'll need to insert the CM dispatching into the existing event loop of that application. Don't just copy the blocking select code from one of our sample applications, because if you're unable to receive ACKs while blocked waiting for a write, your program may deadlock. This would be unfortunate.

  2. RTT calculations and response times Because applications must perform their own RTT calculations, there are a few things you need to be careful of if your application requires highly accurate RTT values or bandwidth estimates. Note that these are helpful, but not critical, for the congestion manager.

    On the sending side:
    If you have a low-bandwidth link, be careful about using the buffered UDP API. Because packets can go into a potentially large kernel buffer and wait there for an indeterminate amount of time _after_ you timestamp them, your observed RTT values may be higher than they should be. The ALF API does not suffer from this drawback. You can also shrink the size of your sending UDP buffer by using setsockopt(sock, SO_SNDBUF, ...).
    On the receiving side:
    Similarly, be aware of the effects of receiver buffering upon the observed RTT. If you calculate your RTTs by calling gettimeofday() after receiving the packet, then you can potentially include buffering time in your RTT calculation. We've found that the best way to avoid this is to request that the kernel timestamp the packets for you as they are recieved, bypassing the problem entirely. We've implemented the SO_TIMESTAMP socket option in the Linux kernel in which we distribute the Congestion Manager; it's also implemented in the latest (2.4pre8 and later) beta Linux kernels, and in {Free,Net,Open}BSD. If you really want this functionality in an older Linux kernel, you can also use the SIOCGSTAMP ioctl, though the SO_TIMESTAMP mechanism is both more portable and better.

    Our test framework provides an example of how to do this.

    Another option is to simply have your program dequeue packets from the socket buffer as rapidly as possible, but in general, you'll likely get best results (and save a system call) by using SO_TIMESTAMP.

  3. Our example framework does not do everything a sophisticated program may need. It's partly a short example, and partly a test framework, but it's not a complete application.

    In particular, if you send out variable sized packets, you'll need to use a little table to store the sizes of unacknowleged (or lost) packets so you can correctly report the size of the packet to the Congestion Manager. For simplicity, our test framework assumes that all packets are the same size.

    If frequent packet reordering is an issue in your network, you'll want to delay reporting losses to the CM until you've determined that it wasn't due to packet reordering. The framework ignores this problem. Our framework does not detect a "lost feedback" condition (e.g. where the program has lost enough packets that it no longer has a reasonable idea of the state of the network).

  4. FAQs and "things to keep in mind"