Strategies for Communications System Software Design

by THAYUMANAVAN SRIDHAR

There's no one "right" way to design and implement a communications software system. Here are some guidelines to put you on the road to success.

Is design and development of software for a communications system any different from other embedded systems development or are there specific issues which you should consider? How would you do the task partitioning and scheduling, for instance? How will the data be processed and what kind of buffer management scheme should you use? Are there specific considerations in the choice of polling vs. interrupts? Can any of the software functionality be tested without the hardware?

This article answers these questions and details some of the techniques that could be used in the design of a communications software system. The techniques I've enumerated are culled from the accumulated knowledge over several communications systems projects and can be considered a checklist for communications software design and programming.

Scheduling
A communications system consists of multiple functions which may or may not require their own thread of execution. Typically, the system can be characterized as one which processes asynchronous events such as arrival of messages and periodic events like timer expiry to perform protocol-related functions. Details may vary depending upon the protocol, but this broadly describes the nature of the communications system.

Division of the system into tasks is based on traditional methods-do the functions relate to and depend on each other? If they can be carried out in parallel, then they could be separated into tasks. However, a finite time is required for context switches, so this method may not always be best. Also, some of the functions may be less critical and can be performed in the background instead of inline with the main task. A typical example is a router in which the routing protocol task can carry out its updates and processing at a lower priority, as opposed to the main forwarding task.

When the system has been partitioned into tasks, the nature of the real-time operating system (RTOS) kernel becomes important. Many embedded systems have a tight requirement on memory and may prohibit a more general-purpose RTOS. Though small-footprint commercial RTOSes may be used, proprietary, small-footprint kernels are preferred in many cases. The nature of the scheduling in a number of these proprietary kernels is a simple, nonpreemptive run-to-completion method. Such kernels require a certain discipline on the part of the task designers to ensure their tasks relinquish control, in order to avoid starvation of the other tasks. Commerical RTOSes on the other hand, use preemptive multitasking. The tasks' priorities and their relation to other tasks are most critical in this scenario.

Consider a router system based on a high-priority forwarding task; all other tasks are of a lower priority. Assume the forwarding task queues up a lot of frames to a wide area network (WAN) task, such as point-to-point protocol (PPP). Unless the WAN task gets a chance to run, the actual forwarding will not be completed as the outgoing frames are all queued up. In such cases, the WAN task may need to have a higher priority than the forwarding task itself.

Another example would be the management task in the same router system. We may consider the management task to be a lower-priority task at first, because it provides noncritical functions. However, this isn't always true-the management task may need to set some configuration to change the behavior of the forwarding task, or to disable it. If it doesn't get a chance to run because the forwarding task is running all the time, we'd have a situation in which the system can't be reined in due to misconfiguration. In such cases, the management task may actually need to have a higher priority than the forwarding task-the caveat being that it shouldn't take an inordinate amount of time to perform its function.

Buffer Management
Every communications system needs to have an efficient buffer management strategy. In many cases, this strategy is key to the performance of the system as a whole. In this discussion, we'll use a simple definition for a buffer: a unit of memory used to hold the data that has been (originally) received from a network interface or is to be (ultimately) sent out over a network interface.

The biggest issue in buffer management is whether the software needs to manage its buffers or use the system services like malloc or free. Each approach has pros and cons. If the system services are used, then the communications system is free to concentrate on its core functions. Other support functions use the standard system services as well. The memory block allocated is closer in size to the request because the standard malloc function uses the best-fit algorithm in most systems. The downside to this approach is that typically, these calls are slower because the routines are more general purpose and not really tuned to the communications system application.

If the software manages its own buffers, it can use a "one-size-fits-all" approach or use multiple buffer pools, with each of the pools involving buffers of specific sizes. In the first case, the software will always allocate buffers from the same pool with all the buffers of the same size. Lower-size requests will result in a waste of buffer space, since the default buffer size will be greater. Higher-size requests will involve chaining of multiple buffers and presenting them to the application. This situation can also result in a waste of buffer space. However, the allocation call will be very fast, because the only calculation to be done is for the number of buffers required, since the buffer size is known. In this way, the software will use a "loose" best-fit algorithm when allocating buffers.

A number of systems use buffers with the highest possible frame size, to avoid fragmentation. Thus the overhead of chaining and associated processing is reduced significantly. The problem with this approach is that if most of the frames are below the maximum frame size (which is often the case), buffer space is wasted.

Whatever the method used, communications software systems must try to implement a method of "minimal copying" in the processing of frames, especially in the data path. Pointers to the appropriate offsets are passed to higher layers in the OSI stack as soon as one layer is done with the incoming packet. On the outgoing side, higher layers reserve space for the headers of the lower layers when creating the frames to be sent out. So the copying of headers on top of frames is a simple task that doesn't involve moving the data bytes around (see Figure 1). A chained buffer scheme provides indirect support for this technique by providing the ability to attach a buffer (or a chain of buffers, if required) to the start of an existing chain.

Buffers may also be used for inter-task communication. Systems can use a separate buffer pool for this communication, or they can use the same buffer pool as the one that's used for incoming or outgoing packets. The same considerations with respect to efficiency of usage and overheads apply.

How does this translate to the schemes that hardware communications controllers use? A lot of variation exists in the buffer management schemes that standard communications controllers use. Many of them use a chained buffer scheme. For reception, a number of buffers are pre-allocated and provided to the controller. The controller moves the received frame into these buffers via DMA, and if the frame doesn't fit into the first buffer, it uses the next buffer in the chain, and so on. It signals via a bit as to which is the last buffer in the frame. The buffers containing a frame are usually delinked from the controller's queue and merged into a message that conforms to the buffer management scheme. The buffer manager also links buffers into the controller's queue to replenish the delinked buffers. In some older controllers, the buffer manager must copy the data from the linked buffers in the controller queue into its own message, and indicate to the controller that the same buffers can be reused.

During transmission, the buffers from the buffer manager queue are delinked from that queue and enqueued to the controller transmit queue. An alternate scheme involves the data from the buffer manager queue being copied to the buffers in the controller transmit queue. Subsequently, the controller is signaled that these buffers are ready for transmission.

When implementing a buffer management scheme, it's best that a designer integrate buffer statistics mechanisms into the scheme. For example, counting the number of "gets" and "releases" in a global structure for each pool is useful. The difference between the number of gets and releases may not always be zero if some buffers have been pre-allocated for driver use. During testing and debugging, this difference should always be the same number (though it may peak out and then come back to this value). If it isn't, then the buffers are being held up somewhere in the system or have not been properly released to the pool. Similarly, a counter that notes the peak usage of the buffers (the maximum value of the difference between the gets and releases during the life of the system) will give the designer a good idea about how to tune the buffer pool sizes.

Buffer statistics can be augmented by counters for allocations and releases per module. This information will help determine the level of usage of buffers by various modules, as well as indicate which modules are holding up the buffers.

Polling vs. Interrupts
The topic of polling vs. interrupts is thoroughly covered in the embedded systems programming realm, so we'll look at it now only in terms of communications software. At the lowest layer, most communications controllers provide a mechanism to generate an interrupt per packet received or transmitted. These controllers also provide the ability to turn off the interrupts, but indicate via status bits that a packet was received or transmitted. The software can poll these status bits and process the reception or transmission.

Interrupts are preferred if the interrupt latency is low and the system's main processing isn't seriously affected by the given frequency of the interrupts. Consider frame reception: a system should be able to process the received frames, not just spend its time handling interrupts for received frames. If the interrupts get too frequent, the software can be changed to poll for received frames at a desired frequency and perform its main function. This approach may also be relevant for high-speed interfaces. While transmitting, communications systems frequently use interrupts for indication of successful or unsuccessful transmission. These interrupts may indicate that the buffers can be released or retransmission can be initiated. In most cases, this interrupt-driven processing can be replaced by polling. Some systems use interrupts only if an error occurs on transmission. These errors usually require immediate action.

Interrupts and polling are used in another less known but nevertheless important function, which has to do with the basic physical connectivity. Consider a router that is connected to a WAN via a channel service unit/data service unit (CSU/DSU). The router and CSU/DSU may be connected via a V.35 interface cable. If a loss of physical connectivity occurs between the router and the CSU/DSU (say the cable is broken or has been pulled out inadvertently), the router software should be signaled. Interrupts appear to be the best option here. However, spurious and transient loss of physical connectivity should be distinguished from the permanent loss of connectivity. So the communications software may need to poll for the status of the connection periodically once it has been signaled via the interrupt about the loss of connectivity. In many communications systems, ignoring transient outages is important. If they aren't ignored, a disruption may occur in the operation of the higher-layer protocols.

Trace and Packet Dumps
When writing code for communications systems, programmers will find it easier if they build in the trace and packet dump functionality up front in the code, rather than introducing it as an afterthought. The trace functionality indicates which modules and functions are executing to process packets on the incoming and outgoing sides. It typically involves calling a trace function with a level for the trace function call, trace type, and the various parameters indicating the state of some key variables at the time of the trace. The levels are usually bit masks which can be set via configuration (which I discuss in the next paragraph). These trace calls are typically written as macros which can be compiled in with the actual call or dummied out to a null function when the system is stable.

Another key function for debugging is the ability to dump packets at various layers/modules of the communications software system. This ability is required to quickly get an idea of the packet trace at each layer of processing. These functions can again be written as macros or dummied out. They can be activated via the use of global masks, with each bit representing a certain module/layer. For example, a mask may be defined in an IP stack with one bit for IP, another for UDP, another for TCP, another for Telnet, and so on. To dump packets at both the IP and UDP layers, both the IP and the UDP bit can be turned on. This feature is very useful for debugging problems both within the same communications system and connected systems. Additionally, a feature that selectively enables the dumping of incoming or outgoing packets, or packets in both directions, will be very useful.

OS-Independent Programming
A number of communications system vendors have taken a common approach to one issue-assumptions of the system-provided facilities like inter-task communication, buffer management, timer management, and the like. This approach has been motivated by factors like nonavailability of the RTOS simulator on the development platform (which used to be very common), nonavailability of the hardware, uncertainty about using the same RTOS across multiple products (even though the communications application has to be migrated to that product), and so forth.

So the designers of these systems have developed generic operating environments (RTOS-independent operating environment, or RIOE) which act like a thin layer on top of the RTOS, as shown in Figure 2. The communications application is written to use the services of this operating environment and be completely independent of the underlying operating system (OS). The areas in which this tactic is applicable include inter-task communication, buffer management, timer management, and the like. Having the communications software (especially the protocol software) write to this common interface makes the software portable, in addition to offering convenient testing in an OS-independent manner. Porting to a new RTOS only involves changing the mapping of the RIOE to the new RTOS and involves no change to the application. The RIOE can be typically ported to run over a standard commercial non-real-time system on the development station, so the programmers can make use of the standard tools available on the host station. Of course, this last feature is no longer a great advantage because a number of embedded systems tools offer excellent debugging and profiling capabilities.

This approach is also useful when a lot of system-independent development and testing needs to be done-say, a communications protocol in which the protocol state machine alone needs to be verified. The programmer can concentrate on writing the interfaces to the generic operating environment and ignore the workings of the drivers, and the like.

Checklist
Now that you have the basics down, here's a checklist for the design and implementation of a communications software system:

Determine your strategy
This article has discussed the various issues and trade-offs related to the design and implementation of a communications software system. As with all other embedded systems, no single, "right" approach can be chosen. Designers and developers can use this article and the checklist to determine a strategy for their specific implementation.

Thayumanavan Sridhar is director of engineering with Future Communica-tions Software in Santa Clara, CA. He has 10 years' experience in the communications software industry and an MS in electrical and computer engineering from the University of Texas at Austin. Reach him at sridhar@futsoft.com.

Return to Embedded.com
Send comments to: Webmaster
Copyright © 1998 Miller Freeman, Inc.,
a United News & Media company.