Frequently Asked Questions about
Well, "frequently asked" is a bit of a misnomer, since no one has
actually asked any of these questions yet, and now with this information
provided here, presumably no one ever will. But somehow "Might Have
Been Frequently Asked Questions" sounds too awkward.
Q. What is NIST Net, and what is it for?
NIST Net is a network emulation package that runs on Linux. NIST Net
allows a single Linux box set up as a router to emulate a wide variety of
network conditions, such as packet loss, duplication, delay and jitter,
bandwidth limitations, and network congestion. This allows testing of
network-adaptive protocols and applications (or those which aren't,
but maybe should be) in a lab setting.
Q. What is "emulation," and how does it differ from "simulation"?
As the terms are used here, "emulation" is basically testing code inserted
into a "live" implementation, allowing the live implementation to emulate
(imitate) the performance characteristics of other networks. "Simulation"
is a totally synthetic test environment, without a live component.
Q. What do I need to run NIST Net?
- A Linux 2.0.XX installation, with XX >= 30. Actually, the emulator
itself should work with any 2.0 kernel, while the required kernel
patches install successfully on at least 2.0.27 - 2.0.33. I have
personally only installed it on Slackware distributions, but can see
no reason it shouldn't work on any other distribution as well.
(
That last remark
falls under the category "famous last words," doesn't it? It turns
out there were some minor quirks installing it under RedHat, which
are fixed in the new version. It installs successfully under both
Slackware 3.4 and Redhat 5.0+updates, as well as earlier versions.)
- One or more network interfaces. Typically, NIST Net is installed
on a box with two Ethernet cards which is routing between two subnets.
This allows it to munge all traffic flowing between the two networks.
It can also be set up on an end node, to munge traffic into that node.
There is no dependence on the interface type; token ring or PPP should
work as well as Ethernet.
- An X11R6 installation, for the user interface. The interface is
built on the Athena widget set, but any of the drop-in replacements
should work as well. I personally use
the NeXT-like libneXtaw widget set,
as seen in the included screen shots.
Q. What are the machine requirements for NIST Net?
Essentially, NIST Net needs enough kernel memory to store any delayed
packets, and enough processor speed such that the additional overhead
it introduces doesn't skew its delay properties too noticeably.
(Currently, NIST Net does not account for its own overhead in computing
delay factors, under the assumption this is negligible.)
As a couple of data points, NIST Net has been run successfully on a
25/50 MHz 486 with 16M of memory doing emulation on 10Mb Ethernet,
and on a 200MHz Pentium with 32M of memory doing emulation on 100Mb
Ethernet. Measured per-packet overhead for the first configuration was
around 28 microseconds, and for the second, around 5-7 microseconds.
Both values are well under the usual minimum inter-packet times on these
networks, so should not have any (inherent) adverse effect on packet
handling. (The emulator reports average observed overhead through the
HITIOCTL_GLOBALSTATS ioctl.)
Q. Will NIST Net be ported to a 2.1 kernel, or an operating system
other than Linux? How about other widget sets?
The driving force to port it to 2.1 would be to run it with IPv6. This
may happen at some point. There don't seem to be any compelling reasons
to port it to other operating systems or widget sets at this time. When
used as a router, NIST Net emulation can affect any IP traffic from any
source to any destination, regardless of their operating systems.
If you want something similar that runs on SunOS, NIST Net was partially
inspired by (and retains some of the interfaces of) the
hitbox emulator used by USC in testing TCP/Vegas.
If for some reason
you want something similar that runs on a Windows operating system, I
understand there are commercial packages along these lines. I do not
have any further details about them, though.
Q. Why does NIST Net emulation only affect incoming traffic and not
outgoing traffic?
Basically because it was easier (i.e., required less disruption of
the kernel code) to do it that way. When NIST Net is used on a router,
catching packets at receive time suffices to affect all traffic.
This isn't true on an end node, of course, but hopefully the
provided capabilities will be sufficient.
For the purposes of the emulator, any congestion-dependant packet dropping
mechanism is really sufficient. The main problem with DRD in a router
implementation is that an instantaneous traffic burst could lead to a large
number of near-simultaneous drops, and hence correlation of the subsequent
restarts, from multiple TCP connections. But since this emulator can
treat any source/destination pair separately, there's no need to end up with
correlated drops.
Of course, if the goal is actually to test RED or some variant thereof,
it can be implemented as an "add-on" packet munger.
Q. Why are kernel patches required? Couldn't it all have been done
as a kernel module?
Well, in a certain sense anything could be done as a kernel module.
But there were a couple of areas where patching the base kernel
seemed like the most expeditious approach, even though it did add
a kernel version dependence:
- "Fast timer" code. In order to get fine enough granularity in
the timer for reasonably precise delay emulation, the timer chip is
reprogrammed to interrupt at a faster rate. In order not to affect
kernel code which relies on the normal rate, the interrupt handler
was somewhat reworked. This would have been rather difficult to do
outside this code.
- Network interface export. Not all the kernel network code
interfaces needed by the emulator were exported, for some reason.
There was also no interface to find existing packet handlers, for
cases (like this one) where we want to supplement, not replace,
the existing handler.
Q. Why didn't they implement a faster timer in ordinary Linux? Are there
bugs/problems with your approach?
Yes, there is a potential problem, which explains why the faster timer
wasn't implemented in ordinary Linux. The problem is that some Linux
device driver code can turn off interrupts for an indefinite period. If
this period is more than one timer tick, it's possible for timer interrupts
to be "lost" so the timer tick count will be off. For the ordinary
Linux timer tick interval of 1/100 of a second, this normally isn't a
much of a problem. With the code here increasing the rate by a factor of 76
(timer tick interval of 131 usec), this becomes more likely. How can
this be dealt with? Here are several approaches:
- Don't worry; be happy. The main effect of missed ticks from NIST
Net's point of view is that some packet delays are a bit longer than
they would be ideally. By and large, I haven't found this to be a major
problem. (NIST Net isn't really a "hard real-time" system.)
- On the machine with NIST Net, don't have a lot of device activity while
NIST Net is running.
- The biggest offender usually is the IDE controller. Try using
hdparm to get it to unmask interrupts sooner. I usually use the
following (put into rc.local, once for each IDE drive):
hdparm -m16 -u1 /dev/hda
See the hdparm manual entry for various caveats on its use, first.
If you can use these settings, do so; they will actually improve the
performance and responsiveness of your system. If you can't use them, though,
the "symptoms" may include massive disk corruption, so a little caution
is indicated...
-
The newest release of NIST Net incorporates a number
of timer fixes, including the latest 2.1.XX fixes from Ingo Molnar
and Ulrich Windl's "Completely Mad Time"
(neither of whom are to be blamed blamed for my timer hacks!)
These seem to compensate quite well for any potential problems,
especially on Pentium machines without APM, where the
cycle counter can be used as a check on the clock.
Q. [This one was an actual asked question!]
I set a straight bandwidth limitation of 8000 bytes/second. For ping
packets 1472 or below in size, I see no delays. When the size reaches
1473, though, the delay suddenly jumps to around 190 ms. What's going
on?
This is a consequence of the way IP behaves, and the
way NIST Net implements bandwidth limitation. When you set
a maximum allowed bandwidth of 8000 bytes/second, then as long
as the bandwidth utilization is below that level, NIST Net won't do any
delaying. That's what you see with packet sizes 1472 and below -
since ping sends one packet per second, its bandwidth utilization is well
below 8000 bytes/second.
So why are there delays for packet sizes 1473 (and above)? Well,
including the IP header, the actual packet size is 1501 bytes. On most
LANs (most networks, in fact), the maximum allowed IP packet size (MTU)
is 1500. So when you try to send a larger packet, IP will fragment it
into two packets. If you trace the traffic (with hitbox -S
src dest), you'll see that for each ping, two packets are sent in
a row, of sizes 1500 and 46 bytes. (46 bytes is the minimum ping packet
size.) When the first packet arrives, NIST Net notes that 1500 bytes
have been sent through that connection; to keep the instantaneous
bandwidth utilization below 8000 bytes/second, it will then delay the
second packet for 1500/8000 of a second (187.5 ms). This delays the
reassembly of the packet fragments at the receiving end by the same
amount.
NIST Net delays the packet because it looks at instantaneous bandwidth
utilization, i.e., it's (roughly) emulating a network where at no time can
you send more than 1 byte per 1/8000 of a second. So the second packet is
delayed, even though the long term average utilization is only 1546
bytes/second. (By the way, you should see the delays stay about the same
value for ping packets of size up to 2952; at 2953, the packet gets
fragmented into three pieces, and the delay times will double.)
Now one quirk of the implementation is that it only takes bandwidth
utilization by the previous packets into account, not the current one. So
when you're only sending packets every second, like here, the first one
essentially gets a free ride. I had thought about taking the current
packet into account, but with sustained traffic this will tend to
overcount bandwidth utilization and delay packets too much.
Q. [Another actual asked question!]
When I put new entries into the NIST Net user interface, it seems to hang
for a long period, then finally comes back. What's going on?
Almost certainly this is due to a problem with domain name resolution on
your system. In an attempt to normalize the appearance of names, when
you enter a new name into the user interface, it does two DNS lookups:
- A forward lookup of name to IP address, to find the address it will
furnish to the kernel emulator.
- A reverse lookup of that IP address to name, to find the "standard"
form of the name it will then display.
Usually, it's the latter that gives problems. Try nslookup host
and nslookup IP address for the entry you're adding. If
one of them fails, you can either fix your DNS server, or as a quick hack
add the problem host/IP address to /etc/hosts on the NIST Net
machine.
Q. [Vaguely based on yet another actual asked question]
What's going on with the units for the various delay/drop/etc. parameters?
And how do random delay and DRD actually work?
Heh, heh, I changed my mind several times about what units to use for the
various parameters, and I'm afraid the X user interface, the command line,
and the API aren't completely consistent. Here's a quick table:
Quantity
| X interface
| Command line
| API
|
Delay times
| milliseconds (floating point)
| microseconds (integer)
| microseconds (integer)
|
Bandwidth
| bytes/second (integer)
| bytes/second (integer)
| bytes/second (integer)
|
Drop/dup probabilities
| percentage of packets (i.e. 100xfraction) dropped or duplicated
(floating point)
| fraction of packets (floating point)
| x/65536 of packets (integer)
|
The random delay stuff was done in a way to make it quick to implement,
though a little clumsy to explain. I use a random number to do a lookup
in a distribution table, generating a "number of standard deviations"
value (multiplied by a scaling factor of 8192 to make it integral). The
delay value is then:
specified (mean) delay + (# of std dev)*(size of std dev)/scale
This gives random values which have the specified mean and standard
deviation, and which match the specified distribution. It's perhaps a
slightly cheesy method (possibly not matching some second-order effects),
but seems good enough for this purpose.
One other slightly confusing note is that while I specify all times
in microseconds, internally, they're rounded off to the nearest "minijiffy"
(minor timer tick), which by default is set to 1/7600 sec, around 131
microseconds. (The weird value is because it needs to be an integral
divisor of the frequency of the 8253 timer chip. Otherwise, the
machine's clock will start drifting off due to roundoff errors. Here at
NIST we have to have precise clocks!)
All the API parameters are integers, so the percentages get converted
to fractions of 2^16. What I do is generate a random number between 0 and
2^16. If it's less than x, the packet is dropped or duplicated.
The DRD parameters are the minimum and maximum queue lengths for the DRD
algorithm. More precisely, if the number of packets queued is less than
the minimum specified, DRD won't drop any packets. When the minimum is
reached, DRD starts randomly dropping 10% of the incoming packets. This
percentage ramps up with an increase in queue length, reaching 95% when
the maximum is reached. (You can actually have more packets queued than
the "maximum," but with 95% of all new packets being dropped, you tend
not to get very much above the maximum.)
The new ECN (explicit congestion notification) parameter must be a value
between the minimum and maximum queue lengths (or 0, which means
congestion notification will not be used for this connection). When
this is set, if a packet arrives which is marked with the ECN_CAPABLE
bit (currently bit 1) and the queue length is between the minimum and
ECN parameter, then NIST Net will mark the packet with the ECN_NOTED bit
(currently bit 0) rather than drop it. Not all packets will be so
marked, but only those that would otherwise have been (randomly)
dropped. If the queue length rises above the ECN parameter, then NIST
Net will drop a packet whether or not it is marked as ECN_CAPABLE.
Q.
I want to create my own delay distribution table. How should it be
set up to match the distribution I want?
The distribution used is governed by a table of short integers you can load.
The code generates a uniformly distributed "random" number between 0 and the
size of the table (4096 is what I used, which should be more than good enough
for any use I can imagine). This is used as an index in looking up the table
value, which is then divided by the table "factor" to get a "number of
standard deviations" value. In my tables, the factor was always 8192, and
the "number of standard deviations" always between -4 and 4. Again, as
mentioned above, the generated delay time is then
(average) delay + (number of standard deviations)*delay sigma.
Mathematically, the values in the distribution table should be
(... gomen, I have to go back to my notes to figure this out ...)
To load a new distribution table, you can of course recompile the
NIST Net driver. However, the driver actually supports loading new
tables at runtime. Simply write the table, which must be an array
of 4096 short words (8192 bytes), to the device driver:
cat binary.table > /dev/hitbox
When updating the table by this means, you must use a table size of 4096
and a table factor of 8192. (More precisely, these must be the same as
whatever table is currently loaded.)
Q. Tell me all that good legal stuff about copyrights and warranties.
As a U.S. government publication, so to speak, NIST Net is not
copyrighted. You can do whatever you want with it, including employing
its code in whole or in part in any other package or product.
You need not credit me or NIST (though not doing so would be a bit rude).
As is usual for code provided on this basis, there is absolutely no
warranty of any sort. We are interested in
receiving any reports
of problems or requests for improvements and will try to help,
but can't make any specific promises!
I had thought the above was fairly explicit, but apparently not explicit
enough. What the lack of copyright means is that you have a non-exclusive
right to use this code in any fashion you wish, including incorporating
it into a product without any further compensation or permission required.
"Non-exclusive" just means that other people can do it as well, so just
because you used NIST Net in your product doesn't mean somebody else can't.
Please note these remarks only apply to the code I originated; the patches
to the base Linux kernel are based on the existing Linux code and hence
carry exactly the same copyright restrictions as Linux does.
Comments? Questions? Let us know at
nistnet-dev@mail.antd.nist.gov.
[
Home Page]
[Installing NISTNet]
[Using NISTNet]
[NISTNet FAQ]
[Mark Carson's home page]