Keeping NFS running and what to do if it isn't.

Tru64 UNIX

Evolving business value

Related links

This page offers NFS knowledge that anyone who supports NFS systems should know. NFS is a key component of many UNIX systems, and is often the prime consumer of network services.

When NFS doesn't work, people seem to say they have "NFS problems", not "Unix problems" or "network problems.” NFS uses much of a Unix kernel – and others – so it can be tricky to triage problems. We’re offering this insight, hints and hopefully, help!

Engineering tips

The key thing to keep in mind when dealing with NFS is that it is two products, a client file system and a server that uses file systems much like other commands an programs do. It is vital to remember that the client and server are separate entities. When reporting problems, always include system and version number information for both client and server

Tcpdump is a vital tool in diagnosing NFS problems. Configure for it, install it. Use it every day. Tcpdump's philosophy is "one frame, one line," i.e. it generally prints one line per network message. This is great when you're trying to get an overview of what's going on, but there are times you need to see everything about a message. Options "-mv" will enable some multiline and verbose output which is sometimes enough. To see everything, the best options are to use snoop on a Solaris system or Ethereal, an open-source X11 based protocol analyzer.

Very few "NFS problems" wind up being software problems within Tru64 UNIX or even other Unices. A kindred spirit commented

The number one cause [of NFS performance problems] is network configuration. So are numbers 2 thru 99.
Lon Stowell
Posted in comp.protocols.nfs

When the question is "Why isn't it working?", the first thing to do is see what the client can coax from the server. You start with the basics and work to more complex cases. Do the following from the client.

• ping server
  Ping sends echo request ICMP messages to a server. The ICMP server is implemented within the kernel and relies on next to no user level daemons. If ping doesn't print replies, you have a network configuration problem, not a NFS problem. If you haven't checked for unplugged power and data wires, now is the time to do so. Then check ifconfig lines and make sure entries in /etc/hosts are consistant.
   
• rpcinfo -p server
  Rpcinfo is a simple ONC-RPC test tool. -p queries the portmap daemon on the server for a list of all RPC services currently registered on the server. If it times out, portmap probably isn't running or is sick.
   
• rpcinfo -u server nfs
  This calls a NULL procedure that is implemented in all applications. In addition to -u (UDP), you can use -t (TCP).
  
• rpcinfo -u server mountd
  NFS V2 uses MOUNT V1. NFS V3 uses MOUNT V3. Don't worry about rpcinfo's complaints if MOUNT V1 and V3 are available. You should see:
  
  program 100005 version 1 ready and waiting
rpcinfo: RPC: Program/version mismatch; low version = 1, high version = 1
program 100005 version 2 is not available
program 100005 version 3 ready and waiting

Mount problems
If all the above works, and the mount command doesn't, problems are often due to restrictions mountd is applying. Mountd often returns little and often doesn't make a syslog entry.

Mountd's options offer a spectrum of privacy (i.e. opportunities for mount failure). The options are better described in mountd(8), but the general range from most permissive to least is

• mountd -n
  This accepts requests from anyone at any site. It is only appropriate outside of a firewall for file systems exported read only that contain no private data. If this is not used, requests from non-root users will be rejected with a complaint about weak credentials. Some non-Unix clients may attempt to use AUTH_NONE, this will always result in a weak credential complaint.
   
• mountd -i
  When ONC RPC programs receive requests, they generally get the IP address of the sender from the OS and the nost name from the authentication header of the request. If mountd can't match the hostname and IP address, it will reject the request. Often the /etc/hosts file on client and server are mismatched, confusing both systems and the system administrators who made the change just before the end of the workday.
   
• mountd -d
  This disallows requests from clients in other DNS domains, rejecting the request with rejected credentials, or whatever the error is.
   
• mountd -s
  This is claimed to enable subdomain checking, but it may do the same thing as -d.

Further narrowing down the problem
Try to reproduce a problem by using common Unix commands. If you can, write a minimal C program to use for final study. The problem with relying on Unix commands is that they are rife with surprises. "cp" does not simply read one file and write another - it does various stat()s and other calls before it reads any data at all. Consider this report:

   
A couple other odd items. If we "dd if=foo.txt of=/dev/ttypn" on the

   client, the mtime does change. If we "cat foo.txt >> /dev/ttypn" on

   the client, the mtime does not change.[Later] Using dd of=/dev/ttyp2

   conv=notrunc does NOT update mtime. 

First, dd is not a simple program. Second, there are at least two ways to truncate a file. One is to include O_TRUNC in the open(2), another is to use ftruncate() or truncate(). Instead of looking at dd to see which it uses, it may be easier to write a pair of test programs to see if either or both of these change mtime. That information combined with a tcpdump trace will put you 2/3s of the way to a diagnosis.

This is a more complicated problem. Whereas before you couldn't even get to NFS, now you can, some of the time or even nearly all of the time. Now the whole array of software NFS touches comes into question - network, file system, VM/UBC. Where do you start?

As always, you start with the simple things. This time you look for places where messages were lost or delayed. If you can find a packet loss rate of more than 1% fixing that will usually solve the problem.

• netstat -is
  This prints MAC level statistics, information about the performance of your network interfaces. For example:

Three counters have reached their upper bounds in the week the system has been up. Look for the number of data blocks sent and received and compare that to send and receive failures. In a clean Ethernet with no one plugging and unplugging cables, the error counts can be zero, except in a grossly overloaded network. In that case, expect several "Excessive collisions" errors. Here there are about 0.16% excessive collisions, high enough to have a measurable effect on NFS. The only solution is a faster network or to break on into multiple subnets.

Any other error is bad. Period. Communications theory claims that no communication channel is perfect, but properly configured Ethernet is astoundingly good. Almost all output errors are especially worrisome. If you see "Remote failure to defer" it means that the Ethernet is too long or that a transceiver is not noticing when it has won the wire. Other errors suggest hardware failures of some sort. A high receive or transmit error rate will cause bad performance.

Receive errors are less exciting than transmit errors, but those 65 errors are higher than you will see on a clean net. The error types are typical – maybe even the full set. On the same machine, apparently tu1 is talking to a cleaner subnet:

Half as many packets received, no errors. That's the way it should be.

Don't let yourself be lulled into thinking that the CRC check is doing its job. Keep in mind that an average 16 bit CRC will still let one message out of 65536 through, but there are situations where it may not be that effective. It's well worthwhile to keep your network as error free as possible so that any hardware or configuration problems will stand out when they do occur.

Many people would worry over the high collision rate on tu0. While there are some issues, a high collision rate ties up very little bandwidth. It's fascinating to watch 10Base2 Ethernet on an oscilloscope. You can see collisions (the signal is twice as strong) and they take something like 51 usecs. Large Ethernet packets take about 300 times that, so even a 50% collision rate may not harm performance. You can often pick out NFS reads and writes, as those show up as multiple packets with the minimum 9.6 usec separation time.

On the other hand, a high collision rate means a highly loaded Ethernet. Given the tools we ship, That may be the best clue that someone is flooding the net with junk. Tcpdump is the best tool to find out who and what.

• netstat -s
  Netstat also reports statistics for IP and it's users. It produces a lot of output so the initial reaction is "Too Much!" The key things to look at are marked with asterisks below. Their importance is listed afterwards. Sections without interesting data for NFS are not included

ip: 103655445 total packets generated here
ip: 112277645 total packets received
tcp: 40524497 packets sent
tcp: 25716962 packets received
udp: 61642016 packets sent
udp: 63183812 packets received
These just set baselines for the amount of activity to compare against the various error counters below.

ip: 166 bad header checksums
tcp: 17 discarded for bad checksums
udp: 3 bad checksums

If you look at Ethernet and FDDI specs and take the time to read up on the subject, you will gain great respect for the error checking that a good CRC offers. If you do the same with the IP checksum, you'll note that it’s an elegant checksum, but still just a checksum. Its goal was largely to provide a warning of software corruption of IP messages in end nodes and routers. If you look at netstat output long enough, especially if you work on weird problems, you'll be amazed at how much manages to evade the CRC check.

Every so often someone innocently posts a suggestion to comp.protocols.nfs that perhaps the IP checksum has outlived its usefulness. The last time someone did that, within a day engineers from most of the major Unix vendors posted very different answers for why maintaining the IP checksum is critical and other people posted accounts of NFS data corruption when the checksum on UDP messages was disabled. An interesting software engineering thesis would be to study these messages and try to uncover what happened to them.


ip: 3692 fragments dropped after timeout
tcp: 57176 data packets (56122576 bytes) retransmitted
tcp: 18979 completely duplicate packets (997570 bytes)
tcp: 14195 retransmit timeouts
These are hallmarks of messages lost in the net. While the NFS timeout rate (see below) is NFS specific, this is helpful because it shows that more than NFS is having trouble.

• ping server
  If NFS traffic between nodes on the same subnet is fine, but going across a router is not, then there is a very high probability that the router is dropping packets. Router statistics are generally available only to the high priest in charge of guarding the router, and he's probably too busy to help out. Ping is a useful check here. If you start up a ping that crosses a router, you should see the echo for each packet you send. If you don't, what does it tell you? First, you should have run ping between pairs of nodes on each subnet first to verify that no packets are lost in that case. You can also use tcpdump on both client and server to see if messages that reach the router show up on the other side. If they don't, then that's a very strong sign that the router is swamped or congested.
   
• nfsstat
  The server statistics aren't too useful, but the client RPC portion is. (nfsstat -cr will print just that.) For example:

alingo 26% nfsstat -cr
Client rpc:
tcp: calls      badxids    badverfs   timeouts   newcreds
     9          0          0          0          0         
     creates    connects   badconns   inputs     avails     interrupts
     2          2          0          20         9          0         

udp: calls      badxids    badverfs   timeouts   newcreds   retrans
     125955     3          0          342        0          0         
     badcalls   timers     waits
     343        229        0          

The key things are UDP calls and timeouts. A timeout rate of more than 1% generally results in awful performance. If badxids is incrementing, that usually means the server's file system is overloaded and duplicate replies are coming back because the client has retransmitted the original request.

• nfsd and nfsiod
  Nfsd must be running for NFS to work at all. The number of server threads you should have it run is a function of load and speed of the exported file system. However, even at the recommended value of 8, you should see good performance. Nfsiod is not needed to let NFS work, but it provides a big boost to client performance by helping to shepard multiple read and write requests at a time. Again, the recommended number of threads, 7, should provide decent performance for casual use.

Gigabit Ethernet came out at what should have been the perfect time – Tru64 could easily saturate 100 Mbs media with uniprocessor systems and a 10X increase would give us new headroom. Besides, there is a lot to be said for not saturating media. One big challenge was keeping up with the packet load, a 1500 byte packet takes only 12 usec of wire time. Some vendors created "Jumbo frames," a 9000 byte alternative that still only takes 72 usec. One reason that size was chosen was that it could hold an 8KB NFS I/O message.

When Gigabit hardware became available, Tru64 NFS only supported double buffered reads, i.e. when an application reads a file sequentially, NFS would send a read request before the application asked for the data. This performed poorly on SMP systems, as it didn't allow all the CPUs to be busy handling reads and had no chance to keep up with Gigabit loads. NFS was changed to issue two readaheads when the application made each read request, with an eight read ceiling. The first tests on Gigabit showed this let 4 CPU ES40s saturate Gigabit when reading cached files on the server. Unlike 10Base2 and FDDI, it wouldn't take years of hardware and software work to swamp the new medium.

Unfortunately, things aren't quite that simple. While those experiments were in 2001, It appears that few Gigabit switches can keep up. Customers and benchmarking people keep running into problems, but it's natural to look to the computers for the source of the problem instead of the infrastructure. Such “NFS problems” may be simply network congestion issues. Eventually switch vendors will increase their buffering, but in the meantime there is a lot of hardware that can't handle some rather simple configurations.

A Tru64 NFS client limits the network traffic it causes to one request per thread plus the number of nfsiod threads. The latter assist by doing read ahead and write behinds to allow the application requesting the I/O to return to user level. A similar thing is done with disk I/O, but NFS I/O is complicated by doing retransmits and whatnot, so clients generally have full fledged threads handing the work.

Tru64’s default number of nfsiod threads is 7, so a program reading or writing a file can have up to 8 requests outstanding at once. The standard I/O size of 48 KB means that there may be 384 KB out. 384 KB appears to be enough to swamp various infrastructures. 384 KB is 3,072 Kb - merely 3 msec of wire time. Consider two simple cases.

You can do a simple test to see how switches handle two fast streams flowing into a single stream. Consider one client reading files from two servers, with all three connected to a Gigabit switch. The switch will see 2 Gb/sec of data arriving and has to squeeze it out a 1 Gb/sec wire. Using separate programs (e.g. cp) to read the files, there will be no more than nine outstanding reads. (The client doesn't try to evenly allocate nfsiod threads to individual programs, while that might be a problem, it's a separate problem and shouldn't affect this at all.) If the result is less than what you got with just one reader, then you probably have a congestion problem. The client's "fragments dropped after timeout" counter will have incremented which says not all fragments of the read replies reached the client. Nfsstat on the client will report timeouts and retransmissions. The servers won't be at fault, as their load is less than in the single server case. If you can change switches to another vendor’s and get a different fragment loss, then that will be more evidence that the switches can’t handle the load. Another experiment that can be very useful is to reduce the number of nfsiod threads or remount the file systems with a mount option like –o rsize=16384 and see if performance improves.

One frustrating aspect of all this is that while network switches track a huge amount of statistics, expect to have trouble finding information about frames discarded due to congestion. This makes it very hard to understand what is going on without a lot more effort or discussions with the switch vendors.

In summary, if NFS seems to work okay as long as you don't read or write files, be sure to consider the infrastructure. Various things to do include

• On the client, verify that retransmits are happening (nfsstat -cr).
  Look under retrans for that.
• On the client (if reading) or server (if writing), see if there are any IP "fragments dropped after timeout".
  This is a very strong indication that the infrastructure is losing fragments.
• Experiment with various number of nfsiod threads or rsize mount option.
  You can kill and restart nfsiod on the fly. Change the argument to change the number of helper threads that are started. If performance jumps up when you decrease the number of threads below a certain point, that's a very clear sign you've crossed a congestion threshold.
• If you can enable “flow control” on your switches, that may greatly improve matters. That will throttle fast senders, so if a system is sending to both fast and slow receivers, the fast receiver may see throughput go down.

When NFS hangs it can be a challenge to figure out exactly why. Before diving into the kernel and hunting for hung threads, the very first thing to do is to go back to the beginning and check to see if anything works.

Assuming that everything else still works, then the important thing to do is to find the threads that are involved and get their stack traces. You already know how to find the user processes on the client, but there's more to look at and most people, even senior OS engineers, don't commonly know where to find it. Sometimes important culprits can be found as daemons or other user level code. You have to consider any process that might be reading or writing any file! If the system is low on memory, a system call may call code that flushes out NFS pages to make space for its own I/O.

The easiest way to find a non-kernel thread stuck in NFS I/O is to look for processes in the uninterruptible state. Of course, processes in U state happen all the time. Consider this from a mail hub:

There was no repeat, therefore none of these processes are hung. The most likely user process to hang is the update daemon which does a sync(2) every thirty seconds. If your system is running multithreaded applications, you may want to use ps axm | more and look for U flags.

Nfsiod and nfsd spawn several kernel threads, threads that belong to Pid 0 or its equivalent on a member. The rest of the time nfsiod and nfsd perform bookkeeping duties and are rarely involved in "NFS problems". While their kernel threads are the standard places to find hangs, be sure to check all the other kernel threads too, especially those involved with managing swapping, VM, and the UBC.

You can get a glimpse of the kernel threads via ps mlp 0 or this alternative:

NFS wchan names were selected to make it easier to pick out the NFS threads. When they are busy, they will not have those wchan names, but they will appear in the same line of the ps's output. That output is useful mainly to see if some threads are hung. It’s not easy to go from a ps line to the thread address, but it's easy to get a list of all kernel threads from the running system or a crash dump:

# dbx -k /vmunix
(dbx) set $pid=0
(dbx) tstack

On the client, the nfsiod program starts several kernel threads (part of Pid 0) that take over reads and writes to NFS files and makes sure they happen. This includes retransmitting requests when replies are not received in a timely fashion. Many times you can find that the application Pid is waiting for I/O to complete and that one or more nfsiod threads are doing the I/O but are waiting for a reply or for transmit done processing to complete. Unfortunately, both of these show similar stack traces:

dbx) set $pid=4362
(dbx) tstack

Thread 0xfffffc0014ebd8c0:
>  0 thread_block()_
   1 mpsleep(0x0, 0xfffffc000ee31400, 0xfffffc0014f32680,
             0x1004, 0x20000000001)
   2 clntkudp_callit_addr(h = 0xfffffc000ee31408, procnum = 7_
           

On the server, things are more variable. Usually one server thread is stuck deep inside file system code waiting for something to happen and all the other threads have identical stacks. These latter ones are not interesting, all they show is that the client has given up waiting for a reply and has retransmitted the request. The server has picked up the request and has called code that is waiting for the original request to finish and release whatever SMP or other locks its thread holds.

Again, look at the stacks for the kernel threads, the NFS server ones will stand out by their names and sizes. From the stack trace you can generally decide if the problem lies with the file system, VM, or NFS.

It's very easy to leap the hundreds of KB of kernel code to the conclusion that NFS is broken. This is due in part to several warning messages NFS prints on the console and user's terminals. It is also due in part to the importance of NFS in many environments. While HTTP may win out sometimes, many web servers return WWW pages that are fetched from NFS servers.

ASE V4.0x systems in particular are extremely heavy NFS consumers, and even access local file systems via NFS. Generally when an ASE system is having problems with NFS, people don't realize that the system is doing little but NFS. Unfortunately, the conclusion is that NFS is at fault when there are actually many possibilities.

Learn to analyze the many clues available to diagnose a problem. Once you learn them, you will find that

• You can often diagnose a problem in a few minutes.
   
• You will have enough evidence to convince your support group that you need their help.
   
• You will develop a reputation for bringing the right problem to the right personnel. When you do ask for help, people will immediately accept you may have a serious problem. Having good data can greatly decrease the time to solving the problem. Just a few minutes collecting data can save you days getting the problem resolved.

A final reminder: collect tcpdump traces, netstat output, tcpdump traces, appropriate stack traces, and good tcpdump traces.