Cyberithub

10 Useful nfsstat and nfsiostat examples to troubleshoot NFS Performance in Linux

In this article, I will take you through the usage of nfsstat and nfsiostat to troubleshoot NFS Performance in Linux. If you are using NFS Server in your Organization then you might be aware of the issue NFS Server could have due to the usage of existing Network Infrastructure. Any slight disruption in the network could affect the NFS Performance adversely.

We have two very important tools which can provide you the useful information about NFS mount points and its performance from both Server and Client end. nfsstat and nfsiostat are the two famous NFS dedicated tools which can be used for this purpose. I will explain more about the usage of nfsstat and nfsiostat command to troubleshoot the NFS Performance issues in below sections.

10 Useful nfsstat and nfsiostat examples to troubleshoot NFS Performance in Linux 1

nfsstat and nfsiostat examples to troubleshoot NFS Performance in Linux

Also Read: 10 Popular nfsstat command examples in Linux for Professionals (Cheatsheet)

Troubleshooting Example 1: How to Check NFS Client Statistics using nfsstat command

If you want to check NFS Client statistics then you need to use nfsstat -c command as shown below. If you want to troubleshoot the network overload then you need to probably look at the retransmission rate(retrans). RPC Calls are synchronous and hence will block the client until it completes the request. If the client does not get a response from NFS Server, it will keep retransmitting the same request to a specific number of times until it finally quits.

Increase in retransmission can happen due to various reasons. It could be due to increase Server overloading which might delay the Server response, due to network congestion which further slow down the transmission, due to network card performance which might result in packet loss, due to disturbance in the network medium or may be due to faulty network devices between Server and Client. All these factors needs to be checked and verified.

[root@localhost ~]# nfsstat -c
Client rpc stats:
calls retrans authrefrsh
23       0       23

Client nfs v4:
null             read        write        commit       open         open_conf
0         0%  0        0%  0        0%  0        0%  0         0%  0      0%
open_noat     open_dgrd    close        setattr       fsinfo         renew
0         0%  0        0%  0        0%  0        0%  3         13% 0       0%
setclntid      confirm      lock        lockt         locku          access
0         0%  0        0%  0        0%  0        0%  0         0%  2       9%
getattr         lookup     lookup_root  remove        rename        link
2         9%  3        13% 1        4%  0        0%  0         0%  0       0%
symlink        create      pathconf     statfs       readlink      readdir
0         0%  0        0%  2        9%  0        0%  0         0%  0       0%
server_caps   delegreturn  getacl       setacl       fs_locations  rel_lkowner
5         22% 0        0% 0         0%  0        0%  0         0%  0       0%
secinfo       exchange_id create_ses    destroy_ses  sequence      get_lease_t
0         0%  0        0% 1         4%  1         4% 0         0%  0       0%
reclaim_comp  layoutget   getdevinfo    layoutcommit layoutreturn  getdevlist
0         0%  1        4%  0        0%  0         0% 0         0%  0       0%
(null)
1 4%

-c : Print only client-side statistics. Check nfsstat command Man Page for more info.

NOTE:

Please note that here I am using root user to run all the below commands.You can use any user with sudo access to run all these commands. For more information Please check Step by Step: How to Add User to Sudoers to provide sudo access to the User.

Troubleshooting Example 2: How to Check NFS Server Statistics using nfsstat command 

If you want to check NFS Server statistics then you need to use nfsstat -s command as shown below. You can use this command to troubleshoot the NFS Server latency issue. Here you need to look at the badcalls value if it goes beyond 0. badcalls value provides the information about number of rejected calls by the RPC Layer. More the value is more will be the latency.

root@localhost:~# nfsstat -s
Server rpc stats:
calls badcalls badfmt badauth badclnt
23       0        0      0       0

Server nfs v4:
null compound
1 4% 22 95%

Server nfs v4 operations:
op0-unused     op1-unused    op2-future      access       close
0         0% 0           0% 0          0%  2         2%  0         0%
commit          create       delegpurge     delegreturn   getattr
0         0% 0           0% 0          0%  0         0%  18        25%
  getfh           link         lock          lockt         locku
4         5% 0           0% 0          0%  0         0%  0         0%
lookup         lookup_root    nverify        open         openattr
3         4% 0           0% 0          0%  0         0%  0         0%
open_conf     open_dgrd        putfh        putpubfh     putrootfh
0         0% 0           0% 17         24% 0         0%  2         2%
read           readdir       readlink       remove         rename
0         0% 0           0% 0          0%  0         0%  0         0%
renew         restorefh       savefh        secinfo       setattr
0         0% 0           0% 0          0%  0         0%  0         0%
setcltid     setcltidconf     verify        write         rellockowner
0         0% 0           0% 0          0%  0         0%  0         0%
bc_ctl       bind_conn      exchange_id    create_ses    destroy_ses
0         0% 0           0% 1          1%  1         1%  0         0%
free_stateid getdirdeleg    getdevinfo     getdevlist    layoutcommit
0         0% 0           0% 0          0%  0         0%  0         0%
layoutget    layoutreturn   secinfononam   sequence       set_ssv
0         0% 0           0% 1          1%  20        28% 0         0%
test_stateid want_deleg     destroy_clid   reclaim_comp  allocate
0         0% 0           0% 0          0%  1         1%  0         0%
copy         copy_notify    deallocate     ioadvise      layouterror
0         0%  0          0% 0          0%  0         0%  0         0%
layoutstats  offloadcancel  offloadstatus  readplus        seek
0         0%  0          0% 0          0%  0         0%  0         0%
write_same
0         0%

-s : Print only server-side statistics. Check nfsstat command Man Page for more info.

Troubleshooting Example 3: How to check iostat for NFS mount point using nfsiostat command 

If you want to check iostat for NFS mount point then you need to use nfsiostat command in Linux as shown below. nfsiostat command will get its input from /proc/self/mountstats and provide the information about the input output performance of NFS mount points. Below are the output parameters from nfsstat command output.

  • op/s : This is the number of operations per second.
  • rpc bklog : This is the length of the backlog queue.
  • kB/s: This is the number of kB written/read per second.
  • kB/op : This is the number of kB written/read per each operation.
  • retrans : This is the number of retransmissions.
  • avg RTT (ms) : This is the duration from the time that client’s kernel sends the RPC request until the time it receives the reply.
  • avg exe (ms) :This is the duration from the time that NFS client does the RPC request to its kernel until the RPC request is completed, this includes the RTT time above. More can be checked on nfsiostat Man Page.
[root@localhost ~]# nfsiostat

192.168.0.108:/mnt/nfs mounted on /nfsclient:

op/s         rpc bklog
  0.02  0.00
read: ops/s kB/s kB/op retrans avg RTT (ms) avg exe (ms)
        0.000 0.000 0.000 0 (0.0%) 0.000 0.000
write: ops/s kB/s kB/op retrans avg RTT (ms) avg exe (ms)
        0.000 0.000 0.000 0 (0.0%) 0.000 0.000

Troubleshooting Example 4: How to Display Attribute Cache Statistics using nfsiostat command

If you want to display attribute cache statistics then you need to use nfsiostat -a command as shown below.

[root@localhost ~]# nfsiostat -a

192.168.0.108:/mnt/nfs mounted on /nfsclient:

op/s         rpc bklog
   0.02 0.00
getattr: ops/s kB/s kB/op retrans avg RTT (ms) avg exe (ms)
           0.000 0.000 0.422 0 (0.0%) 5.500 11.000
access: ops/s kB/s kB/op retrans avg RTT (ms) avg exe (ms)
           0.000 0.000 0.355 0 (0.0%) 0.000 0.000

1 inode revalidations, hitting in cache -100.00% of the time
0 open operations (mandatory GETATTR requests)

-a : displays statistics related to the attribute cache. Check nfsiostat Man Page for more info.

Troubleshooting Example 5: How to Sort NFS Mount Point by ops/sec using nfsiostat command

If you want to sort NFS mount point by ops/sec then you need to use nfsiostat -s command as shown below.

[root@localhost ~]# nfsiostat -s

192.168.0.108:/mnt/nfs mounted on /nfsclient:

op/s           rpc  bklog
  1.26   0.00
read: ops/s          kB/s    kB/op     retrans avg RTT (ms) avg exe (ms)
        0.000 0.000 0.000 0 (0.0%) 0.000 0.000
write: ops/s         kB/s    kB/op     retrans avg RTT (ms) avg exe (ms)
        0.000 0.000 0.000 0 (0.0%) 0.000 0.000

-s : Sort NFS mount points by ops/second. Check nfsiostat Man Page for more info.

Troubleshooting Example 6: How to display page cache related Statistics using nfsiostat command

If you want to display all the page cache related statistics then you need to use nfsiostat -p command as shown below.

[root@localhost ~]# nfsiostat -p

192.168.0.108:/mnt/nfs mounted on /nfsclient:

op/s        rpc bklog
  0.02 0.00
read: ops/s  kB/s kB/op retrans avg RTT (ms) avg exe (ms)
        0.000 0.000 0.000 0 (0.0%) 0.000 0.000
write: ops/s kB/s kB/op retrans avg RTT (ms) avg exe (ms)
        0.000 0.000 0.000 0 (0.0%) 0.000 0.000

0 nfs_readpage() calls read 0 pages
0 nfs_readpages() calls read 0 pages


0 nfs_updatepage() calls
0 nfs_writepage() calls wrote 0 pages
0 nfs_writepages() calls wrote 0 pages

-p : displays statistics related to the page cache. Check nfsiostat Man Page for more info.

Troubleshooting Example 7: How to Display Statistics related to directory operations

If you want to display directory related statistics then you need to use nfsiostat -d command as shown below.

[root@localhost ~]# nfsiostat -d

192.168.0.108:/mnt/nfs mounted on /nfsclient:

op/s        rpc bklog
  0.02 0.00
lookup: ops/s kB/s kB/op retrans avg RTT (ms) avg exe (ms)
         0.000 0.000 0.488 0 (0.0%) 0.000 0.000
readdir: ops/s kB/s kB/op retrans avg RTT (ms) avg exe (ms)
         0.000 0.000 0.000 0 (0.0%) 0.000 0.000

0 open operations (pathname lookups)
0 dentry revalidates and 1 vfs lookup requests
resulted in 1 LOOKUPs on the wire
0 vfs getdents calls resulted in 0 READDIRs on the wire

-d : displays statistics related to directory operations. Check nfsiostat Man Page for more info.

Troubleshooting Example 8: How to display nfsiostat output at a certain Interval

If you want to display nfsiostat command output 2 times in every 3 seconds then you need to use nfsiostat 3 2 command as shown below. You can change the interval and number of output that you require as per your requirements.

[root@localhost ~]# nfsiostat 3 2

192.168.0.108:/mnt/nfs mounted on /nfsclient:

op/s rpc bklog
0.02 0.00
read: ops/s kB/s kB/op retrans avg RTT (ms) avg exe (ms)
0.000 0.000 0.000 0 (0.0%) 0.000 0.000
write: ops/s kB/s kB/op retrans avg RTT (ms) avg exe (ms)
0.000 0.000 0.000 0 (0.0%) 0.000 0.000

192.168.0.108:/mnt/nfs mounted on /nfsclient:

op/s rpc bklog
0.00 0.00
read: ops/s kB/s kB/op retrans avg RTT (ms) avg exe (ms)
0.000 0.000 0.000 0 (0.0%) 0.000 0.000
write: ops/s kB/s kB/op retrans avg RTT (ms) avg exe (ms)
0.000 0.000 0.000 0 (0.0%) 0.000 0.000

Troubleshooting Example 9: How to Display nfsiostat output for a Specific Partition

Above example will display the nfsiostat output for all the mounted NFS partition but if you want to see the statistics of a specific partition then you need to use below nfsiostat command. In this example, we are checking stats of /nfsclient mount point using nfsiostat 5 2 /nfsclient command. This command will display output 2 times with an interval of 5 seconds.

[root@localhost ~]# nfsiostat 5 2 /nfsclient

192.168.0.108:/mnt/nfs mounted on /nfsclient:

op/s rpc bklog
0.02 0.00
read: ops/s kB/s kB/op retrans avg RTT (ms) avg exe (ms)
0.000 0.000 0.000 0 (0.0%) 0.000 0.000
write: ops/s kB/s kB/op retrans avg RTT (ms) avg exe (ms)
0.000 0.000 0.000 0 (0.0%) 0.000 0.000

192.168.0.108:/mnt/nfs mounted on /nfsclient:

op/s        rpc bklog
   0.00 0.00
read: ops/s kB/s kB/op retrans avg RTT (ms) avg exe (ms)
        0.000 0.000 0.000 0 (0.0%) 0.000 0.000
write: ops/s kB/s kB/op retrans avg RTT (ms) avg exe (ms)
        0.000 0.000 0.000 0 (0.0%) 0.000 0.000

Troubleshooting Example 10: How to check RPC Call related information using nfsstat command

If you want to only check the RPC Call related information and ignore others stats then you can simple use the nfsstat -r command as shown below.

[root@localhost ~]# nfsstat -r
Client rpc stats:
calls retrans authrefrsh
331      2       331

-r : Print only RPC statistics.

 

Popular Recommendations:-

How to Install VLC Media Player in RHEL / CentOS 8 Using 6 Easy Steps

52 Useful cut command in Linux/Unix with Examples for Beginners

How to Delete Elasticsearch Unassigned Shards in 4 Easy Steps

How to Delete Elasticsearch Red Status Indices in 3 Easy Steps

Understanding Kafka Console Producer and Consumer in 10 Easy Steps

Popular Apache Kafka Architecture Explained Using 4 Basic Components

17 Useful Apache Kafka Examples on Linux (RedHat/CentOS 7/8)

How to Install Apache Kafka on Ubuntu 18.04

How to Install Apache Zookeeper on Ubuntu 18.04

Leave a Comment