PNFS information on cmspnfs1.fnal.gov

Page generated: Sat Jul 5 12:17:02 CDT 2008



TOP sources of PNFS traffic in last 60 seconds:
0% 	 cmssrv41-phedex-production 14
0% 	 cmssrv45-phedex-ddt 35
0% 	 cmswn 77
4% 	 localhost-PnfsManager 36019
25% 	 cmssrv28-monitoring 211129
70% 	 cmsstor 578428
825702 PNFS tcpdump responses or 13761/sec

TOP worker node source of PNFS traffic in last 60 seconds:
0% 	 cmswn993 1
0% 	 cmswn994 1
0% 	 cmswn945 2
0% 	 cmswn960 2
0% 	 cmswn977 2

TOP storage node source of PNFS traffic in last 60 seconds:
12% 	 cmsstor96 99127
12% 	 cmsstor97 102372
12% 	 cmsstor95 102745
12% 	 cmsstor98 105004
13% 	 cmsstor99 110840

---------------------------------------------------------------------------------------------------------------------------

PNFS statistics during last 60 seconds:
0% 	 null 0
82% 	 getattr 42778
0% 	 setattr 66
0% 	 root 0
10% 	 lookup 5562
0% 	 readlink 0
6% 	 read 3107
0% 	 wrcache 0
0% 	 write 66
0% 	 create 90
0% 	 remove 8
0% 	 rename 0
0% 	 link 0
0% 	 symlink 0
0% 	 mkdir 0
0% 	 rmdir 0
0% 	 readdir 0
0% 	 fsstat 0
51677 PNFS operations or 861/sec

---------------------------------------------------------------------------------------------------------------------------

UPTIME:
 12:18:20 up 93 days,  3:21,  1 user,  load average: 1.62, 0.86, 0.63

---------------------------------------------------------------------------------------------------------------------------

VMSTAT 60 2:
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 4  0   1292 759540 167376 30434184    0    0    34    80    0     1  2  3 94  1
 6  0   1292 752388 167376 30436564    0    0     1   428 11248 132557  5 11 84  0

---------------------------------------------------------------------------------------------------------------------------

atop -c 60 2:

ATOP - cmspnfs1           2008/07/05  12:18:02               60 seconds elapsed
PRC | sys  64.88s | user  31.19s | #proc    216 | #zombie    0 | #exit     98 |
CPU | sys     90% | user     52% | irq      33% | idle    624% | wait      1% |
CPL | avg1   1.70 | avg5    0.83 | avg15   0.62 | csw 10959688 | intr  774807 |
MEM | tot   31.4G | free  730.5M | cache  29.0G | buff  163.5M | slab    1.0G |
SWP | tot   64.0G | free   64.0G |              | vmcom   2.0G | vmlim  79.7G |
NET | transport   | tcpi    3324 | tcpo    3595 | udpi  896255 | udpo  896332 |
NET | network     | ipi   899662 | ipo   899882 | ipfrw      0 | deliv 899580 |
NET | eth0     1% | pcki  781785 | pcko  781399 | si   19 Mbps | so   15 Mbps |
NET | lo     ---- | pcki  107222 | pcko  107222 | si 2835 Kbps | so 2835 Kbps |

  PID  CPU COMMAND-LINE                                                  1/1   
 3709  25% ./pnfsd
 3710  25% ./pnfsd
 3712  25% ./pnfsd
 3711  24% ./pnfsd
18362  20% /usr/sbin/tcpdump -i eth0 port 2049 and udp
 3652   9% ./dbserver cms11
 3651   8% postgres: enstore cms11 [local] idle                   
 3624   6% ./dbserver admin
  918   5% /opt/pnfs/tools/linux/dbserver cms
27528   4% /usr/java/jdk1.6.0_06/bin/java -server -Xmx512m -XX:MaxDirectMemoryS
 3672   2% ./dbserver resilient
18365   2% /usr/sbin/tcpdump -i lo port 2049 and udp
18339   1% grep IP cmspnfs1.fnal.gov.nfs
18340   1% cut -f2 -d>
 3520   1% postgres: stats buffer process                         
18341   1% cut -f1 -d.

---------------------------------------------------------------------------------------------------------------------------

system tap: cstap_bnh_2.stp

---------------------------------------------------------------------------------------------------------------------------

ipcs

------ Shared Memory Segments --------
key        shmid      owner      perms      bytes      nattch     status      
0x0052e2c1 0          enstore   600        1671258112 38                      
0x00001122 32769      root      600        984        26                      
0x00000000 65538      root      700        12288      2                       
0x00000000 98307      root      700        12288      22                      
0x00000000 131076     root      700        12288      22                      
0x00000000 163845     root      700        12288      22                      
0x00000000 196614     root      700        12288      22                      
0x7a01821b 950279     zabbix    666        1099804    0                       
0x00000000 1933320    root      700        8192       1          dest         
0x00000000 1277962    root      700        8192       1          dest         
0x00000000 1310731    root      700        8192       1          dest         
0x00000000 1343500    root      700        8192       1          dest         
0x00000000 1376269    root      700        8192       1          dest         
0x00000000 1409038    root      700        8192       1          dest         
0x00000000 1441807    root      700        8192       1          dest         
0x00000000 1474576    root      700        8192       1          dest         
0x00000000 1900561    root      700        8192       1          dest         
0x00000000 1540114    root      700        8192       1          dest         
0x00000000 1605651    root      700        8192       1          dest         
0x7a0185c9 1114132    zabbix    666        1099804    0                       
0x00000000 1638421    root      700        8192       1          dest         
0x00000000 1703958    root      700        8192       1          dest         
0x00000000 1736727    root      700        8192       1          dest         
0x00000000 1769496    root      700        8192       1          dest         
0x00000000 1802265    root      700        8192       1          dest         
0x00000000 1835034    root      700        8192       1          dest         
0x00000000 1867803    root      700        8192       1          dest         
0x00000000 1572892    root      700        8192       1          dest         
0x7a018411 1146909    zabbix    666        1099804    6                       
0x00000000 1507358    root      700        8192       1          dest         
0x00000000 1671199    root      700        8192       1          dest         

------ Semaphore Arrays --------
key        semid      owner      perms      nsems     
0x0052e2c1 0          enstore   600        17        
0x0052e2c2 32769      enstore   600        17        
0x0052e2c3 65538      enstore   600        17        
0x0052e2c4 98307      enstore   600        17        
0x0052e2c5 131076     enstore   600        17        
0x0052e2c6 163845     enstore   600        17        
0x0052e2c7 196614     enstore   600        17        
0x0052e2c8 229383     enstore   600        17        
0x0052e2c9 262152     enstore   600        17        
0x0052e2ca 294921     enstore   600        17        
0x0052e2cb 327690     enstore   600        17        
0x0052e2cc 360459     enstore   600        17        
0x0052e2cd 393228     enstore   600        17        
0x00000000 425997     root      600        1         
0x00000000 458766     root      700        2         
0x00000000 491535     root      700        2         
0x00000000 524304     root      700        2         
0x00000000 557073     root      700        2         
0x00000000 589842     root      700        2         
0x00000000 622611     root      700        2         
0x00000000 655380     root      700        2         
0x00000000 688149     root      700        2         
0x00000000 720918     root      700        2         
0x00000000 2261015    root      700        2         
0x00000000 786456     root      700        2         
0x00000000 819225     root      700        2         
0x00000000 851994     root      700        2         
0x00000000 884763     root      700        2         
0x00000000 917532     root      700        2         
0x00000000 950301     root      700        2         
0x00000000 983070     root      700        2         
0x00000000 1015839    root      700        2         
0x00000000 1048608    root      700        2         
0x00000000 1081377    root      700        2         
0x00000000 1114146    root      700        1         
0x00000000 1146915    root      700        1         
0x00000000 1179684    root      700        1         
0x00000000 1212453    root      700        1         
0x00000000 1245222    root      700        1         
0x7a01821b 2031655    zabbix    666        1         
0x002fa327 1966120    root      666        2         
0x7a0185c9 2195497    zabbix    666        1         
0x7a018411 2228266    zabbix    666        1         
0x00000000 2293803    root      700        2         

------ Message Queues --------
key        msqid      owner      perms      used-bytes   messages    


---------------------------------------------------------------------------------------------------------------------------

time df /pnfs/fs:
Filesystem           1K-blocks      Used Available Use% Mounted on
localhost:/fs           400000     80000    284000  22% /pnfs/fs

real	0m0.001s
user	0m0.000s
sys	0m0.000s

---------------------------------------------------------------------------------------------------------------------------

PNFSMANAGER INFO:
In 78 sec 1332 transactions  = 17.076923 trans/sec


    dCache Admin (VII) (user=enstore)


[cmssrv32.fnal.gov] (local) enstore > cd PnfsManager
[cmssrv32.fnal.gov] (PnfsManager) enstore > info
$Revision: 8854 $
NameSpace Provider: 
$Revision: 8542 $
 Server         : fnal.gov(cmspnfs1.fnal.gov)
 RealMountPoint : 000000000000000000001040 /pnfs/fs
       VirtualMountId : 000000000000000000001040
      VirtualPnfsPath : /root/fs
     VirtualLocalPath : /pnfs/fs
    VirtualGlobalPath : /pnfs/fnal.gov

CacheLocation Provider: 
$Id: SQLNameSpaceProvider.java,v 1.19 2007-08-22 12:24:38 tigran Exp $
StorageInfo Provider: 
$Revision: 8542 $
 Server         : fnal.gov(cmspnfs1.fnal.gov)
 RealMountPoint : 000000000000000000001040 /pnfs/fs
       VirtualMountId : 000000000000000000001040
      VirtualPnfsPath : /root/fs
     VirtualLocalPath : /pnfs/fs
    VirtualGlobalPath : /pnfs/fnal.gov

 Threads (4) Queue
    [0] 0
    [1] 0
    [2] 0
    [3] 0
Statistics:
addCacheLocation                   1868664         0
clearCacheLocation                 2086482         0
getCacheLocations                  6432507         0
createDirectory                      24722        48
createEntry                         908282       478
deleteEntry                         238638       897
getStorageInfo                     8850461   1318754
setStorageInfo                           1         0
setLength                           898335      2358
getCacheStatistics                       0         0
updateCacheStatistics                    0         0
rename                                   0         0
setLevelData                             0         0
fileFlushed                         177842         0
mapPath2Id                          398141        10
mapId2Path                               0         0
getMetadataInfo                    7958503         0
setMetadataInfo                          0         0
[cmssrv32.fnal.gov] (PnfsManager) enstore > ..
[cmssrv32.fnal.gov] (local) enstore > 
[cmssrv32.fnal.gov] (local) enstore > logoff
dmg.util.CommandExitException: (0) Done
[cmssrv32.fnal.gov] (local) enstore > 



Legal Notices