Page generated: Sat Jul 5 12:17:02 CDT 2008
TOP sources of PNFS traffic in last 60 seconds:
0% cmssrv41-phedex-production 14
0% cmssrv45-phedex-ddt 35
0% cmswn 77
4% localhost-PnfsManager 36019
25% cmssrv28-monitoring 211129
70% cmsstor 578428
825702 PNFS tcpdump responses or 13761/sec
TOP worker node source of PNFS traffic in last 60 seconds:
0% cmswn993 1
0% cmswn994 1
0% cmswn945 2
0% cmswn960 2
0% cmswn977 2
TOP storage node source of PNFS traffic in last 60 seconds:
12% cmsstor96 99127
12% cmsstor97 102372
12% cmsstor95 102745
12% cmsstor98 105004
13% cmsstor99 110840
---------------------------------------------------------------------------------------------------------------------------
PNFS statistics during last 60 seconds:
0% null 0
82% getattr 42778
0% setattr 66
0% root 0
10% lookup 5562
0% readlink 0
6% read 3107
0% wrcache 0
0% write 66
0% create 90
0% remove 8
0% rename 0
0% link 0
0% symlink 0
0% mkdir 0
0% rmdir 0
0% readdir 0
0% fsstat 0
51677 PNFS operations or 861/sec
---------------------------------------------------------------------------------------------------------------------------
UPTIME:
12:18:20 up 93 days, 3:21, 1 user, load average: 1.62, 0.86, 0.63
---------------------------------------------------------------------------------------------------------------------------
VMSTAT 60 2:
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
4 0 1292 759540 167376 30434184 0 0 34 80 0 1 2 3 94 1
6 0 1292 752388 167376 30436564 0 0 1 428 11248 132557 5 11 84 0
---------------------------------------------------------------------------------------------------------------------------
atop -c 60 2:
ATOP - cmspnfs1 2008/07/05 12:18:02 60 seconds elapsed
PRC | sys 64.88s | user 31.19s | #proc 216 | #zombie 0 | #exit 98 |
CPU | sys 90% | user 52% | irq 33% | idle 624% | wait 1% |
CPL | avg1 1.70 | avg5 0.83 | avg15 0.62 | csw 10959688 | intr 774807 |
MEM | tot 31.4G | free 730.5M | cache 29.0G | buff 163.5M | slab 1.0G |
SWP | tot 64.0G | free 64.0G | | vmcom 2.0G | vmlim 79.7G |
NET | transport | tcpi 3324 | tcpo 3595 | udpi 896255 | udpo 896332 |
NET | network | ipi 899662 | ipo 899882 | ipfrw 0 | deliv 899580 |
NET | eth0 1% | pcki 781785 | pcko 781399 | si 19 Mbps | so 15 Mbps |
NET | lo ---- | pcki 107222 | pcko 107222 | si 2835 Kbps | so 2835 Kbps |
PID CPU COMMAND-LINE 1/1
3709 25% ./pnfsd
3710 25% ./pnfsd
3712 25% ./pnfsd
3711 24% ./pnfsd
18362 20% /usr/sbin/tcpdump -i eth0 port 2049 and udp
3652 9% ./dbserver cms11
3651 8% postgres: enstore cms11 [local] idle
3624 6% ./dbserver admin
918 5% /opt/pnfs/tools/linux/dbserver cms
27528 4% /usr/java/jdk1.6.0_06/bin/java -server -Xmx512m -XX:MaxDirectMemoryS
3672 2% ./dbserver resilient
18365 2% /usr/sbin/tcpdump -i lo port 2049 and udp
18339 1% grep IP cmspnfs1.fnal.gov.nfs
18340 1% cut -f2 -d>
3520 1% postgres: stats buffer process
18341 1% cut -f1 -d.
---------------------------------------------------------------------------------------------------------------------------
system tap: cstap_bnh_2.stp
---------------------------------------------------------------------------------------------------------------------------
ipcs
------ Shared Memory Segments --------
key shmid owner perms bytes nattch status
0x0052e2c1 0 enstore 600 1671258112 38
0x00001122 32769 root 600 984 26
0x00000000 65538 root 700 12288 2
0x00000000 98307 root 700 12288 22
0x00000000 131076 root 700 12288 22
0x00000000 163845 root 700 12288 22
0x00000000 196614 root 700 12288 22
0x7a01821b 950279 zabbix 666 1099804 0
0x00000000 1933320 root 700 8192 1 dest
0x00000000 1277962 root 700 8192 1 dest
0x00000000 1310731 root 700 8192 1 dest
0x00000000 1343500 root 700 8192 1 dest
0x00000000 1376269 root 700 8192 1 dest
0x00000000 1409038 root 700 8192 1 dest
0x00000000 1441807 root 700 8192 1 dest
0x00000000 1474576 root 700 8192 1 dest
0x00000000 1900561 root 700 8192 1 dest
0x00000000 1540114 root 700 8192 1 dest
0x00000000 1605651 root 700 8192 1 dest
0x7a0185c9 1114132 zabbix 666 1099804 0
0x00000000 1638421 root 700 8192 1 dest
0x00000000 1703958 root 700 8192 1 dest
0x00000000 1736727 root 700 8192 1 dest
0x00000000 1769496 root 700 8192 1 dest
0x00000000 1802265 root 700 8192 1 dest
0x00000000 1835034 root 700 8192 1 dest
0x00000000 1867803 root 700 8192 1 dest
0x00000000 1572892 root 700 8192 1 dest
0x7a018411 1146909 zabbix 666 1099804 6
0x00000000 1507358 root 700 8192 1 dest
0x00000000 1671199 root 700 8192 1 dest
------ Semaphore Arrays --------
key semid owner perms nsems
0x0052e2c1 0 enstore 600 17
0x0052e2c2 32769 enstore 600 17
0x0052e2c3 65538 enstore 600 17
0x0052e2c4 98307 enstore 600 17
0x0052e2c5 131076 enstore 600 17
0x0052e2c6 163845 enstore 600 17
0x0052e2c7 196614 enstore 600 17
0x0052e2c8 229383 enstore 600 17
0x0052e2c9 262152 enstore 600 17
0x0052e2ca 294921 enstore 600 17
0x0052e2cb 327690 enstore 600 17
0x0052e2cc 360459 enstore 600 17
0x0052e2cd 393228 enstore 600 17
0x00000000 425997 root 600 1
0x00000000 458766 root 700 2
0x00000000 491535 root 700 2
0x00000000 524304 root 700 2
0x00000000 557073 root 700 2
0x00000000 589842 root 700 2
0x00000000 622611 root 700 2
0x00000000 655380 root 700 2
0x00000000 688149 root 700 2
0x00000000 720918 root 700 2
0x00000000 2261015 root 700 2
0x00000000 786456 root 700 2
0x00000000 819225 root 700 2
0x00000000 851994 root 700 2
0x00000000 884763 root 700 2
0x00000000 917532 root 700 2
0x00000000 950301 root 700 2
0x00000000 983070 root 700 2
0x00000000 1015839 root 700 2
0x00000000 1048608 root 700 2
0x00000000 1081377 root 700 2
0x00000000 1114146 root 700 1
0x00000000 1146915 root 700 1
0x00000000 1179684 root 700 1
0x00000000 1212453 root 700 1
0x00000000 1245222 root 700 1
0x7a01821b 2031655 zabbix 666 1
0x002fa327 1966120 root 666 2
0x7a0185c9 2195497 zabbix 666 1
0x7a018411 2228266 zabbix 666 1
0x00000000 2293803 root 700 2
------ Message Queues --------
key msqid owner perms used-bytes messages
---------------------------------------------------------------------------------------------------------------------------
time df /pnfs/fs:
Filesystem 1K-blocks Used Available Use% Mounted on
localhost:/fs 400000 80000 284000 22% /pnfs/fs
real 0m0.001s
user 0m0.000s
sys 0m0.000s
---------------------------------------------------------------------------------------------------------------------------
PNFSMANAGER INFO:
In 78 sec 1332 transactions = 17.076923 trans/sec
dCache Admin (VII) (user=enstore)
[cmssrv32.fnal.gov] (local) enstore > cd PnfsManager
[cmssrv32.fnal.gov] (PnfsManager) enstore > info
$Revision: 8854 $
NameSpace Provider:
$Revision: 8542 $
Server : fnal.gov(cmspnfs1.fnal.gov)
RealMountPoint : 000000000000000000001040 /pnfs/fs
VirtualMountId : 000000000000000000001040
VirtualPnfsPath : /root/fs
VirtualLocalPath : /pnfs/fs
VirtualGlobalPath : /pnfs/fnal.gov
CacheLocation Provider:
$Id: SQLNameSpaceProvider.java,v 1.19 2007-08-22 12:24:38 tigran Exp $
StorageInfo Provider:
$Revision: 8542 $
Server : fnal.gov(cmspnfs1.fnal.gov)
RealMountPoint : 000000000000000000001040 /pnfs/fs
VirtualMountId : 000000000000000000001040
VirtualPnfsPath : /root/fs
VirtualLocalPath : /pnfs/fs
VirtualGlobalPath : /pnfs/fnal.gov
Threads (4) Queue
[0] 0
[1] 0
[2] 0
[3] 0
Statistics:
addCacheLocation 1868664 0
clearCacheLocation 2086482 0
getCacheLocations 6432507 0
createDirectory 24722 48
createEntry 908282 478
deleteEntry 238638 897
getStorageInfo 8850461 1318754
setStorageInfo 1 0
setLength 898335 2358
getCacheStatistics 0 0
updateCacheStatistics 0 0
rename 0 0
setLevelData 0 0
fileFlushed 177842 0
mapPath2Id 398141 10
mapId2Path 0 0
getMetadataInfo 7958503 0
setMetadataInfo 0 0
[cmssrv32.fnal.gov] (PnfsManager) enstore > ..
[cmssrv32.fnal.gov] (local) enstore >
[cmssrv32.fnal.gov] (local) enstore > logoff
dmg.util.CommandExitException: (0) Done
[cmssrv32.fnal.gov] (local) enstore >