Page generated: Sat Nov 7 05:13:15 CST 2009
TOP sources of PNFS traffic in last 60 seconds:
0% cmssrv45-phedex-ddt 1116
0% cmswn 2540
2% cmssrv33 7996
4% cmsstor 13316
16% localhost-PnfsManager 52365
75% cmssrv28-monitoring 232111
309444 PNFS tcpdump responses or 5157/sec
TOP worker node source of PNFS traffic in last 60 seconds:
0% cmswn574 20
0% cmswn579 20
0% cmswn731 20
0% cmswn764 20
0% cmswn800 20
TOP storage node source of PNFS traffic in last 60 seconds:
0% cmsstor99 293
0% cmsstor149 409
0% cmsstor66 420
0% cmsstor95 422
0% cmsstor98 422
---------------------------------------------------------------------------------------------------------------------------
PNFS statistics during last 60 seconds:
0% null 0
95% getattr 60063
0% setattr 3
0% root 0
0% lookup 86
0% readlink 0
4% read 2812
0% wrcache 0
0% write 3
0% create 3
0% remove 0
0% rename 0
0% link 0
0% symlink 0
0% mkdir 0
0% rmdir 0
0% readdir 0
0% fsstat 0
62970 PNFS operations or 1049/sec
---------------------------------------------------------------------------------------------------------------------------
UPTIME:
05:14:20 up 36 days, 14:42, 1 user, load average: 0.89, 1.10, 0.86
---------------------------------------------------------------------------------------------------------------------------
VMSTAT 60 2:
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free buff cache si so bi bo in cs us sy id wa st
5 0 0 2153044 186796 29107912 0 0 142 214 1 1 4 8 87 1 0
3 0 0 2142844 186960 29111596 0 0 2 850 11392 65915 6 10 83 0 0
---------------------------------------------------------------------------------------------------------------------------
atop -c 60 2:
ATOP - cmspnfs1 2009/11/07 05:14:15 60 seconds elapsed
PRC | sys 31.33s | user 21.61s | #proc 330 | #zombie 1 | #exit 13937 |
CPU | sys 65% | user 46% | irq 21% | idle 666% | wait 2% |
CPL | avg1 0.71 | avg5 1.07 | avg15 0.85 | csw 3944268 | intr 679928 |
MEM | tot 31.4G | free 2.0G | cache 27.8G | buff 182.7M | slab 688.3M |
SWP | tot 64.0G | free 64.0G | | vmcom 3.0G | vmlim 64.6G |
DSK | sda | busy 1% | read 0 | write 2973 | avio 0 ms |
NET | transport | tcpi 31813 | tcpo 16903 | udpi 364493 | udpo 364575 |
NET | network | ipi 404539 | ipo 381481 | ipfrw 0 | deliv 402060 |
NET | eth0 1% | pcki 300775 | pcko 282119 | si 11 Mbps | so 6539 Kbps |
NET | lo ---- | pcki 106176 | pcko 106176 | si 2132 Kbps | so 2132 Kbps |
PID CPU COMMAND-LINE 1/1
16857 11% /usr/sbin/tcpdump -i eth0 port 2049 and udp
7979 10% ./pnfsd
7982 10% ./pnfsd
7981 10% ./pnfsd
7980 10% ./pnfsd
7902 6% ./dbserver cms11
7900 3% postgres: enstore cms11 [local] idle
29338 2% /usr/java/jdk1.6.0_16/bin/java -server -Xmx2048m -XX:MaxDirectMemory
16873 2% /usr/sbin/tcpdump -i lo port 2049 and udp
7866 2% ./dbserver admin
7912 1% ./dbserver cms
7968 1% ./dbserver cms5
8316 1% kipmi0
? 1%
? 1%
---------------------------------------------------------------------------------------------------------------------------
ipcs
------ Shared Memory Segments --------
key shmid owner perms bytes nattch status
0x0052e2c1 0 enstore 600 2247483392 66
0x00001122 32769 root 600 1480 26
0x00000000 65538 root 700 12288 2
0x00000000 98307 root 700 12288 22
0x00000000 131076 root 700 12288 22
0x00000000 163845 root 700 12288 22
0x00000000 196614 root 700 12288 22
0x7a01809b 229383 zabbix 666 1099804 6
------ Semaphore Arrays --------
key semid owner perms nsems
0x0052e2c1 3538944 enstore 600 17
0x0052e2c2 3571713 enstore 600 17
0x0052e2c3 3604482 enstore 600 17
0x0052e2c4 3637251 enstore 600 17
0x0052e2c5 3670020 enstore 600 17
0x0052e2c6 3702789 enstore 600 17
0x0052e2c7 3735558 enstore 600 17
0x002fa327 3768327 root 666 2
0x00000000 3801096 root 600 1
0x00000000 3833865 root 700 2
0x00000000 3866634 root 700 2
0x00000000 3899403 root 700 2
0x00000000 3932172 root 700 2
0x00000000 3964941 root 700 2
0x00000000 3997710 root 700 2
0x00000000 4030479 root 700 2
0x00000000 4063248 root 700 2
0x00000000 4096017 root 700 2
0x00000000 4128786 root 700 2
0x00000000 4161555 root 700 2
0x00000000 4194324 root 700 2
0x00000000 4227093 root 700 2
0x00000000 4259862 root 700 2
0x00000000 4292631 root 700 2
0x00000000 4325400 root 700 2
0x00000000 4358169 root 700 2
0x00000000 4390938 root 700 2
0x00000000 4423707 root 700 2
0x00000000 4456476 root 700 2
0x00000000 4489245 root 700 2
0x00000000 4522014 root 700 1
0x00000000 4554783 root 700 1
0x00000000 4587552 root 700 1
0x00000000 4620321 root 700 1
0x00000000 4653090 root 700 1
0x00000000 5013539 root 600 1
0x00000000 5046308 root 600 1
0x00000000 5079077 root 600 1
0x00000000 5111846 root 600 1
0x00000000 5144615 root 600 1
0x00000000 5177384 root 600 1
0x00000000 5210153 root 600 1
0x00000000 5242922 root 600 1
0x7a01809b 5275691 zabbix 666 1
------ Message Queues --------
key msqid owner perms used-bytes messages
---------------------------------------------------------------------------------------------------------------------------
time df /pnfs/fs:
Filesystem 1K-blocks Used Available Use% Mounted on
localhost:/fs 400000 80000 284000 22% /pnfs/fs
real 0m0.002s
user 0m0.000s
sys 0m0.008s
---------------------------------------------------------------------------------------------------------------------------
PNFSMANAGER INFO:
In 65 sec 212 transactions = 3.261538 trans/sec
dCache Admin (VII) (user=enstore)
[cmssrv32.fnal.gov] (local) enstore > cd PnfsManager
[cmssrv32.fnal.gov] (PnfsManager) enstore > info
$Revision: 11501 $
NameSpace Provider:
$Revision: 11330 $
Server : fnal.gov(cmspnfs1.fnal.gov)
RealMountPoint : 000000000000000000001040 /pnfs/fs
VirtualMountId : 000000000000000000001040
VirtualPnfsPath : /root/fs
VirtualLocalPath : /pnfs/fs
VirtualGlobalPath : /pnfs/fnal.gov
CacheLocation Provider:
$Id: SQLNameSpaceProvider.java,v 1.19 2007-08-22 12:24:38 tigran Exp $
StorageInfo Provider:
$Revision: 11330 $
Server : fnal.gov(cmspnfs1.fnal.gov)
RealMountPoint : 000000000000000000001040 /pnfs/fs
VirtualMountId : 000000000000000000001040
VirtualPnfsPath : /root/fs
VirtualLocalPath : /pnfs/fs
VirtualGlobalPath : /pnfs/fnal.gov
Store Default AL and RP: false
Threads (4) Queue
[0] 0
[1] 0
[2] 0
[3] 0
Thread groups (1)
[0] 0
Cache Location Queues
[0] 0
[1] 0
[2] 0
[3] 0
Statistics:
addCacheLocation 3084443 421 0
clearCacheLocation 2814362 0 0
getCacheLocations 13249192 0 11086
createDirectory 2448 41 0
createEntry 1564980 693 0
deleteEntry 33427 2157 0
getStorageInfo 15714468 5080884 1218047
setStorageInfo 0 0 0
setLength 1563896 3886 0
getCacheStatistics 0 0 0
updateCacheStatistics 0 0 0
rename 1454 0 0
setLevelData 0 0 0
fileFlushed 401772 0 0
mapPath2Id 380083 3 0
mapId2Path 0 0 0
getMetadataInfo 2731327 285477 18231
setMetadataInfo 0 0 0
[cmssrv32.fnal.gov] (PnfsManager) enstore > ..
[cmssrv32.fnal.gov] (local) enstore >
[cmssrv32.fnal.gov] (local) enstore > logoff
dmg.util.CommandExitException: (0) Done
[cmssrv32.fnal.gov] (local) enstore >