Page generated: Sat Nov 21 03:08:20 CST 2009
TOP sources of PNFS traffic in last 60 seconds:
0% lhc04n02 11
0% cmssrv33 180
1% cmssrv45-phedex-ddt 2460
1% cmswn 2659
7% cmsstor 10898
88% localhost-PnfsManager 128697
144905 PNFS tcpdump responses or 2415/sec
TOP worker node source of PNFS traffic in last 60 seconds:
0% cmswn1393 20
0% cmswn1405 20
0% cmswn958 20
0% cmswn993 20
0% cmswn1747 23
TOP storage node source of PNFS traffic in last 60 seconds:
0% cmsstor173 890
0% cmsstor245 1179
0% cmsstor96 1239
1% cmsstor51 2444
1% cmsstor78 2473
---------------------------------------------------------------------------------------------------------------------------
PNFS statistics during last 60 seconds:
0% null 0
95% getattr 129967
0% setattr 103
0% root 0
1% lookup 2081
0% readlink 0
2% read 3320
0% wrcache 0
0% write 103
0% create 107
0% remove 2
0% rename 0
0% link 0
0% symlink 0
0% mkdir 0
0% rmdir 0
0% readdir 0
0% fsstat 156
135839 PNFS operations or 2263/sec
---------------------------------------------------------------------------------------------------------------------------
UPTIME:
03:09:23 up 50 days, 12:37, 1 user, load average: 0.35, 0.29, 0.59
---------------------------------------------------------------------------------------------------------------------------
VMSTAT 60 2:
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free buff cache si so bi bo in cs us sy id wa st
2 0 0 3524668 177228 27696380 0 0 144 218 1 0 4 9 87 1 0
0 0 0 3521792 177260 27695764 0 0 13 954 2948 25489 1 3 95 0 0
---------------------------------------------------------------------------------------------------------------------------
atop -c 60 2:
ATOP - cmspnfs1 2009/11/21 03:09:20 60 seconds elapsed
PRC | sys 16.91s | user 8.05s | #proc 315 | #zombie 2 | #exit 878 |
CPU | sys 23% | user 14% | irq 9% | idle 752% | wait 1% |
CPL | avg1 0.35 | avg5 0.29 | avg15 0.59 | csw 1803661 | intr 203082 |
MEM | tot 31.4G | free 3.4G | cache 26.4G | buff 173.1M | slab 713.8M |
SWP | tot 64.0G | free 64.0G | | vmcom 3.0G | vmlim 64.6G |
DSK | sdb | busy 1% | read 85 | write 3501 | avio 0 ms |
NET | transport | tcpi 8170 | tcpo 8546 | udpi 275161 | udpo 275224 |
NET | network | ipi 290962 | ipo 283771 | ipfrw 0 | deliv 288830 |
NET | lo ---- | pcki 261658 | pcko 261658 | si 5333 Kbps | so 5333 Kbps |
PID CPU COMMAND-LINE 1/1
29338 10% /usr/java/jdk1.6.0_16/bin/java -server -Xmx2048m -XX:MaxDirectMemory
4535 5% /usr/sbin/tcpdump -i lo port 2049 and udp
7982 5% ./pnfsd
7980 4% ./pnfsd
7981 4% ./pnfsd
7979 4% ./pnfsd
7902 4% ./dbserver cms11
7866 1% ./dbserver admin
7900 1% postgres: enstore cms11 [local] BIND
4532 1% /usr/sbin/tcpdump -i eth0 port 2049 and udp
8316 1% kipmi0
---------------------------------------------------------------------------------------------------------------------------
ipcs
------ Shared Memory Segments --------
key shmid owner perms bytes nattch status
0x0052e2c1 0 enstore 600 2247483392 66
0x00001122 32769 root 600 1480 26
0x00000000 65538 root 700 12288 2
0x00000000 98307 root 700 12288 22
0x00000000 131076 root 700 12288 22
0x00000000 163845 root 700 12288 22
0x00000000 196614 root 700 12288 22
0x7a01809b 229383 zabbix 666 1099804 6
------ Semaphore Arrays --------
key semid owner perms nsems
0x0052e2c1 3538944 enstore 600 17
0x0052e2c2 3571713 enstore 600 17
0x0052e2c3 3604482 enstore 600 17
0x0052e2c4 3637251 enstore 600 17
0x0052e2c5 3670020 enstore 600 17
0x0052e2c6 3702789 enstore 600 17
0x0052e2c7 3735558 enstore 600 17
0x002fa327 3768327 root 666 2
0x00000000 3801096 root 600 1
0x00000000 3833865 root 700 2
0x00000000 3866634 root 700 2
0x00000000 3899403 root 700 2
0x00000000 3932172 root 700 2
0x00000000 3964941 root 700 2
0x00000000 3997710 root 700 2
0x00000000 4030479 root 700 2
0x00000000 4063248 root 700 2
0x00000000 4096017 root 700 2
0x00000000 4128786 root 700 2
0x00000000 4161555 root 700 2
0x00000000 4194324 root 700 2
0x00000000 4227093 root 700 2
0x00000000 4259862 root 700 2
0x00000000 4292631 root 700 2
0x00000000 4325400 root 700 2
0x00000000 4358169 root 700 2
0x00000000 4390938 root 700 2
0x00000000 4423707 root 700 2
0x00000000 4456476 root 700 2
0x00000000 4489245 root 700 2
0x00000000 4522014 root 700 1
0x00000000 4554783 root 700 1
0x00000000 4587552 root 700 1
0x00000000 4620321 root 700 1
0x00000000 4653090 root 700 1
0x00000000 5013539 root 600 1
0x00000000 5046308 root 600 1
0x00000000 5079077 root 600 1
0x00000000 5111846 root 600 1
0x00000000 5144615 root 600 1
0x00000000 5177384 root 600 1
0x00000000 5210153 root 600 1
0x00000000 5242922 root 600 1
0x7a01809b 5275691 zabbix 666 1
------ Message Queues --------
key msqid owner perms used-bytes messages
---------------------------------------------------------------------------------------------------------------------------
time df /pnfs/fs:
Filesystem 1K-blocks Used Available Use% Mounted on
localhost:/fs 400000 80000 284000 22% /pnfs/fs
real 0m0.001s
user 0m0.000s
sys 0m0.004s
---------------------------------------------------------------------------------------------------------------------------
PNFSMANAGER INFO:
In 63 sec 1352 transactions = 21.460317 trans/sec
dCache Admin (VII) (user=enstore)
[cmssrv32.fnal.gov] (local) enstore > cd PnfsManager
[cmssrv32.fnal.gov] (PnfsManager) enstore > info
$Revision: 11501 $
NameSpace Provider:
$Revision: 11330 $
Server : fnal.gov(cmspnfs1.fnal.gov)
RealMountPoint : 000000000000000000001040 /pnfs/fs
VirtualMountId : 000000000000000000001040
VirtualPnfsPath : /root/fs
VirtualLocalPath : /pnfs/fs
VirtualGlobalPath : /pnfs/fnal.gov
CacheLocation Provider:
$Id: SQLNameSpaceProvider.java,v 1.19 2007-08-22 12:24:38 tigran Exp $
StorageInfo Provider:
$Revision: 11330 $
Server : fnal.gov(cmspnfs1.fnal.gov)
RealMountPoint : 000000000000000000001040 /pnfs/fs
VirtualMountId : 000000000000000000001040
VirtualPnfsPath : /root/fs
VirtualLocalPath : /pnfs/fs
VirtualGlobalPath : /pnfs/fnal.gov
Store Default AL and RP: false
Threads (4) Queue
[0] 0
[1] 0
[2] 0
[3] 0
Thread groups (1)
[0] 0
Cache Location Queues
[0] 0
[1] 1
[2] 0
[3] 0
Statistics:
addCacheLocation 4936613 553 0
clearCacheLocation 4554516 0 0
getCacheLocations 20070904 0 29293
createDirectory 4236 62 0
createEntry 2520784 822 0
deleteEntry 81453 2569 0
getStorageInfo 24099597 6352740 1295369
setStorageInfo 0 0 0
setLength 2518328 6056 0
getCacheStatistics 0 0 0
updateCacheStatistics 0 0 0
rename 2309 0 0
setLevelData 0 0 0
fileFlushed 636077 114 0
mapPath2Id 428030 3 0
mapId2Path 0 0 0
getMetadataInfo 4192361 540373 35336
setMetadataInfo 0 0 0
[cmssrv32.fnal.gov] (PnfsManager) enstore > ..
[cmssrv32.fnal.gov] (local) enstore >
[cmssrv32.fnal.gov] (local) enstore > logoff
dmg.util.CommandExitException: (0) Done
[cmssrv32.fnal.gov] (local) enstore >