Working directories and files are under /grid/data/${GROUP}/LOCK

On the grid, the username does not reflect the identity of the
person who submitted the job.
So the lock script gets the identity from the grid proxy.

/LOCKS - active lock files
The lock files are empty, with names contining
date, time queued, host, pid, user, identity

/QUEUE - locks pending, empty files containing
date, host, pid, user, identity

/LOG - empty files with names reflecting completed locks
date, time queued, time locked, host, pid, user, identity

/LOGS - monthly text summaries built from LOG file names.

/STALE - record of locks that have timed out

glimit - global activity limit, including all user groups
set this near the actual Bluearc capacity
this is not implemented as of 2012-11-06

limit - local activity limit, for the users' own group
set this well under Bluearc capacity

perf - performance MB/sec required in PERF before locking

PERF - actual MB/sec performance, measured by external agent
( No agents implemented as of 2010-08-02 )

rate - net retry rate target, in retries per second

small - MBytes: files smaller than this are not locked by cpn.

wait - mininum time to wait before retrying, regardless of the load.
the time delay before retrying a lock is the minimum of
  • wait
  • (number of queued locks)/rate


lock files should be owned by some appropriate group account, like mindata.

That account should occasionally remove expired locks and queue entries,
and concatenate LOG entries into monthly summary files.

You can run the lockclean script manually, which will do this hourly :
But be careful, interactive logins on gpsn01 are in group gpcf.
Use 'sg' to set the proper group first

    set nohup ; sg ${GRP} -c /grid/fermiapp/common/tools/lockclean &

There should be a crontab entry for each account like
@reboot sg ${GRP} -c /grid/fermiapp/common/tools/lockclean

Group Account@Host crontab
annie ifmon@ifmongpvm02 @reboot sleep 300; sg annie /grid/fermiapp/common/tools/lockclean
cdms ifmon@ifmongpvm02 @reboot sleep 300; sg scdms /grid/fermiapp/common/tools/lockclean
coupp ifmon@ifmongpvm02 @reboot sleep 300 ; sg coupp -c /grid/fermiapp/common/tools/lockclean
darkside ifmon@ifmongpvm02 @reboot sleep 300 ; sg darkside -c /grid/fermiapp/common/tools/lockclean
des desdata@ifmongpvm02 @reboot sleep 300 ; sg des -c /grid/fermiapp/common/tools/lockclean
dune ifmon@ifmongpvm02 @reboot sleep 300 ; /grid/fermiapp/common/tools/lockclean
e875 mindata@minos27 @reboot sleep 300 ; /grid/fermiapp/common/tools/lockclean
e898 ifmon@ifmongpvm02 @reboot sleep 300 ; sg e898 -c /grid/fermiapp/common/tools/lockclean
e938 ifmon@ifmongpvm02 @reboot sleep 300 ; sg e938 -c /grid/fermiapp/common/tools/lockclean
gm2 gm2dat@ifmongpvm02 @reboot sleep 300 ; sg gm2 -c /grid/fermiapp/common/tools/lockclean
gpcf ifmon@ifmongpvm02 @reboot sleep 300 ; sg gpcf -c /grid/fermiapp/common/tools/lockclean
lar1nd ifmon@ifmongpvm02 @reboot sleep 300 ; sg lar1nd -c /grid/fermiapp/common/tools/lockclean
marslbne marslbne@ifmongpvm02 @reboot sleep 300 ; /grid/fermiapp/common/tools/lockclean
marsmu2e marsmu2e@detsim @reboot sleep 300 ; /grid/fermiapp/common/tools/lockclean
mu2e ifmon@ifmongpvm02 @reboot sleep 300 ; /grid/fermiapp/common/tools/lockclean
mu2epro mu2epro@mu2egpvm01 RETIRED FEB 4 2013
nova novadata@gpcf028 @reboot sleep 300; /grid/fermiapp/common/tools/lockclean
numix ifmon@ifmongpvm02 @reboot sleep 300; sg numix -c /grid/fermiapp/common/tools/lockclean
sbnd ifmon@ifmongpvm02 @reboot sleep 300; sg sbnd -c /grid/fermiapp/common/tools/lockclean
sdss ifmon@ifmongpvm02 @reboot sleep 300; sg sdss /grid/fermiapp/common/tools/lockclean
t-962 argoneut@argoneutgpvm01 @reboot sleep 300; /grid/fermiapp/common/tools/lockclean
t-1034 ifmon@ifmongpvm02 @reboot sleep 300 ; sg t-1034 -c /grid/fermiapp/common/tools/lockclean
uboone uboone@uboonegpvm01 @reboot sleep 300; /grid/fermiapp/common/tools/lockclean


Get an idea of activity by counting lines in log files.

For example, for Minos,

$ wc -l /grid/data/e875/LOCK/LOGS/*.log
9124 /grid/data/e875/LOCK/LOGS/200908.log
140794 /grid/data/e875/LOCK/LOGS/200909.log
181895 /grid/data/e875/LOCK/LOGS/200910.log
196327 /grid/data/e875/LOCK/LOGS/200911.log
125084 /grid/data/e875/LOCK/LOGS/200912.log
272598 /grid/data/e875/LOCK/LOGS/201001.log
284000 /grid/data/e875/LOCK/LOGS/201002.log
275479 /grid/data/e875/LOCK/LOGS/201003.log
354725 /grid/data/e875/LOCK/LOGS/201004.log
1840026 total
$ wc -l /grid/data/e875/LOCK/STALE/LOCKS/*.log
$ wc -l /grid/data/e875/LOCK/STALE/QUEUE/*.log


New LOCK areas should be created by the ifmon account.
We can use root@if-admin-minos to create this area.


GROU=<name of the group>
sg ${GROU}
GROU=<name of the group>

. /grid/fermiapp/products/common/etc/
ups tailor cpn          # will echo the commands proposed 
ups tailor cpn -O write # will execute the commands

cd /grid/data/${GROU}/LOCK
Update gpsn01:ifmon:crontab.gpsn01
adding an entry for the new group
and start the lockclean process manually
crontab crontab.gpsn01

set nohup ;  sg ${GROU} -c /grid/fermiapp/common/tools/lockclean &
Test the locks with a normal user
GROU=<name of the group>

. /grid/fermiapp/products/common/etc/

setup cpn


LOCK - Wed Jan 22 16:37:07 UTC 2014 lock  /grid/data/t-1034/LOCK/LOCKS/20140122.16:37:07.0.minos27.7436.kreymer.kreymer

lock free
LOCK - Wed Jan 22 16:37:09 UTC 2014 freed /grid/data/t-1034/LOCK/LOCKS/20140122.16:37:07.0.minos27.7436.kreymer.kreymer

cpn /usr/bin/crash /dev/null
LOCK - Wed Jan 22 16:37:44 UTC 2014 lock  /grid/data/t-1034/LOCK/LOCKS/20140122.16:37:44.1.minos27.30874.kreymer.kreymer
LOCK - Wed Jan 22 16:37:44 UTC 2014 freed /grid/data/t-1034/LOCK/LOCKS/20140122.16:37:44.1.minos27.30874.kreymer.kreymer

If the /grid/data/${GROU} does not exist, or needs to be linked to alternate name, see the following notes

DATA=${GROU}  # directory to be locked, usually ${GROU}/data, but not always

Determine what the gid is for the area.
stat /${DATA}/data | grep Gid


IFUID=45438  # ifmon UID
GRGID=....   # group GID from /${DATA}/data
GROU=<name of the group>

mkdir -p                /grid/data/$GROU}  # if necessary       
ls -ld                  /grid/data/$GROU}  # note original owner
chown ${IFUID}.${GRGID} /grid/data/$GROU}  # set ownership while tailoring cpn

REX will verify that the group id name is the same on Fermigrid nodes and in the Lab GID registry, at

WARNING - if the group name contains a special character like -,
that group name will not be used on Fermigrid worker nodes.
You need to find out what the group name is there,
and make a symlink in /grid/data for compatibility

GRID=<name of Fermigrid group>

ln -s /grid/data/${GROU} /grid/data/${GRID}