Project

General

Profile

On ubdaq-prod-evb » History » Version 7

Michael Kirby, 12/28/2018 10:29 AM

1 1 Lu Ren
h1. Running out of Disk Space on ubdaq-prod-evb ?
2 1 Lu Ren
3 1 Lu Ren
If this is the case there are several things one should do:
4 5 Afroditi Papadopoulou
5 1 Lu Ren
0) Make sure that dCache is normally operating:
6 6 Afroditi Papadopoulou
--> Take a look at the log file for prod_transfer_binary_evb2dropbox_evb
7 7 Michael Kirby
    /home/uboonepro/pubs/log/ubdaq-prod-near1.fnal.gov/prod_transfer_binary_evb2dropbox_evb.log  #A note on history, this project is listed as running on evb, but is actually run on near1
8 5 Afroditi Papadopoulou
    and search for errors / transfer failures like those below
9 5 Afroditi Papadopoulou
[ ERROR   ] transfer (L: 234) >> {transfer_file} Issuing deletion command: ifdh rm /pnfs/uboone/scratch/uboonepro/dropbox/data/uboone/raw/PhysicsRun-2018_12_28_6_39_57-0020483-00002.ubdaq.json
10 5 Afroditi Papadopoulou
[ ERROR   ] transfer (L: 238) >> {transfer_file} TRIED TO DELETE /pnfs/uboone/scratch/uboonepro/dropbox/data/uboone/raw/PhysicsRun-2018_12_28_6_39_57-0020483-00002.ubdaq.json but got an error from ifdhc
11 5 Afroditi Papadopoulou
If that is the case and the errors persist, activate the bypass by following the instructions here [[What to do if dCache/enstore go down (no access to pnfs area)]]
12 1 Lu Ren
13 1 Lu Ren
1) Idenfity who is using up the disk space. Options:
14 5 Afroditi Papadopoulou
--> a) /data/uboonedaq/rawdata/   is where data from "official" runs goes. Files here are seen (and should be eventually removed) by PUBS.
15 6 Afroditi Papadopoulou
--> b) /data/uboonedaq/TestRuns/ - is disk-space DAQ people use to test things. It is not seen by PUBS and needs to be removed by hand in order to be cleared.
16 4 Lu Ren
17 7 Michael Kirby
useful info: there are ~ 33 TB of disk space in /data/ on the evb machine. PUBS will try and clear data in /data/uboonedaq/rawdata/ once it has been transferred to the FTS dropbox and verified. Note that files in other directories will not be deleted (e.g. /data/uboonedaq/TestRuns/).
18 7 Michael Kirby
19 1 Lu Ren
If most of the space is not being used by /data/uboonedaq/rawdata/ we need to free space manually. If it is urgent to free up space (i.e. data-taking should not be interrupted and the disk will fill up rather soon) you are authorized to clear /data/uboonedaq/TestRuns/. Contact any other person who is using up a considerable amount of space and ask them to quickly remove contents in their /data/ folder.
20 1 Lu Ren
If /data/uboonedaq/rawdata/ is using up a significant amount of space, the problem is probably PUBS' fault.
21 6 Afroditi Papadopoulou
2) identify the cause of the problem. Why is disk space not being freed? Possible causes:
22 7 Michael Kirby
--> a) prod_clear_binary_evb is having issues.  Look in this log for errors: /home/uboonepro/pubs/log/ubdaq-prod-evb.fnal.gov/prod_clean_evb_binary_evb.log
23 7 Michael Kirby
--> b) prod_clear_binary_evb does not find any new files to clear. This indicates a possible problem with one of the projects that prod_clear_binary_evb depends on. A possible cause could be poor network speed to drain data out of the evb machine. Look for projects upstream from prod_clear_binary_evb (e.g. /home/uboonepro/pubs/log/ubdaq-prod-near1.fnal.gov/prod_verify_binary_evb2dropbox_near1.log)