On ubdaq-prod-evb » History » Version 6

Afroditi Papadopoulou, 12/28/2018 10:10 AM

1 1 Lu Ren
h1. Running out of Disk Space on ubdaq-prod-evb ?
2 1 Lu Ren
3 5 Afroditi Papadopoulou
useful info: there are ~ 33 TB of disk space in /data/ on the evb machine. PUBS will try and clear data in /data/uboonedaq/TestRuns/ until the disk-usage reaches 40% of /data/uboonedaq/TestRuns/.
4 1 Lu Ren
5 1 Lu Ren
If this is the case there are several things one should do:
6 5 Afroditi Papadopoulou
7 6 Afroditi Papadopoulou
0) Make sure that dCache is normally operating:
8 5 Afroditi Papadopoulou
--> Take a look at the log file for prod_transfer_binary_evb2dropbox_evb
9 5 Afroditi Papadopoulou
10 5 Afroditi Papadopoulou
    and search for errors / transfer failures like those below
11 5 Afroditi Papadopoulou
[ ERROR   ] transfer (L: 234) >> {transfer_file} Issuing deletion command: ifdh rm /pnfs/uboone/scratch/uboonepro/dropbox/data/uboone/raw/PhysicsRun-2018_12_28_6_39_57-0020483-00002.ubdaq.json
12 5 Afroditi Papadopoulou
[ ERROR   ] transfer (L: 238) >> {transfer_file} TRIED TO DELETE /pnfs/uboone/scratch/uboonepro/dropbox/data/uboone/raw/PhysicsRun-2018_12_28_6_39_57-0020483-00002.ubdaq.json but got an error from ifdhc
13 6 Afroditi Papadopoulou
If that is the case and the errors persist, activate the bypass by following the instructions here [[What to do if dCache/enstore go down (no access to pnfs area)]]
14 5 Afroditi Papadopoulou
15 5 Afroditi Papadopoulou
16 5 Afroditi Papadopoulou
17 6 Afroditi Papadopoulou
1) Idenfity who is using up the disk space. Options:
18 4 Lu Ren
--> a) /data/uboonedaq/rawdata/   is where data from "official" runs goes. Files here are seen (and should be eventually removed) by PUBS.
19 4 Lu Ren
--> b) /data/uboonedaq/TestRuns/ - is disk-space DAQ people use to test things. It is not seen by PUBS and needs to be removed by hand in order to be cleared.
20 4 Lu Ren
21 1 Lu Ren
If most of the space is not being used by /data/uboonedaq/rawdata/ we need to free space manually. If it is urgent to free up space (i.e. data-taking should not be interrupted and the disk will fill up rather soon) you are authorized to clear /data/uboonedaq/TestRuns/. Contact any other person who is using up a considerable amount of space and ask them to quickly remove contents in their /data/ folder.
22 1 Lu Ren
If /data/uboonedaq/rawdata/ is using up a significant amount of space, the problem is probably PUBS' fault.
23 6 Afroditi Papadopoulou
2) identify the cause of the problem. Why is disk space not being freed? Possible causes:
24 1 Lu Ren
--> a) clear_binary_evb is having issues.
25 1 Lu Ren
--> b) clear_binary_evb does not find any new files to clear. This indicates a possible problem with one of the projects that clear_binary_evb depends on. A possible cause could be poor network speed to drain data out of the evb machine.