DM - Expert Documentation » History » Version 9

« Previous - Version 9/72 (diff) - Next » - Current version
David Kaleko, 01/21/2016 07:01 AM

DM - Expert Documentation


... is finally initiated by David Caratelli :)

Keep updated! Also attach the latest version to this Wiki.

Starting the PUBS daemon running

Details are on this page. Starting the PUBS online daemon

Building up the PUBS online testbed

Details are on this page. Building up the PUBS online testbed

Running out of Disk Space on ubdaq-prod-evb ?

useful info: there are ~ 33 TB of disk space in /data/ on the evb machine. PUBS will try and clear data in /data/uboonedaq/TestRuns/ until the disk-usage reaches 40% of /data/uboonedaq/TestRuns/ is empty.

If this is the case there are several things one should do:
0) Idenfity who is using up the disk space. Options:
--> a) /data/uboonedaq/rawdata/ > this is where data from "official" runs goes. Files here are seen (and should be eventually removed) by PUBS.
> b) /data/uboonedaq/TestRuns/ > this is disk-space DAQ people use to test things. It is not seen by PUBS and needs to be removed by hand in order to be cleared.
> c) /data/uboonedaq/lukhanin/ > test-space for Gennadiy. Also needs to be removed manually in order to free up space.
> d) /data/OTHER/ > data used by someone else.
If most of the space is not being used by /data/uboonedaq/rawdata/ we need to free space manually. If it is urgent to free up space (i.e. data-taking should not be interrupted and the disk will fill up rather soon) you are authorized to clear /data/uboonedaq/TestRuns/. Contact any other person who is using up a considerable amount of space and ask them to quickly remove contents in their /data/ folder.
If /data/uboonedaq/rawdata/ is using up a significant amount of space, the problem is probably PUBS' fault.
1) identify the cause of the problem. Why is disk space not being freed? Possible causes:
> a) clear_binary_evb is having issues.
--> b) clear_binary_evb does not find any new files to clear. This indicates a possible problem with one of the projects that clear_binary_evb depends on. A possible cause could be poor network speed to drain data out of the evb machine.

Questions? Ask Kirby

Running out of Disk Space on /datalocal/ @ near1 ?

If the disk-usage @ /datalocal/ is above 95% as an immediate action please stop the "mv_binary_evb" project. Notify the PUBS team that you just did this and start addressing the disk-space issue

What to do if dcashe/enstore go down (no access to pnfs area)

First dump the current projects configs to a temporary file:
cfg_dump_project current_cfg_dump.cfg

Then, edit current_cfg_dump.cfg. Find the project "prod_transfer_binary_evb2dropbox_near1", and change the ENABLE parameter to False

sbin/pmod_enable prod_transfer_binary_evb2dropbox_near1 0
sbin/pmod_enable prod_transfer_binary_near12dropbox_near1 0
sbin/pmod_enable prod_transfer_binary_evb2near1_near1 1
sbin/register_project current_cfg_dump.cfg

Restarting the daemons is not necessary.