Project

General

Profile

DM - Expert Documentation » History » Version 69

Version 68 (Michael Kirby, 11/20/2019 12:43 PM) → Version 69/72 (Michael Kirby, 11/20/2019 12:45 PM)

{{>toc}}

h1. DM - Expert Documentation

h2. Tasks for DM Expert Shifts

h3. Starting the PUBS daemon running

*%{color:red}The daemons should be restarted every Monday-Wednesday-Friday-Sunday, and PUBS logs rotated every Monday.%*

Details are on this page. [[Starting the PUBS online daemon]]

h3. Check to make sure there are no *hidden* errors in the PUBS GUI.

On Monday at the start of shift, the DM Expert should start the "PUBS GUI":https://cdcvs.fnal.gov/redmine/projects/uboone-operations/wiki/DM_-_Shifters#2-PUBS-project-status-and-restarting-PUBS-GUI-follow-these-instructions-to-make-sure-that-PUBS-projects-are-actively-running-on-ubdaq-prod-evb-and-ubdaq-prod-near1 Once the GUI is up and running, click on the "Use Relative Counters" to get the absolute numbers of successful and failed processing for each PUBS project. If there are any residual errors, correct them and make an e-log entry.

h3. Correcting Errors in PUBS

* Querying DB for errors. [[DB Query]]

* Errors in *Metadata Generation* From Incomplete Files [[Correcting Failed Metadata Generation]]

* Failed *Near1 Binary Transfers* [[Correcting Failed Near1 Binary Transfer]]

* Errors in *Registering File Metadata* and crontab entries for kerberos tickets and grid proxies [[Correcting Failed Metadata Registration]]

h3. Reconfigure PUBS if there is a dCache or network downtime

[[What to do if dCache/enstore go down (no access to pnfs area)]]

h3. If there are errors for online servers (EVB, NEAR1, or WS02) running out of Disk Space?

* [[on ubdaq-prod-evb]]
* [[on near1 (/datalocal/)]]
* [[on sebXX (uB_DataMgmt_PCXX_seb06_data/disk_occ)]]

h3. Respond to collaborator requests to prevent the deletion of one or more SN runs.

To prevent the deletion of one or more runs in the SN stream login as uboonepro. Head over the to the SN PUBS script directory located here /home/uboonepro/pubs/dstream_online/snova. Here you will find "frozen_runs.txt". In this file insert *new line separated* run numbers. The monitoring script will read this ASCII text file, and prevent the deletion of files in this text file.

h2. Documentation!

... is finally initiated by David Caratelli [[:File:PUBS.pdf|PUBS.pdf]] https://www.overleaf.com/3384459hxbyns#/9541217/

For super-duper experts, if interested in, PUBS base framework documentation is on "DocDB 5400":http://microboone-docdb.fnal.gov:8080/cgi-bin/ShowDocument?docid=5400

Keep updated! Also attach the latest version to this Wiki.

h2. Moving all projects to a single online machine

Details are on this page. [[Running all PUBS projects on single server]]

h2. Building up the PUBS online testbed

Details are on this page. [[Building up the PUBS online testbed]]

h2. Mapping project name to names on GUI.

How do I find the project name (database table name) given the name of a specific box on the monitoring gui? [[Project GUI Map]]

h2. Changing the Database Configuration for Online PUBS [[Online PUBS Database Reconfig]]

h2. Expired Certificate on Near1 "Request OSG Production Service Certificate":https://cdcvs.fnal.gov/redmine/projects/uboonecode/wiki/CSR

h2. Running out of Disk Space?

* [[on ubdaq-prod-evb]]
* [[on near1 (/datalocal/)]]
* [[on sebXX (uB_DataMgmt_PCXX_seb06_data/disk_occ)]]

h2. [[What to do if dCache/enstore go down (no access to pnfs area)]]

h2. Collaborator has asked me, the DM expert, to prevent the deletion of one or more SN runs.

To prevent the deletion of one or more runs in the SN stream login as uboonepro. Head over the to the SN PUBS script directory located here /home/uboonepro/pubs/dstream_online/snova. Here you will find "frozen_runs.txt". In this file insert *new line separated* run numbers. The monitoring script will read this ASCII text file, and prevent the deletion of files in this text file.


h2. The daemon on ws02 has mysteriously died.

DM experts are currently debugging an issue related to the daemon on ws02 being killed by the kernel. If you are a DM expert on shift and you find the ws02 daemon has mysteriously died. Please execute the following command to copy the log files to a safe location, then please restart the daemon.
<pre>
mkdir -p /data/uboonepro/ws02_daemon_failures/`date +%D`; cp /home/uboonepro/pubs/log/ubdaq-prod-ws02.fnal.gov/* /data/uboonepro/ws02_daemon_failures/`date +%D`/
</pre>

h2. [[DM - Work Notes]] - Archived DM Work Notes

Questions? "Ask Kirby":mailto:kirby@fnal.gov