Project

General

Profile

Running all PUBS projects on single server

There are times when all of the PUBS projects may need to be moved onto a single online server (e.g. near1 crash, evb overloading, etc). These instructions tell you how to move all of the projects from spread across two servers onto a single server. NOTE: this does NOT discuss moving any of the SuperNova cleanup projects off of ws02 onto evb.

Dump the current configuration and modify

The first thing to do is to dump the current configuration of all of the PUBS projects running.

$> ssh -Y ubdaq-prod-ws01.fnal.gov
ws01 $> ssh -Y uboonepro@ubdaq-prod-evb.fnal.gov
evb $>cd pubs
evb $>source config/setup_uboonepro_online.sh
evb $>cfg_dump_project `klist | grep principal: | sed 's/@FNAL.GOV//g' | sed 's/Default principal: //g'`_default_`date +%F`.cfg

This dumps the current configuration of all of the PUBS projects into the file similarly named to:

kirby_default_2018-04-05.cfg 
That file name isn't special other than for tracking changes and knowing who performed the modifications. There are two modifications that need to be made, disable some projects and move another set of projects from near1 to evb. For the first set, the value of "SERVER" in the project configuration will go from "ubdaq-prod-near1.fnal.gov" to "ubdaq-prod-evb.fnal.gov". While for the second set, the value of "ENABLE" will change from "True" to "False". Everything can be undone by reloading the file you created with cfg_dump_project.

Projects to disable:

prod_transfer_binary_evb2near1_near1
prod_transfer_binary_near12dropbox_near1
prod_verify_binary_near12dropbox_near1
monitor_machine_resources_near1
prod_clean_near_binary_near1

Projects to move to evb:

prod_binary_metadata_near1
prod_verify_binary_evb2dropbox_near1
prod_transfer_binary_evb2dropbox_evb

Loading the new configuration

You now have to load a new configuration file it into the PUBS database. We first have to stop all of the daemons. This is done by logging into EVB, NEAR1, and WS02 each separately and stopping the daemon.

$> ssh -Y ubdaq-prod-ws01.fnal.gov
ws01 $> ssh -Y uboonepro@ubdaq-prod-evb.fnal.gov
evb $> cd pubs
evb $>source config/setup_uboonepro_online.sh
evb $> daemon.sh stop

Do the same for NEAR1 and WS02 just changing the second command line. Once all of the daemons have stopped we load the configuration file. There is a default file in $PUB_TOP_DIR/dstream_online/kirby_evb_only_2018-04-05.cfg that can be used to make these configuration changes.

> $PUB_BIN_DIR/sbin/register_project $PUB_TOP_DIR/dstream_online/kirby_evb_only_2018-04-05.cfg

You will be asked to confirm the changes, type "Y" and proceed. This make take about a minute to update everything. Once that is done, then you should restart the daemon on the single server that is running and ws02.

$> ssh -Y ubdaq-prod-ws01.fnal.gov
ws01 $> ssh -Y uboonepro@ubdaq-prod-evb.fnal.gov
evb $> cd pubs
evb $>source config/setup_uboonepro_online.sh
evb $> daemon.sh start

Returning the configuration to the original config.

We first have to stop all of the daemons and then load the old "current" configuration file that was originally dumped from the PUBS database. This is done by logging into EVB, NEAR1, and WS02 each separately and stopping the daemon.

$> ssh -Y ubdaq-prod-ws01.fnal.gov
ws01 $> ssh -Y uboonepro@ubdaq-prod-evb.fnal.gov
evb $> cd pubs
evb $>source config/setup_uboonepro_online.sh
evb $> daemon.sh stop

Do the same for NEAR1 and WS02 just changing the second command line. Once all of the daemons have stopped we reload the configuration file

$> $PUB_BIN_DIR/sbin/register_project `klist | grep principal: | sed 's/@FNAL.GOV//g' | sed 's/Default principal: //g'`_default_`date +%F`.cfg

NOTE: this assumes that you created a copy of the default configuration when you started!!! You will be asked to confirm the changes, type "Y" and proceed. This make take about a minute to update everything. Once that is done, then you should restart the daemon on the single server that is running and ws02.

$> ssh -Y ubdaq-prod-ws01.fnal.gov
ws01 $> ssh -Y uboonepro@ubdaq-prod-evb.fnal.gov
evb $> cd pubs
evb $>source config/setup_uboonepro_online.sh
evb $> daemon.sh start
repeat on NEAR1 and WS02. Assuming that they are up and running.

Manual modification of project configuration

There is a chance that the projects will have been retuned after April 05, 2018. In that case, you will have to go through the list of projects above to be modified (either disabled or moved to evb) by hand within the configuration file created with the cfg_dump_project command above. Make a copy of the file renaming it to indicate it is for evb only processing, and then edit the projects listed at the top of this page. Here is an example of a project configuration snippet:

PROJECT_BEGIN
NAME      prod_binary_metadata_near1
COMMAND   python dstream_online/get_metadata_no_hang.py prod_binary_metadata_near1
CONTACT   kazuhiro@nevis.columbia.edu:kirby@fnal.gov
SLEEP     0
PERIOD    20
SERVER    ubdaq-prod-near1.fnal.gov
RUNTABLE  MainRun
RUN       3176
SUBRUN    0
ENABLE    True

In order to have projects moved from near1 to evb, we must replace "SERVER ubdaq-prod-near1.fnal.gov" with "SERVER ubdaq-prod-evb.fnal.gov" or vice versa. To disable a project, change "ENABLE True" to "ENABLE False". Please maintain the whitespace of the file. NOTE: Do not change the name of the project, just the server the project is running on. Leaving the project name the same will allow us to know if the project are returned to the correct server when all servers are operating again. Once the file is edited completely, you would load the configuration with:

$> $PUB_BIN_DIR/sbin/register_project <your_modified_config_file>

Resource Monitoring during testing

Resource monitoring of the test switchover performed on 04/05 was documented in elog entry 68506: [[http://dbweb6.fnal.gov:8080/ECL/uboone/E/show?e=68506]]