Project

General

Profile

Starting the PUBS online daemon

>kinit
>ssh -Y <USERID>@ubdaq-prod-ws01.fnal.gov #where USERID is your kerberos principal ID, e.g. <USERID>@FNAL.GOV

Once you are logged into ubdaq-prod-ws01, you can now access the online cluster. There are three machines where we run PUBS daemons, ubdaq-prod-evb, ubdaq-prod-near1, and ubdaq-prod-ws02. To start the daemon, you will want to log in as user uboonepro.
ws01>ssh -Y uboonepro@ubdaq-prod-evb.fnal.gov

Once there, you will need to setup the PUBS environment. The production version of PUBS is installed in /home/uboonepro/pubs/ and is the only version that should be run for production operations.
evb> cd pubs
evb> source config/setup_uboonepro_online.sh
evb> conn_test

This last commands makes sure that the account can connect to the procdb database with the correct authorization. Everything should connect and disconnect successfully. Once you've done that, you need to make sure that the daemon is running on ubdaq-prod-evb, ubdaq-prod-near1, and ubdaq-prod-ws02. You can check the status of the daemon using these commands.
evb> daemon.sh status

And check each of the servers for the last time that they were updated. Note that if there are no projects enable on a server, the log won't update even if the daemon is running. And if you update the configuration of a project, you must restart the daemon.

In order to start and stop the daemon on each of the machines, EVB, NEAR1, and WS02 using the following commands as uboonepro.

evb>cd pubs
evb>source config/setup_uboonepro_online.sh
evb>daemon.sh stop

You should do the same on ubdaq-prod-ws02 for the SuperNova Stream projects from a different terminal than the one for evb.

>ssh -Y <USERID>@ubdaq-prod-ws01.fnal.gov
ws01> ssh -Y uboonepro@ubdaq-prod-ws02.fnal.gov
ws02> cd pubs
ws02> source config/setup_uboonepro_online.sh
ws02> daemon.sh stop # MAKE SURE TO WAIT FOR THE DAEMON TO STOP

Finally, you stop the daemon on near1. In order to do this, you need to log into ubdaq-prod-near1 in a different terminal (keep the first two open to restart the evb and ws02 daemons later), setup the pubs environment, and stop the near1 daemon. On Monday, you should also archive the logs for the PUBS projects by issuing the dstream_online/rotate_logs.sh command:

#LOG ONTO UBDAQ-PROD-NEAR1 TO PERFORM THESE STEPS!!!!!
>ssh -Y <USERID>@ubdaq-prod-ws01.fnal.gov
ws01> ssh -Y uboonepro@ubdaq-prod-near1.fnal.gov
near1> cd pubs
near1> source config/setup_uboonepro_online.sh
near1> daemon.sh stop # MAKE SURE TO WAIT FOR THE DAEMON TO STOP - you should get an email when it happens
near1> daemon.sh status #to check that the daemon has stopped, you should also check EVB and WS02. EVB is often the slowest daemon to stop
near1> dstream_online/rotate_logs.sh #ONLY DO THIS ON MONDAY

Then restart the daemon on both EVB, WS02, and NEAR1

evb>daemon.sh start

and WS02

ws02>daemon.sh start

don't forget NEAR1

near1>daemon.sh start

Then check again with list_log to make sure that the projects are updating. You should do this every Monday, Wednesday, Friday, and Sunday.

Questions? Ask Kirby