Starting the PUBS online daemon¶
>kinit >ssh -Y <USERID>@ubdaq-prod-ws01.fnal.gov #where USERID is your kerberos principal ID, e.g. <USERID>@FNAL.GOV
Once you are logged into ubdaq-prod-ws01, you can now access the online cluster. There are three machines where we run PUBS daemons, ubdaq-prod-evb, ubdaq-prod-near1, and ubdaq-prod-ws02. To start the daemon, you will want to log in as user uboonepro.
ws01>ssh -Y firstname.lastname@example.org
Once there, you will need to setup the PUBS environment. The production version of PUBS is installed in /home/uboonepro/pubs/ and is the only version that should be run for production operations.
evb> cd pubs evb> source config/setup_uboonepro_online.sh evb> conn_test
This last commands makes sure that the account can connect to the procdb database with the correct authorization. Everything should connect and disconnect successfully. Once you've done that, you need to make sure that the daemon is running on ubdaq-prod-evb, ubdaq-prod-near1, and ubdaq-prod-ws02. You can check the status of the daemon using these commands.
evb> daemon.sh status
And check each of the servers for the last time that they were updated. Note that if there are no projects enable on a server, the log won't update even if the daemon is running. And if you update the configuration of a project, you must restart the daemon.
In order to start and stop the daemon on each of the machines, EVB, NEAR1, and WS02 using the following commands as uboonepro.
evb>cd pubs evb>source config/setup_uboonepro_online.sh evb>daemon.sh stop
You should do the same on ubdaq-prod-ws02 for the SuperNova Stream projects from a different terminal than the one for evb.
>ssh -Y <USERID>@ubdaq-prod-ws01.fnal.gov ws01> ssh -Y email@example.com ws02> cd pubs ws02> source config/setup_uboonepro_online.sh ws02> daemon.sh stop # MAKE SURE TO WAIT FOR THE DAEMON TO STOP
Finally, you stop the daemon on near1. In order to do this, you need to log into ubdaq-prod-near1 in a different terminal (keep the first two open to restart the evb and ws02 daemons later), setup the pubs environment, and stop the near1 daemon. On Monday, you should also archive the logs for the PUBS projects by issuing the
#LOG ONTO UBDAQ-PROD-NEAR1 TO PERFORM THESE STEPS!!!!! >ssh -Y <USERID>@ubdaq-prod-ws01.fnal.gov ws01> ssh -Y firstname.lastname@example.org near1> cd pubs near1> source config/setup_uboonepro_online.sh near1> daemon.sh stop # MAKE SURE TO WAIT FOR THE DAEMON TO STOP - you should get an email when it happens near1> daemon.sh status #to check that the daemon has stopped, you should also check EVB and WS02. EVB is often the slowest daemon to stop near1> dstream_online/rotate_logs.sh #ONLY DO THIS ON MONDAY
Then restart the daemon on both EVB, WS02, and NEAR1
don't forget NEAR1
Then check again with list_log to make sure that the projects are updating. You should do this every Monday, Wednesday, Friday, and Sunday.
Questions? Ask Kirby