We are transitioning from Google Hangouts to Slack for shift communications. This does not affect domestic call forwarding, which uses Google Voice, not Hangouts. However, international call forwarding still uses hangouts.
There have been updates to the PUBS/DataManagement monitoring. The PUBS gui has changed queued processes from orange to blue. Additionally, all PUBS resource monitoring, previously on a separate website, are now monitored by the Slow Controls. The Slow Controls Alarm Panel now has a DataMgmt box which keeps track of resource monitoring and PUBS daemon status. Because of this many checklist questions have been removed. Shifter instructions for monitoring Data Management have changed. Please review the DM_-_Shifters page.
Added one more bullet point for shifter's task for DM monitoring: on PUBS GUI window, watch out for the "Queued" counter below each project's progress bar. Contact the expert if this counter never goes down over ~30 minutes period.
At the beginning of your shift, please communicate your name and where you are shifting from in the google hangouts chat, so that experts and RunCo's are updated in real time about who is on shift and where.
We are running with single remote only shifters, and the operations wiki has been updated to reflect that. If you see comments about video connections to ROC West or your co-shifter, they are probably relics. Let the RunCos know by email so we can address it.
The biggest changes occurred in the off site shift requirements and procedures, so read them carefully. Note that google hangouts is mandatory (plugin if needed) which is a new requirement recently.
Always read the Run Plan at the start of every 8 hour shift block!
Updated the cryo IFIX username and password in the routine shift procedure section of the main page of the Operations Wiki.
There is a "latching" keytree alarm in the slow controls now, which will go into minor alarm when someone removes a key from the keytree at LArTF (with some delay) and will remain in alarm even after the key is replaced unless it is acknowledged. If you are unaware of what access is being made in conjunction with this alarm, please contact LArTF and find out who is making the entrance. Then let the Run Coordinator know immediately of the access (by phone), if it was not cleared beforehand.
With respect to the header below titled "January 25th, 2016": Currently, all PUBS projects displayed on the monitoring GUI are enabled and should remain enabled. Shifters can resume following the usual PUBS monitoring procedure.
When performing the "Is the Slow Controls archiver running?" check on the checklist, please be sure to refresh the web page that the instructions say to look at, and notify the slow controls expert if there is a non-zero number in a row labeled "Disconnected" on that page. As of now, the archiver should never be seeing any disconnected channels.
Please note that currently, the "Binary File Drain [EVB --> Near1]" project is disabled and so appears grey in the PUBS monitoring GUI. This is the correct configuration currently and so the expert should not be contacted.
Added a missing step of "setup pyqtgraph" @ DataManagement shift procedure to start PUBS monitoring GUI.
Updated the instructions on who to contact when a particular alarm panel goes into an alarm state in the Slow Controls Guide.
There is a new feature in the slow monitoring that will play a little alarm music on a web page when certain critical channels have an alarm. In order for it to work, you have to have the alarm sounder page open in a browser and the volume turned up. (Use cr02 in ROC-W for best results.) See the Alarm Sounder wiki page for more info.
Very slight change in PUBS monitoring instructions for shifters ( DM_-_Shifters ). Now the monitoring GUI is run from the ubdaq-prod-evb machine, where previously it was run from the ubdaq-prod-smc machine. No other changes.
If you use any link to ECL in the Operations Wiki, be aware that due to the recent URL change the given default in the wiki is dbweb0, but you could be redirected to any number from 1 to 6. So if a link doesn't work, you have to manually change the number to the one you were assigned when you first logged into ECL on that computer.
The shifter checklist that needs to be filled has instructions on the far right side of the form. Maximize this window to ensure that you can read the instructions.
Please do not run kinit or kdestroy commands on any of the DAQ or control room machines anymore. We have cron jobs to renew special tickets for the ubooneshift and uboonedaq accounts every three hours.
In the "header" page of the Online Monitoring Lizard, we only care about the alignment of crate frame numbers w.r.t. trigger headers. Those should be aligned but are allowed to be offset by one. The crate event numbers are NOT expected to be always aligned w.r.t. trigger headers. Moreover, the Online Monitoring Lizard will raise a "big red box" alarm when there is a problematic alignment issue.
We have an audible DAQ alarm that is triggered when a run is stopped and won't automatically restart (as happens when the run control encounters an unforeseen failure mode). This indicates that shifter intervention is necessary to begin taking data again. If you have not been informed about the default run plan (either by reading the Run Plan page if it is up-to-date or being told by the Run Coordinator or DAQ expert), please contact the DAQ expert or Run Coordinator.
The run control is now making use of a VNC session on evb instead of one on ws01. Instructions in RC Guide have been updated.