Operations Wiki Updates

6th May 2019

  • We are transitioning from Google Hangouts to Slack for shift communications. This does not affect domestic call forwarding, which uses Google Voice, not Hangouts. However, international call forwarding still uses hangouts.

10th April 2018

  • There have been updates to the PUBS/DataManagement monitoring. The PUBS gui has changed queued processes from orange to blue. Additionally, all PUBS resource monitoring, previously on a separate website, are now monitored by the Slow Controls. The Slow Controls Alarm Panel now has a DataMgmt box which keeps track of resource monitoring and PUBS daemon status. Because of this many checklist questions have been removed. Shifter instructions for monitoring Data Management have changed. Please review the DM_-_Shifters page.

18th March 2018

  • Added one more bullet point for shifter's task for DM monitoring: on PUBS GUI window, watch out for the "Queued" counter below each project's progress bar. Contact the expert if this counter never goes down over ~30 minutes period.

8th February 2018

  • At the beginning of your shift, please communicate your name and where you are shifting from in the google hangouts chat, so that experts and RunCo's are updated in real time about who is on shift and where.

3rd January 2017

  • We are running with single remote only shifters, and the operations wiki has been updated to reflect that. If you see comments about video connections to ROC West or your co-shifter, they are probably relics. Let the RunCos know by email so we can address it.
  • The biggest changes occurred in the off site shift requirements and procedures, so read them carefully. Note that google hangouts is mandatory (plugin if needed) which is a new requirement recently.
  • Always read the Run Plan at the start of every 8 hour shift block!

2nd September 2016

  • Updates to the run plan have been made recently to accommodate shutdown running. Please review the Run Plan before each shift as it will be updated often.

20th July 2016

  • Increased monitoring of Disk/RAM usage on evb has been obviated by a new DAQ release. The Run Plan has been updated.

17th July 2016

  • Cryo-related IFIX variables are not currently being loaded into slow controls. Please check the IFIX display every 2 hours until this is resolved.

8th June 2016

  • Purity monitors are turned off. Please pay close attention to the new pulse height metric in the online monitor and RunCat. This is how we are tracking the purity now.

26th May

  • A number of minor changes have been made recently:
    • Start of shift form - if there is no expert listed, contact the Run Coordinator and subsystem lead
    • PUBS - online PUBS resource monitoring streamlined, and checklist updated
    • Additional check in online monitor during checklist - pulse height vs time (indicator of purity)

11th May Run Catalogue live!!!

  • The MicroBooNE Run Catalogue is now live. Shifters should replace filling out the run information document with using this catalogue to check the quality of data runs during their shift

5th May, 2016

  • A special DAQ release was put in place to troubleshoot online-monitor issues. The Run Plan has been updated to include close monitoring of resources on near1.

14th April, 2016

  • Monitoring of the MCC7 progress is no longer necessary and has been removed from the Run Plan.
  • The MicroBooNE Run Catalogue is now live. Shifters should replace filling out the run information document with using this catalogue to check the quality of data runs during their shift.

13th April, 2016

  • Added an additional check of PMT rates (using online monitor) to the checklist.

March 29, 2016

  • Updated the cryo IFIX username and password in the routine shift procedure section of the main page of the Operations Wiki.
  • There is a "latching" keytree alarm in the slow controls now, which will go into minor alarm when someone removes a key from the keytree at LArTF (with some delay) and will remain in alarm even after the key is replaced unless it is acknowledged. If you are unaware of what access is being made in conjunction with this alarm, please contact LArTF and find out who is making the entrance. Then let the Run Coordinator know immediately of the access (by phone), if it was not cleared beforehand.

March 17, 2016

February 19, 2016

  • With respect to the header below titled "January 25th, 2016": Currently, all PUBS projects displayed on the monitoring GUI are enabled and should remain enabled. Shifters can resume following the usual PUBS monitoring procedure.

January 26th, 2016

  • When performing the "Is the Slow Controls archiver running?" check on the checklist, please be sure to refresh the web page that the instructions say to look at, and notify the slow controls expert if there is a non-zero number in a row labeled "Disconnected" on that page. As of now, the archiver should never be seeing any disconnected channels.

January 25th, 2016

  • Please note that currently, the "Binary File Drain [EVB --> Near1]" project is disabled and so appears grey in the PUBS monitoring GUI. This is the correct configuration currently and so the expert should not be contacted.
  • Added a missing step of "setup pyqtgraph" @ DataManagement shift procedure to start PUBS monitoring GUI.
  • Updated the instructions on who to contact when a particular alarm panel goes into an alarm state in the Slow Controls Guide.

January 11th, 2016

  • There is a new feature in the slow monitoring that will play a little alarm music on a web page when certain critical channels have an alarm. In order for it to work, you have to have the alarm sounder page open in a browser and the volume turned up. (Use cr02 in ROC-W for best results.) See the Alarm Sounder wiki page for more info.

January 6th, 2016

  • Very slight change in PUBS monitoring instructions for shifters ( DM_-_Shifters ). Now the monitoring GUI is run from the ubdaq-prod-evb machine, where previously it was run from the ubdaq-prod-smc machine. No other changes.

January 4th, 2016

  • New page for Workstation Layout demonstrating which windows should be open, and where they should be placed.

December 28th, 2015

  • RC trouble shooting wiki page RC_-_Troubleshooting is updated to include both xmit_trigger_ctr_diff_calc and xmit_frame_ctr_diff_calc under the same trouble shooting handling procedure.

December 6th, 2015

  • The shift procedure was updated to include information about the MicroBooNE cryo IFIX display.

November 13th, 2015

  • If you use any link to ECL in the Operations Wiki, be aware that due to the recent URL change the given default in the wiki is dbweb0, but you could be redirected to any number from 1 to 6. So if a link doesn't work, you have to manually change the number to the one you were assigned when you first logged into ECL on that computer.
  • The shifter checklist that needs to be filled has instructions on the far right side of the form. Maximize this window to ensure that you can read the instructions.

November 10th, 2015

  • Updated shift procedure (see main wiki page) to include instructions for the shifter checklist that was recently created.
  • Included new procedure for accessing ODH-1 areas at LArTF (wait 60 seconds before descending below the top of the stairwells) in the LArTF access procedures.

October 16th, 2015

  • Now you can shut up Verdi (a.k.a. the audible DAQ alarm) by keyboard interrupt.

October 15th, 2015

  • The shift procedure on the main Operations Wiki page has been updated, including shifter instructions about monitoring of the BNB as well as some other minor details.

October 14th, 2015

  • The Operations Wiki Updates page was created.
  • Please do not run kinit or kdestroy commands on any of the DAQ or control room machines anymore. We have cron jobs to renew special tickets for the ubooneshift and uboonedaq accounts every three hours.
  • In the "header" page of the Online Monitoring Lizard, we only care about the alignment of crate frame numbers w.r.t. trigger headers. Those should be aligned but are allowed to be offset by one. The crate event numbers are NOT expected to be always aligned w.r.t. trigger headers. Moreover, the Online Monitoring Lizard will raise a "big red box" alarm when there is a problematic alignment issue.
  • We have an audible DAQ alarm that is triggered when a run is stopped and won't automatically restart (as happens when the run control encounters an unforeseen failure mode). This indicates that shifter intervention is necessary to begin taking data again. If you have not been informed about the default run plan (either by reading the Run Plan page if it is up-to-date or being told by the Run Coordinator or DAQ expert), please contact the DAQ expert or Run Coordinator.
  • The run control is now making use of a VNC session on evb instead of one on ws01. Instructions in RC Guide have been updated.