After the issue with lost NuMI data this past weekend, there was a lot of discussion about the quality of our shifting procedures. This week the discussion was focused on ideas/proposals to improve shifters performance to diagnose issues promptly.
The conversation was centered around improvements which can be made on plots and checklists which are currently available to the shifters.
Main suggestions on which people showed general agreement:
- Introduce small checklist with a few yes/no or checkmark items to be filled every hour. For example: Have you switched through the OnMon/Shift plots? Have you checked that trigger scalars are advancing? Is the event display showing normal behavior? etc. The intention would be to have short checklists for shifters to acknowledge of having done basic monitoring once per hour.
ACTION => Bring this up in operations meeting to request approva
- Suggest some kind of feedback be given to shifters after their shift. Either a mention by the run coordinator or watchdog group for shifters who did a good job or email contact with suggestions after their shift.
- Suggest watchdog meeting be strongly encouraged or even required to attend by shifters.
- Mandatory NOvA shifter training. This idea had a lot of support and a few people (Mat, Michael, Justin and Fernanda) volunteered to contribute content for a training video in the style of other Fermilab training requirements.
ACTION => Bring this up in operations meeting to request approval and start drafting a tutorial.
- New Trigger (NuMI and cosmic) rate and spill rate plot on nearline. Strong suggestions to have this new metric in place of the track fractions plot on the front page as well as the nearline checklist.
ACTION => Bring this up in operations meeting to request approval
- Start OnMon wiki page explaining the plots in the shift folder. No volunteer has been found for this but Michael has offered to contribute some content when a volunteer for this task is found.