Operations Procedures


These procedures are a copy of those in docdb-42298. The version below is V1.0 (last updated Dec 30, 2019).

They may be edited by Run Coordinators or Operational Managers as appropriate, with the version in doc-db periodically updated to reflect the changes.

1 Operational Hierarchy

NOvA Operations Manager
Test Beam Run Coordinator
Test Beam Deputy Run Coordinator
Test Beam Shifter NOvA On-Call DAQ Expert

1.1 Definition of Roles

NOvA Operations Manager: Ultimately responsible for all NOvA operations, including Far Detector, Near Detector and Test Beam. Approves all operational decisions required of Test Beam.
Test Beam Run Coordinator (RC): Responsible for operations of the Test Beam experiment, defines the run plan describing the conditions required for running and collecting data and is responsible for all data-taking.
Test Beam Deputy Run Coordinator (DRC): Provides assistance to the run coordinator. Manages the day-to-day operations and is available as the first point of contact for the test beam shifters.
Test Beam Shifter: Operate the experiment and monitor the detector and data- taking whilst on shift. Responsible for ensuring good-quality data is collected by the test beam experiment.
NOvA On-Call DAQ Expert: Experts available 24/7 for the NOvA shifters to contact in case of DAQ problems. Will provide additional support for test beam if required.

1.2 Appointment to Roles

Test beam shifters are members of the NOvA collaboration with an interest in running the test beam experiment and contributing to the project.
Run Coordinators are collaborators at postdoc or higher level with significant expertise in the test beam experiment. Run Coordinators are approved by the Operations Manager and the Spokespeople.
Deputy Run Coordinators are collaborators with significant expertise in the test beam experiment and experience in operations. Deputy Run Coordinators are approved by the Run Coordinators, the Operations Manager and the Spokespeople.

2 Relationship with NOvA Operations

2.1 General

All necessary communication between Test Beam operations and its potential impact on NOvA operations will be at the Run Coordinator level. The Operational Manager can be included if required to resolve any problems.
Test Beam operations will be discussed at the weekly Monday operations meeting, following the culmination of the regular NOvA agenda (which will as much as reasonable possible limited to half an hour).

2.2 Test Beam Run Coordinator and NOvA Run Coordinator

The Test Beam Run Coordinator and the NOvA Run Coordinator will communicate any relevant information which may affect the operations of the other.
Additional specific details can be added if and when required.

3 Responsibilities

3.1 Test Beam Run Coordinator

  • Clearly define the operational planning and maintain active involvement in all operations.
  • Be present at, or nearby, Fermilab, available to be on site when required and available on-call 24/7.
  • (At least initially) attend all (non-trivial) controlled accesses.
  • Interface with other operational personnel as often as necessary.
  • Interface with outside groups as and when required; be the first point of contact for NOvA Test Beam operations (including with MCR).
  • Ensure all shifts blocks (Deputy Run Coordinator, Control Room Shifters etc) are filled. Deal with situations which may arise related to shifts which cannot be taken.

3.2 Test Beam Deputy Run Coordinator

  • Manage operations of control rooms, including ensuring incoming shifters are prepared before their shift, briefed periodically and understand their roles.
  • Interface with shifters and be the first point of contact for expert activity and resolving any uncertainties in operations.
  • Communicate with RC any relevant details and ensure RC is kept informed of developments.
  • Ensure the run plan is carried out as defined by the RC.
  • Actively manage operations and communicate problems or suggestions to RC, who may then approve changes.
  • Be present at Fermilab, available for controlled accesses, and available on-call 24/7.

4 Control Room Shift Procedures

4.1 Beginning of shift

  • Check in on ECL.
  • Debrief with the previous shifter to understand the current status of operations and complete required knowledge transfer.
  • Call MCR (x3721) to check in as the new NOvA Test Beam shifter and give your location (either MCenter or ROC West).
  • Receive contact from the NOvA shifter, including their location and the status of the other NOvA detectors, so you feel a sense of collaboration and are aware of any conflicting problems.
  • Ensure all machines are logged in, the VNC sessions are connected and the required information is displayed.
  • Ensure the Zoom meeting is active.
  • Familiarize yourself with current running conditions and any ongoing expert
  • Fill out start of shift form in the ECL in the first hour of the shift.

4.2 During shift

  • Follow all instructions on the NOvA Test Beam shift wiki and as detailed in the shift manual.
  • Document everything in the ECL.
  • Contact Deputy Run Coordinator if something is unclear, you need help with something or there is a problem with data-taking which you can’t fix.
  • Monitor the data-taking:
  • Ensure both DAQ are running and collecting data.
  • Beam: ACNET beam intensity plot shows smooth beam delivery.
  • No alarms in any column on detector dashboard or in beamline message viewer.
  • Detector and beamline synoptics not in alarm.
  • Monitor DQM and event displays to ensure good data are being collected.
  • Fill out hourly control-room checklists.
  • Coordinate with collaborators via phone and Zoom.
  • Towards the end of your shift, fill out the Shift Summary form.

4.3 End of shift

  • If the next shifter doesn’t arrive, contact the Deputy Run Coordinator. Don’t leave without the next shifter being present.
  • Communicate to next shifter to relay important information.
  • Check out on ECL when you leave.

5 Test Beam Deputy Run Coordinator Procedures

5.1 Before beginning DRC term

  • Review recent operational activity via the ECL and by communicating with Run Coordinator/previous DRC.
  • Discuss with the Run Coordinator the plans for operations during your DRC shift.
  • Ensure all shifts are covered during your week as DRC.
  • Email all shifters to introduce yourself and inform them of the procedures and
    the personnel on-call whilst they are on shift.

5.2 During DRC term

  • Be available with your phone accessible during the hours of your shift.
  • Interface with the Facility to maintain communication and inform each other of any relevant information. Pass anything relevant on to the RC.
  • Support the shifter by answering general questions and helping with specific problems.
  • Be available to be a general expert and also to address detector or beamline DAQ issues when they occur.
  • If necessary, following consultation with the shifter, communicate with the Run Coordinator if there is anything you are uncertain about.
  • Document all expert activity in the ECL.
  • Ensure the run plan is enacted as defined by the Run Coordinator.

5.3 After DRC term

  • Communicate any necessary information with the Run Coordinator and the next DRC.
  • Provide an update at the next Operations meeting.

6 Interaction with Main Control Room

6.1 General

The NOvA Test Beam experiment will collect data in the MCenter beamline during 2019 and 2020.
NOvA Test Beam will have shifters monitoring the experiment 24/7.
NOvA Test Beam has two control stations at Fermilab; one in ROC West (note: independent of the other NOvA control station) and one in MCenter. Shifts may be taken from either of these locations, and from nowhere else.
NOvA Test Beam will operate three shift blocks per day (aligned with the regular NOvA shift patterns): ‘day’, 8am-4pm; ‘swing’, 4pm-12am, ‘night’, 12am-8am.

6.2 Interactions during shift

Beginning of shift
  • The NOvA Test Beam shifter will contact MCR (x3721) to check in. They will
    report as the NOvA Test Beam shifter and will give their location:
    • ROC West (x5413)
    • MCenter (x3726)
  • Both MCR and the NOvA Test Beam shifter will enter this information in their respective ECLs.
  • The status or requesting of beam or current to the NOvA test beam analyzer magnet may be discussed.
During shift
  • The NOvA Test Beam shifter will monitor data-taking and contact MCR if required.
  • MCR may contact the NOvA Test Beam shifter at any time. Please ensure to contact the correct control room.
    End of shift
  • The NOvA Test Beam shifters will check-out in the ECL and communicate
    important information to the next shifter.
  • No requirement to contact MCR.

7 Interaction with Facility (FTBF)

Both the RC and the DRC will maintain close communication with FTBF. Day-to- day interaction may involve only the DRC, unless there are more significant issues which the RC should be aware of or there are downtimes scheduled beyond the current DRC shift.

8 Resolving Problems

8.1 Shifter Procedures

The Test Beam Deputy Run Coordinator is the usual first point of contact for the control room shifter in the event of a problem. The DRC will address the problem and may call in additional help from the On-Call DAQ Expert or the Test Beam Run Coordinator.
If the DRC is unavailable, contact either the On-Call DAQ Expert or the Test Beam Run Coordinator. If the problem is related to the NOvA DAQ (not beamline), the NOvA On-Call DAQ Expert may be contacted. If the issue is related a test beam specific system, such as beam or beamline instrumentation, or if you uncertain, contact the RC.

8.2 Deputy Run Coordinator Procedures

If contacted with a problem, address the issue concerned.
If unable to address the problem, contact any of the additional experts listed on the Blue Sheet. If the problem is test beam specific, contact the RC.
If there is an issue which you don’t feel equipped to handle, contact the RC, who will provide assistance. Do not feel you need to deal with all situations yourself.
If there is a problem endangering the health of the detector or could potentially hard the experiment, shut down the detector and contact MCR to request the cessation of beam.

8.3 On-Call DAQ Expert Procedures

If contacted with a problem related to the test beam detector, proceed as you usually would when handling DAQ issues.
The relative priority of our detectors is: Far Detector, Test Beam, Near Detector.
If you are unable to resolve the problem, or you need additional help, inform the DRC. The RC may be contacted if additional support is required.
If you are contacted whilst already busy recovering the Far Detector, inform the DRC you are unable to currently help. If you are dealing with a Test Beam problem which is not imminently resolvable and you receive a call regarding the Far Detector, contact the DRC to let them know you cannot continue until the new issue is resolved. At any point, the RC may be contacted to provide additional assistance.