Project

General

Profile

YOUR SHIFT » History » Version 15

« Previous - Version 15/40 (diff) - Next » - Current version
Benton Pahlka, 09/01/2011 01:35 PM


YOUR SHIFT: Important notes for the shifters:

Where to go for help

Shifter's ToDo List

• ensure the highest uptime of both Far and Near detectors
• diagnose and fix problems quickly and efficiently
• alert the system experts and work with them in case of problems
• perform regular inspections of the data taken, monitor detector systems and environment conditions by filling out shifter's checklists

The approximate shifter's ToDo list:
1. Be at the control room early. Arrive 30 minutes before your official shift start time.
2. Carefully read all CRL entries entered in the last 24 hours (if it's your first day on shift, then browse thorugh last several days, paying attention to all special tasks accomplished, problems discovered and fixes performed).
3. Begin filling Start_Shift.htm form in the Control Room Logbook.
4. Restart both run control GUIs on minos-rc.fnal.gov screen. A window will appear asking for the current operator's name. Type your name in the appropriate box. Restarting the RC GUI ensures that you are identified as the current shifter throughout your shift.
5. Find out who are the shift physicists and crew supervisor. The shift schedule is posted at http://www-numi.fnal.gov/internal/fardet/shift_index.html
6. Talk to either the previous shift physicist or a crew member and discuss what happened during their shift. Note any new problems found or old ones fixed. Inquire whether there were any hardware changes, special activities etc. Ask if there are any requests for special data taking and remind them to release DAQ control before they leave. Then wish them a safe drive home.
7. Take control of the DAQ using the run control GUI.
8. Proceed with filling out shifter's checklists (described in the next section); try to find and diagnose hardware problems; record your findings in the online logbook.
9. Regularly check DAQ, DCS status webpages (keep both of them open on top of one of the multiple control room screens).
10. Spend time looking at the real time events using the event display.
11. Browse through Online Monitoring plots, not just the ones from the checklist.
12. Record anything happening in the online logbook.
13. Enter all new problems into the JIRA issue tracking system.

Shifter's Checklist Forms

  • Start_Shift
  • DCSChecklist_Near
  • DCSChecklist_Far
  • OMCheckListNear
  • OMCheckListFar
  • End_Shift Write a shift summary, list all the problems discovered and suggestions on how they should be fixed.

The Online Monitoring checklist form is subdivided in two parts – Om1CheckListFar.htm (Om1CheckListNear.htm) and Om2CheckListFar.htm (Om2CheckListNear.htm). In order to fill them out, the shifter needs to start the Online Monitoring Histogram Viewer for the appropriate detector and browse through all plots in the Checklist folder. It's important to let the online statistics accumulate, so make sure that the data run has been running for at least two hours before proceeding with the checklist. To answer the question whether a certain OM plot is “good” or shows problems shifters should consult the OM manual at http://www-numi.fnal.gov/numinotes/restricted/pdf/numi0903/numi0903.html and browse through the Online Monitoring Gallery at http://minos-om.fnal.gov/cgi-bin/photo/index.cgi – a collection of sample OM plots with comments and guides how to find problems

Please click "Update" or "Latest" (or "Current") on NuMI elog before inserting plots

NT02 has been installed, remember it's the target that ran for 3 years and has some serious mass depletion, also we will be running at full power so we need to be diligent about monitoring.

Job List

  • You should have a pair of shoes with you. Sandals, flip-flops, high heels (!) and the like are not appropriate attire for underground access, and you may need to go underground.
  • The Far Detector needs to be notified of beam downtime as follows.
    If the beam will be down for long enough from 07:30-17:30, please notify the mine crew.
    • If there is downtime of at least half an hour notify Soudan.
    • If you learn of an all-day shutdown coming in the near future, inform the Run Coordinator and Soudan.
  • Please stop and restart OM Gui at the beginning of your shift, it leaks memory.
  • When filling in the Near Detector OM check-list please make sure that the socket is "open", and that the status canvas gives a current time stamp.
    • When the MINERvA shifter completes their ND OM checklist and shuts down their monitoring window it closes our connection.
  • The 'Big Green Button' shows file write latency. It flashes orange if this goes over 3 seconds, and turns red if the file is badly out of date. In this case, restart the Beam data logger, as described at BEAMDATA
    • The 'Big Green button' should no longer be getting stuck, showing stale data. Please restart the Button promptly if it gets stuck, and note the time in the ECL.
  • You need to post the Beam Shift Plots using NUMI Elog and Minos CRL sessions on minos-acnet, from the files in /home/minos/BeamPlots . The NFS server which made these available in /misc/mcr/... has been retired,
  • If G:RD1224 comes into alarm at 14:30, this is expected. If the alarm won't clear after 14:45, call MCR and ask them to remove multiwire 112 from the NuMI beamline. An alert box will pop up on minos-om console to remind you.
  • Please check your own training at: [training status|http://minos-om.fnal.gov/cgi-bin/fluffy.cgi]. Has it expired? Then renew it immediately. Is it about to expire? Then renew it so it won't lapse. Shifters are expected to be able to go underground if necessary.
  • Please check the Run Control GUI, DCS status pages, and JAS regularly for warnings. The shifter is the 1st responder to these error conditions and should notice alarms on the time scale of minutes (not hours).
  • Please watch the LCW output temperature on the ND DCS Magnet Status section. We have had problems with a water filter clogging up, which will take down both the ND and the NuMI absorber if it stopped up. If the temperature gets very high (>75) or increases quickly (so that it will reach 80 F in less than a day) call the Run Coordinator, then John Voirin or Bill Moorhouse. The temperature regulation is set at 67 F.
    • This will also restrict cooling to the NuMI absorber. Please let the NuMI Beam Coordinator (Sam Childress or Peter Lucas) that there is a problem so they can watch that temperature.
  • Due to budget cuts, we are able to access the far detector only during regular hours (weekend days are still ok) except in emergencies, where loss of data doesn't count as an emergency.
  • The Soudan On Call Phone number will not work at night. Instead, call Bill Miller or Jerry Meier, plus the run coordinator (Greg or Chris) and we will figure out if it's a real emergency or if budget cuts will be costing us data.
    • If Bill doesn't answer his cell, try his home number. If he's at home, his land line works and cell coverage doesn't dip down next to the lake.
  • The ND magnet can now be controlled remotely (E:NDMAG on ACNET). This includes changing polarity, turning the power supply on and off, and setting the current (a setting value of 4990 produces a current readback of 4973 amperes). The setting is an unsigned number; the current readback is a negative number. There is a status bit that indicates polarity. For further information see [this link|http://www-numi.fnal.gov/Minos/ControlRoom/nd_magnet.html].
  • If Firefox starts giving errors, then restart it.
  • Answering the telephone
    • Please let people know where they've called and who you are! E.g.
    • "MINOS Control Room, this is Fred Jones" would suffice.
    • Please try to always answer the telephone in a timely manner. If you need to step out for a few minutes check the caller ID on the phones when you get back. Call back all numbers that went unanswered.
  • Put a note in the logbook when you call experts or the run coordinator, at least the first time in a thread.

Swing and Owl shifts are no longer covered by Minerva

The swing and owl shifts are not currently being run by the Minerva shifters. For reference, here are the instructions that applied in that situation:

The MINOS shifters should arrive at the control room at the start of their shifts, and should:

  • Deal with any problems they find
  • Go through the checklists with the previous MINOS shifters
  • Bring themselves up-to-date on any outstanding issues or problems likely to occur during the shift. Make sure the Minerva shifters are away of these, and explain what they should do if the problem arises.

Since there is now no overlap between MINOS shifters, it is vital that regular use is made of the CRL to pass on information, and ensure the following shifter knows the status of the detectors.

The MINOS on-call shifter should make a logbook entry from home any time they are called. They should log:
  • What caused the call to be made
  • What action they took
  • Any action they told the Minerva shifter to take
  • Any phonecalls with MINOS experts they made or received.

Minerva currently doesn't have access to the MINOS CRL. They should use the blue electronic post-it note on the minos-om. They should log in this any interventions with the MINOS systems, or relevant information such as phonecalls with MINOS experts. The MINOS day shifter should transfer this information to the MINOS CRL at the start of their shift.

  • Congratulations you made it this far! Please note in the CRL logbook that you actually read the memopad and know what to do on shift.