Guide for Shifters

Running/restarting the Tracker DQM

This is done now by the MIDAS button "TrackerDQM" under "Programs". When you press this it will go yellow and count down to 1 and then pop up a window asking you run the command "/home/daq/startTrackerDQM" by hand on g2be1 but you should simply click OK to this and ignore the advice and simply wait about a minute until the button goes green. This is because the StrawTrackerDQMServer takes a long time to initialise before it connects to the MIDAS instance which is what ultimately causes the button to go green.

The DQM processes run as screens on g2be1: there are two one called StrawTrackerDQMServer and one called StrawTrackerDQMClient -- the server is the one likely to crash and if it does then on the DQM page it will have a red "NOT CONNECTED" button.

Running the Tracker DAQ

As for all the other frontends they can be started by clicking the corresponding start button in the programs tab of MIDAS. The general procedure for starting all the tracker frontends is :

  • Check the logic boards are powered by checking the "Straw Tracker Power" page (they should always be powered).
  • Start the StrawTrackerLVandSCXX frontends for the trackers present , with XX the tracker station number (if not already running).
  • Start the StrawTrackerHVXX frontends for the trackers present (if not already running).
  • Start the StrawTrackerDAQ frontend - this MUST be done last.
  • Once all the frontends are green on the status page the trackers are ready to take data.

Each of the frontends starts up in a screen on g2be1 with the same name as in the MIDAS page. All error messages are sent to MIDAS and can be seen in the messages tab. The screen can also be connected to in order to get more information in the case of a crash, e.g.

screen -r StrawTrackerDAQ
Additionally the output is stored in a log file on the PC that the corresponding frontend is running on :
  • StrawTrackerDAQ runs on g2tracker0 (in the counting room)
  • StrawTrackerLVandSC0X runs on g2trackerX-priv (in the ring) where X=2 for 7 o'clock tracker and 3 for 10 o'clock
  • StrawTrackerHV0X runs on g2trackerX-priv (in the ring)

See Troubleshooting for troubleshooting information. There is also a doc-db with some information available : TrackerDAQ: Shift Procedures

The start of shift checklist asks for the number of tracker modules running - this can be found by clicking on the "Straw Tracker Settings" from the MIDAS Status page. For each tracker in the FC7 box it shows the number of modules running. The total number of modules is the sum of this number from each tracker (or FC7).

While running there are several things that should be monitored regularly to ensure the data being taken is useful data. Most of these are on webpages described below.

Straw Tracker DQM

The straw tracker DQM can be accessed from the main DQM page (g2be2:3333) by clicking on the tracker under the subsystems menu or on g2tracker0:3344. Each page along with the things to check are described below.

Overview Page

The overview page has plots to confirm that the data is coming in and is not corrupted. Things to check :

  • There is tracker data in the event
  • The data is unpacking and there are no unpacking errors
  • There are hits in all the modules
  • None of the TDC buffers are almost full (>1000 hits)

Station Summary Page

There is a summary page for each tracker station showing the hits in each straw. Things to check :

  • The hits are roughly even across all straws with a slight increase close to the beam (right of the plot, higher straw numbers)
  • There are no channels with vastly more hits than the others

Time Islands Page

This shows the time islands, or hits grouped within a small time window and acts as a fast proxy to roughly the number of tracks in an event . In general the maximum number of hits in a track is 32 so mostly the number of hits per time island is lower than this.

Tracker Occupancy Page

This is like a tracker event display showing the straws hit in three different time windows - early, middle and late in the spill. Generally the occupancy is high at the start and decreases throughout the spill so at later times you can see hits that form tracks by eye.

Tracker Web Monitor

At http://g2tracker0:5002 there are web displays for the HV, water-cooling, slow-controls and vacuum.

Slow control monitor
The slow control monitoring page displays the variables recorded by the StrawTrackerLVandSC frontends. A screenshot is shown below. In general the distributions of the variables should stay pretty much flat :

  • the board temperatures stay flat between 30 and 40 degrees. The temperatures rising could be an indication of a problem with the air or water cooling. MIDAS will also thrown an alarm if the temperatures get too high.
  • stable currents and voltages supplied to the boards.

High Voltage Monitor

The high voltage monitoring page shows the status of all the high voltage channels per tracker. One should be looking for channels that are off or tripped. If all is well the first row of the status table will be green. At present the HV is off, pending the installation of the Ar/Ethane line, and so all channels are red (OFF). Tripped channels appear in yellow and channels that are on as expected in green. If there is a trip MIDAS will also throw an alarm which can be cross-checked at this page.

High Voltage trip

If there is a trip on one of the modules please make a note of the module (T[1-2] M[1-8] [U/V][1-2]) and call the detector on call. They will reset the HV, and in doing so will disrupt the monitoring system, so it is likely there will be another alarm, which can be reset. When they have finished, if you see a "HV Monitor not running" warning simply restart that particular HV monitor on the straw power setting page, which can be accessed from the main MIDAS page - NOTE: take car not to turn off the HV! only press the button that says restart monitors, if in doubt ring detector on call. When HV is fully restored the trackers plot on the main MIDAS page (orange and purple) will show hits in all layers - when the HV has tripped on a module there will be a gap.

Water Cooling Monitor

There is one webcam image per tracker, updated every 5 minutes showing the 8 spinning gauges measuring the water flow to each tracker.
If water is being provided the image of the gauge (due to the rotating needle) will be blurred. This is is presently the case for modules 0,5,6,7 where 0 is the leftmost tracker and 7 the rightmost for Tracker-2 ("T2") At C10. The water is not yet flowing to Tracker=1 ("T1") at C7.


Below some of the more common errors and their solutions are explained. In case of any errors an elog entry containing as much information as possible should be written and emailed to Becky Chislett, including :

  • All MIDAS error messages
  • Any additional information in the screen session
  • The run number and time
  • Any steps taken to rectify the problem

Cannot communicate via USB

[ USBManager :: readBackUSB ] Looped 10 times during USB read back without finding new data, exitting
[ AMC13-0/FC7-2/LB-0 :: checkSCConnection ] Cannot communicate with logic board via slow controls. Check (a) logic board is powered, (b) ethernet cables are connected, (c) USB connection information in ODB is correct, (d) another DAQ process (frontend/python script) isn't running
Critical error occurred
[ USBManager :: readBackUSB ] Looped 10 times during USB read back without finding new data, exitting

This usually means that the LB is not turned on. The status of the Low Voltage power supply can be checked on the straw tracker power page, or manually by logging onto the machine attached to that tracker (FC7-1 is on g2tracker1, FC7-2 is on g2tracker2, FC7-3 is on g2tracker3), finding the LV channel that the logic board reporting problems is connected to in the ODB or on the straw tracker settings page and then running the command :

python LowVoltage/ -lv 1 -c "status <LV channel>"

The board can be turned on and off using the buttons on the LV power page. It is also worth checking that there aren't multiple screen sessions for the StrawTrackerDAQ or StrawTrackerLVandSC.

StrawTrackerHV alarms

There are two categories:

  • an actual HV trip -- for this please call a tracker expert.
  • the StrawTrackerHV Monitor program is not running. This can be started by clicking on "Straw Tracker Power" on the MIDAS sidebar and then "Start HV Monitor" : C7/HV02 is Tracker-1 and C10/HV03 is Tracker-2. Sometimes this fails to start. If so page the DAQ expert.

The StrawTrackerHV02/03 programs are restarted automatically (on a 5 minute cycle) if they are idle for more than 5 minutes. If either of these programs are red/stopped for more than 15 minutes, page the detector-on-call expert. If they are red the run should NOT be stopped. These programs do not affect data taking or control the tracker HV. They just aggregate data into postgres and issue MIDAS alarms if there is a trip or the program reading the HV values from the CAEN units has stopped.

StrawTrackerDAQ starts falling behind on events

This was seen during the commissioning run due to noise in the ring causing the TDCs to temporarily drop the clock. During the shutdown this has not been observed and hence has not been fixed so it is possible this will crop up again. In this case :

  • Stop the run
  • The CCC will throw an error about not receiving all the triggers
  • The MIDAS message page should contain an error from the StrawTrackerDAQ (or it can be seen from the screen session) :

LB-X, LB-Y did not respond to all triggers, please turn off the StrawTrackerDAQ and StrawTrackerLVandSC frontends and go to the Straw Tracker Power page and turn them off and on again. Then restart the frontends.

  • Stop the StrawTrackerDAQ frontend
  • Stop the StrawTrackerLVandSC frontend
  • Power cycle the LBs listed in the error (and ONLY these ones) from the straw tracker power page
  • Restart all the MIDAS frontends
  • Start the run

This may have to be done a few times to get the boards in a good state. But generally once they power up in a good state they run well.