Project

General

Profile

Trolley Operation Manual ( Only for Experts)

Important Notes

  • Do not operate the trolley unless you are trained
  • No hall access is required. If you have to go in for any reason:
    • Wear dosimetry badge when going into the ring to do the trolley checklist
    • Need 2 people to go into the ring for trolley checklist (MR region requirement: need two people to access MR region)
  • If something strange happens to the trolley system (unexpected trolley alarm such as "trolley power trip", etc.) abort trolley motion and contact a trolley expert right away (Simon & Suvarna)
  • Trolley abort switches off the trolley motor so you have to turn it on before clicking on the trolley motion switch again
  • Alarm: Cannot disable by clicking reset. First go to the "Trolley Control Page" from the Midas Status Page, and click the "Reset Alarms" button, and then reset the Midas Alarm. Do not ignore any alarms!
  • On the trolley control page there are error counters for "Missed Frames" and "Checksum Errors". The counters turn red if they are larger than 5. If this is the case at the end of a trolley run, add a comment in the ELOG.

Getting ready: Trolley Check List

  • Beam off and keys:
    • If in controlled access: Call main control room and get 2 keys (1 for yourself and 1 for a shifter/operator, checking out the controlled access key automatically stops the beam). If you actually go in side the ring (not needed in normal operation mode)fill the details in the control access sheet.
    • Hold on to your keys until you finish the trolley run
  • Main DAQ: Ask shifters to stop the run and do the laser study following instructions in the shifter's wiki page. Note down the main daq run number.
  • Ramp down (or ask shifters) Quads and Kickers and retract the collimators:
    • Kicker/Quad inhibit: after ramping down the kicker and quads as per the above instructions, click the 'Quad/Kicker Inhibit' switch under 'Manual Motion Control' in the Trolley Control page. It should go red and say ON. After ~20 seconds the kicker GUI will update to say 'Trolley inserted! Triggers stopped'. This prevents the kicker triggers from being enabled if the trolley is in the ring.

The trolley checklist can be opened in the browser on g2field-server-2:5000. On the control room field ops stations, it is localhost:5000.

Make sure all entries are green and have a look at them. The "Last Update" column tells you when the displayed values are last written to the various databases. If a value is too old it turns yellow.

Complete the Trolley Checklist in the ELOG. A screenshot of the checklist above needs to be uploaded. This checklist needs to be APPROVED by the runco before you move the trolley.

  • Quad and Kicker Status: these values are only updated whenever the buttons are clicked. It's ok that these values are "old"
  • the "Collimator & Fibre Harps" and the "Plunging Probe" lines need the "Galil" frontend to be on. The frontend can be started without stopping a field run.
  • The magnetometer status needs to be checked with the runcos (at the moment)
  • Troubleshooting: if the IBMS3 shows "old" you might need to restart the application

Restart the Script

In case you need to restart the script, ssh to newg2@g2field-server-2,


screen -r trolleyChecklist

Stop the program with CTRL-C and restart it

python app.py

in the folder "/home/newg2/trolleyChecklist".

QuadQUI

The "old" quad electronic shows sometimes pulser faults during a trolley run. This is nothing to worry about but annoying. In order to suppress them, we set the checkmark "Trolley run in progress" in the quad QUI whenever we perform a trolley run.

General Instructions for Trolley Motion Control

  • For trolley runs, we start new Field DAQ runs for each stage.
  • Important Azimuth Positions: Drive CW: 262 deg, Garage: 175 deg, Drive CCW: -98 deg
  • The trolley cannot pass the drive position (262 deg/-98 deg)
  • Turn on Gail control and trolley interface under programs and also power cycle trolley DQM. Everything except plunging probe & Plunging probe DQM are generally turned on.

  • Open "Trolley Control Page" from the Midas Status page in another screen
  • Make sure tension is below 3
  • Make sure Trolley Motor Forward and backward limit status is at 1. If any one of them is zero, the trolley motion alarm will trip, and all motors are turned off.
  • Make sure the field configuration is consistent with the daily run plan: right surface coil and ps-feedback set point. (See also https://gm2-docdb.fnal.gov/cgi-bin/private/ShowDocument?docid=10285)
  • Be familiar with the Alarm handling.

Normal Trolley Run Operation

Documentation

Trolley Insertion into the Storage Ring

  • Start the run, use “Trolley insertion from the Garage” as comment.

  • Click Trolley ON and Garage ON. The motor control will show that they are ON
  • Click on Garage Insert trolley: Limit switch will become 1 for Garage forward and velocity will be above zero ( It will take about 5 minutes for the trolley to fully insert)
  • Turn on the trolley garage camera to see the trolley in the garage. Enter kerberos credentials if necessary. Then ssh newg2@g2field-fe1 , login and type cheese, then press enter. The camera window will pop up (see above)

  • DQM will show steps in cable position under Motion section of the trolley DQM page (g2field-be:3335 on the field station)
  • When the trolley is fully inserted: Garge velocity will read zero and Reverse limit status of Garage will read zero ( limit switch might not read zero all the time)
  • Turn off garage. Garage encoder count should be less than 2000.
  • Stop the Run.

Trolley Garage Camera Operation

  • Turning on the camera: Enter kerberos credentials if necessary. Then ssh newg2@g2field-fe1, log in and type cheese, then press enter. The camera window will pop up (see above)
  • Turning on the LEDs: If the garage is not illuminated, the LED needs to be power cycled or turned on. Navigate to TrolleyLEDControl directory (cd ~/TrolleyLEDControl). Execute the command ./trolleyLEDcontrol 1 to turn on the LED. Using an argument of 0 turns off the LED. Argument r or R resets the LED.
  • Camera selection: There are two cameras on the trolley garage. The one needed for parking points at the area where the black sharpie mark on the trolley rails is visible (and the arrow on the trolley) for parking. To select this camera, in the cheese app select Edit > Preferences, which shows a few pull-down menus.  From the Device pull-down menu, select /dev/video0 to see the upstream side camera that we need to verify the trolley location before parking.
  • For instruction on how to remotely power cycle the cameras see: https://muon.npl.washington.edu/elog/g2/Trolley+system/722
  • There is a "Camera Power Cycle" button on top of the "Trolley Control" page
  • Alternatively: The cameras can be power-cycled by typing "TrolleyCameraPowerCycle.sh" anywhere on g2field-fe1 (It lives in ~/bin/). This power-cycles both cameras and also resets and switches on the LEDs.

Garage to Drive CW

  • Start the Run with comment "Trolley run to Drive CW". After the run starts, click on "Trly. To Drive CW" button
  • Check DQM (g2field-be:3335 on the field station) for FIDs (make sure they are updating), Position, Barcode etc. to make sure everything is fine. Keep an eye on ps-feedback DQM ((g2field-be:3336 on the field station) and make sure the mean field is with in the green bands.
  • It will stop at around 262.4 degrees. Velocity will read zero. Record the azimuthal position of where the trolley stopped, in the UW Elog.
  • Once the trolley is within ~2deg of the drive the barcode DQM (http://g2field-be:3335/trolleyBarcode) starts to display "Barcode Drive Abs 1" and "Abs 2".
  • Check on the trolley barcode DQM (http://g2field-be:3335/trolleyBarcode) that you see the structure highlighted in the following picture plus one abs barcode with atleast 4 peaks down in "Barcode Drive Abs 1"

    If not move manually closer in very small steps of +0.005 (Manual Motion Control: Trolley (Rel. Deg) ).
  • Take a screenshot of this barcode DQM page and attach it to the trolley run ELOG.
  • Stop the Run

Drive to Drive CCW
  • Start a new Run with comment "Drive to drive CCW" and click on "Trly. To Drive CCW".
  • It will take about an hour to reach other side. Check vital DQM plots once in a while, keep an eye on ps-feedback DQM and make sure the mean field is with in the green bands.
  • It will stop at around -98 degrees. Velocity will read zero. Record the azimuthal position of where the trolley stopped, in the UW Elog.
  • Check on the trolley barcode DQM (http://g2field-be:3335/trolleyBarcode) that you see the structure highlighted in the following picture.

    If not move manually closer in very small step of -0.005. (Manual Motion Control: Trolley (Rel. Deg))
  • Take a screenshot of this barcode DQM page and attach it to the trolley run ELOG.
  • Stop the Run

Drive to Garage CW

  • Start a new Run with comment "Drive to garage CW" and click on "Trly. To Garage".
  • It will take about 45 minutes to reach the garage. Check vital DQM plots once in a while, keep an eye on ps-feedback DQM and make sure the mean field is with in the green bands.
  • Trolley will go to 179 and come back to 175. Velocity will read zero. Record the azimuthal position of where the trolley stopped, in the UW Elog.
  • Stop the Run

Parking

  • Start a new run with comment "Trolley Parking"
  • The black arrow should be approximately in the middle of black mark. If not try to do move trolley manually in small steps. (Manual Motion Control: Trolley (Rel. Deg)). A typical trolley alignment for parking will look like this:

  • System needs trolley position to be with in 175.00+- 0.5 for the parking to work
  • Switch the Garage On
  • Click Garage park trolley —> Trolley might slide left a little bit, it is perfectly normal. It takes about 5 min to park
  • When fully parked, gargae velocity will read zero and forward limit status for Garage turns to zero.
  • Turn Garage off and Trolley off
  • Stop the run, go to programs and stop Galil control, trolley interface and trolley DQM
  • Start a new run with comment "Normal run"
  • Check the midas messages. If there were any "Skipping trolley data frame at iteration" messages (red) copy them to the ELOG.

After Parking

  • Perform the "Trolley Parked Checklist" in the ECL
  • Let shifters know that you are done with the trolley run. Remind them to uncheck the "Trolley run in progress" flag in the Quad GUI. Remind them to uncheck the 'Quad/Kicker inhibit' switch in the Manual Motion control on the Trolley Control page.
  • Return the control access keys by calling MCR. If beam is needed RC/shifters will let them know. Return other keys to person in charge at the control room.
  • Post another fermi elog saying that you finished the trolley run

Trouble Shooting and Emergency Response

  • These instructions should be familiarized by the operator. In some situations, operators may not have time to read through these instructions and then react.
  • At ANY time, if the Galil frontend crashed and CANNOT BE RECONNECTED, push the CRASH BUTTON ON THE WALL.
  • If a "Motion Trip" alarm is triggered upon switching on the Galil Frontend, check the Limit Switch Status. If the trolley motor limit switches are "0", it indicates that the Collimator and Fiber Harp controllers are powered. Then go to the ring to power them on and start the slow control frontends for them in the Main DAQ. If they cannot be switched on properly, contact the trolley experts.
  • During the motion, if the "Motion Trip" alarm is triggered, check if the motion is stopped and all motors are powered off. If not, use the "abort" button to stop the motion.
    • Check the tension. If the tensions are all normal (<3), the alarm may be caused by some transients in the interlock system. Log the incident to the Elog, resume the motion, and watch the tension carefully. If the tension rises sharply towards 4, abort the motion and treat it as the "High Tension Trip"
    • If the driving tension is high (>4), turn on the trolley motors, back up by 0.1 deg to release the tension. Log this incident and write down the location. Contact the field on-call and trolley experts for further instructions. DO NOT KEEP OPERATING THE TROLLEY IN THIS CASE!
  • Better keep watching the NMR signals repeatedly, and if anyone of them disappeared, contact the field on-call.

OLD Trolley Check List (Activities in the ring are marked as Bold, which are IBMS-3, Kicker Magnetometer,Plunging Probe, Fiber Harp and RF System)

Fill out the electronic trolley check list (in the Fermilab ELog drop down menu). Check each entry in the checklist and attach screenshots (or pictures using phone) for the Kicker GUI, Quad GUI, Collimator control, FiberHarp control, Plunging Probe coordinates, RF units(on the false floor) and Q024 & Q025 after checking all of the respective systems as detailed below:

  • Beam off and keys to go into the ring:
    • If in controlled access: Call main control room and get 2 keys (1 for yourself and 1 for a shifter/operator, checking out the controlled access key automatically stops the beam). Fill the details in the control access sheet. Get 2 MR keys from person in charge at control room.
    • If not in controlled access (Beam is already off in this case, so get keys for ring access): Get 2 High Bay keys and 2 MR keys from person in charge at control room. (1 for yourself and 1 for a shifter)
    • Hold on to your keys until you finish the trolley run
  • Main DAQ: Ask shifters to stop the run and do the laser study following instructions in the shifter's wiki page
  • Quads off: Go to quads GUI. Click on "Set voltage to zero" and pulse it at Burst mode. Make sure the voltage is coming down. Stop pulsing when voltage is at least below 50 Volts(Note: Values are in KV on the GUI, It might take some time to come down to 50 volts so you can proceed with the rest of the checklist and stop pulsing before the trolley run). Take a photo and close the quad GUI ( There will be ./startGUI.sh running on one of the terminals, you can do Ctrl+z to close the GUI)

  • Kicker off and Triggers stopped:
    • Turn Off: Slide all 3 kicker voltage sliders down to zero. ** Triggers stopped: After sliding down the voltages to zero, click on enable stop triggers at the bottom of the page and make sure the kicker GUI shows that the triggers are stopped in pink at the bottom of the page. Take a photo. Make sure that the kickers are not pulsing (Enabled trigger stop trigger button is Red)

Collimator

  • Collimator : Main DAQ —> collimator control. On the right side click on retract and wait till status shows retract 1, do one at a time. Make sure the retracted status for all 5 collimators are at 1. Also make sure the demand status is 'retracted' for all collimators. The collimator control front end must be ON. If it cannot be turned on, it indicates that the hardware controller is not powered, and in such case the trolley motion hardware interlock will not be cleared, and a motion trip alarm will be generated when the trolley motion control frontend is switched on. Take a photo of the control page.

  • Fiber harps: Make sure Fiber harp control is running (green) in the Main DAQ status page. Then , Main DAQ —> Fiber harp control, same procedure as collimator. Make sure retracted status for all 4 are at 1. Also make sure the request rectract status is 1 for all 4. The fiber harp control frontend must be ON. If it cannot be turned on, it indicates that the hardware controller is not powered, and in such case the trolley motion hardware interlock will not be cleared, and a motion trip alarm will be generated when the trolley motion control frontend is switched on. Take a photo of the control page. Also double check if the fiber harps are retracted while you are in the ring.

  • Q024 and Q025: Go to the power supply room and locate the power supplies labeled Q024 and Q025 (location is shown in the diagram below). Verify that they are ON (Both A and B), with ~400 Amp (420 Amp) on the DC current read back for Q024 (Q025). The display on Q025A may be flaky, but it should be ON. Take a photo of Q024 (A&B) and Q025 (A&B).

  • IBMS3: Go into the ring and check that the sliver color screw holding the IBMS3 is in the retracted position and can be seen through the glass window. ( need 2 people present anytime you go into MR region). If retracted, this is what you would see through the IBMS3 glass window.

  • Plunging probe: Take the MR flash light from the MR-cabinet and make sure the plunging probe is out of the way for trolley by looking through the glass flange ( replace the aluminum foil on the flange when done).* Also check Plunging Probe Control Page in the field DAQ and make sure (x,y,z) = (-4005±10, 792±10, 543±5 ). Take a photo of the control page.

  • Kicker Magnetometer: Make sure the kicker magneometer is in the retrated position and held with rods and cable ties. (The kicker magnetometer is located just to the right of Bazooka 1. Lift up the insulation from the Velcro at the bottom to see the kicker magnetometer, and reattach the insulation after checking )

  • RF system is off: Check on the RF system rack (right across from IBMS-3) that RF power amplifier 'power' and 'enable' switches are turned off. There are a total of 12 units. Top 4 units are shown in the image below (Unit between 4 and 5 is not in use). Take a photo, you might need 2 photos to get all 12 *