Trolley Operation Manual ( Only for Experts)¶
Refresh video: https://gm2-docdb.fnal.gov/cgi-bin/private/ShowDocument?docid=24498
Trolley Zoom may be used at any time: https://fnal.zoom.us/j/96627867397?pwd=U0hKRVBOQlJUbTRZcGwxUERXYU9tdz09
- Do not operate the trolley unless you are trained
- No hall access is required. If you have to go in for any reason:
- Make sure the RWP for the hall access is signed if you need to go into the hall for some reason (need to sign every month).
- Wear dosimetry badge when going into the ring to do the trolley checklist
- Need 2 people to go into the ring for trolley checklist (MR region requirement: need two people to access MR region)
- If something strange happens to the trolley system (unexpected trolley alarm such as "trolley power trip", etc.) abort trolley motion and contact a trolley expert right away (Simon (733 630 6125 & Peter)
- Trolley abort switches off the trolley motor so you have to turn it on before clicking on the trolley motion switch again
- Alarm: General Field Alarm_Handling. Trolley alarms cannot be disabled by just clicking reset. First, go to the "Trolley Control Page" from the Midas menu, and click the "Reset Alarms" button, and then reset the Midas Alarm. Do not ignore any alarms!
- On the trolley control page there are error counters for "Missed Frames" and "Checksum Errors". The counters turn red if they are larger than 5. If this is the case at the end of a trolley run, add a comment in the ELOG.
Getting ready: Trolley Check List¶
- Beam off
and keys: If in controlled access: Call main control room and get 2 keys (1 for yourself and 1 for a shifter/operator, checking out the controlled access key automatically stops the beam). If you actually go in side the ring (not needed in normal operation mode)fill the details in the control access sheet. Hold on to your keys until you finish the trolley run
- Ask the shifter to call MCR to inform them that we are performing a trolley run. Call the Operator (or ask the shifter to do so) at MC-1 (630-840-6866) to press the "beam inhibit button" located at the back wall next to the large screen.
- Main DAQ: Ask shifters to stop the run and do the laser study after ramping down following instructions in the shifter's wiki page. Note down the main daq run number.
- Ramp down (or ask shifters) Quads and Kickers and retract the collimators:
- Kickers: Kicker Instructions
- Collimators see Collimator
- Kicker/Quad inhibit: after ramping down the kicker and quads as per the above instructions, click the 'Quad/Kicker Inhibit' switch under 'Manual Motion Control' in the Trolley Control page. It should go red and say ON. After ~20 seconds the kicker GUI will update to say 'Trolley inserted! Triggers stopped'. This prevents the kicker triggers from being enabled if the trolley is in the ring.
The trolley checklist can be opened in the browser on localhost:5005.
ssh -nNT -L 5005:g2field-server-2-priv:5005 -o GSSAPIAuthentication=yes G2Muon@g2gateway01.fnal.govallows you to see this page localhost:5005.
Complete the Trolley Checklist in the ELOG. A screenshot of the checklist below needs to be uploaded. This checklist needs to be APPROVED by the runco before you move the trolley.
- Quad and Kicker Status: these values are only updated whenever the buttons are clicked. It's ok that these values are "old"
- the "Collimator & Fibre Harps" and the "Plunging Probe" lines need the "Galil" frontend to be on. The frontend can be started without stopping a field run.
The magnetometer status needs to be checked with the runcos (at the moment)
- Beam (permit, button): If MCR tells you that they are performing mu2e studies this field might not turn green. If that's the case note it in the checklist and talk with the runco.
- Troubleshooting: if the IBMS3 shows "old" you might need to restart the application (if that doesn't help, there is a "ibmsSC" session on "g2sc" that might need to be restarted)
Make sure all entries are green and have a look at them. The "Last Update" column tells you when the displayed values are last written to the various databases. If a value is too old it turns yellow.
ACTION REQUIRED (Inflector current adjustment)¶
The inflector is drifting over time. Every trolley run we check if an adjustment is required. If so, we adjust the infletor current after the drive to drive CCW run before the drive to garage CW run. If the checklist shows the following message
make a note in the trolley checklist that the inflector current will be adjusted. Once at the drive after the long drive to drive CCW trolley run, coordinate with the shifter to call the operator and ask them to increase their SETPOINT of the inflector current buy the amount stated in the checklist. If corrections >5A are required please contact Hogan.
Some background: We are monitoring the variable "G2_S1.IN_PS_DCCT.F_CV" which is believed to be the most accurate. On other locations (operator IFIX, overview pages) "G2_S1.IN_DC_AMPS.F_CV" might be displayed. That's the reason for potential different values. The target for "G2_S1.IN_PS_DCCT.F_CV" is 2765.0A. The operator should increase the current SETPOINT.
Restart the Script¶
In case you need to restart the script, ssh to newg2@g2field-server-2,
screen -r trolleyChecklist
Stop the program with CTRL-C and restart it
in the folder "/home/newg2/trolleyChecklist".
General Instructions for Trolley Motion Control¶
- For trolley runs, we start new Field DAQ runs for each stage.
- Important Azimuth Positions: Drive CW: 262 deg, Garage: 175 deg, Drive CCW: -98 deg
- The trolley cannot pass the drive position (262 deg/-98 deg)
- Turn on Gail control and trolley interface under programs and also power cycle trolley DQM. Everything except plunging probe & Plunging probe DQM are generally turned on.
- Open "Trolley Control Page" from the Midas Status page in another screen
- Make sure tension is below 3 kg
- Make sure Trolley Motor (Fishing Line and Signal Cable, not the Garage) Forward and backward limit status is at 1 (this is the collimator and fibre harp interlock system). If any one of them is zero, the trolley motion alarm will trip, and all motors are turned off.
Make sure the field configuration is consistent with the daily run plan: right surface coil and ps-feedback set point. (See also https://gm2-docdb.fnal.gov/cgi-bin/private/ShowDocument?docid=10285)
- Be familiar with the Alarm handling.
Normal Trolley Run Operation¶
- Write a UW elogs during the trolley runs. Add a comment in the ECL elog (to the trolley checklist) linking to the UW entry.
- Record the Field Midas run number and details in the UW elog.
* spreadsheet with trolley runs (on-calls, trolley experts take care of that): [[https://docs.google.com/spreadsheets/d/1hEBYgMB-xcef-iNfzrv9geypDqZcFJ20X09YJ7iRo04/edit#gid=672167309]
Trolley Insertion into the Storage Ring¶
- Start the run, use “Trolley insertion from the Garage” as comment.
- Click Trolley ON and Garage ON. The motor control will show that they are ON
- Click on Garage Insert trolley: Limit switch will become 1 for Garage forward and velocity will be above zero ( It will take about 5 minutes for the trolley to fully insert)
- The trolley garage cameras can be accessed on localhost:2080 after running
ssh -nNT -L 2080:g2field-fe1-priv:2080 -L 2081:g2field-fe1-priv:2081 -o GSSAPIAuthentication=yes G2Muon@g2gateway01.fnal.govEnter kerberos credentials if necessary.
- DQM will show steps in cable position under Motion section of the trolley DQM page (g2field-be:3335 [remote: localhost:3333] on the field station)
- When the trolley is fully inserted: Garge velocity will read zero and Reverse limit status of Garage will read zero ( limit switch might not read zero all the time)
- Turn off garage. Garage encoder count should be less than 2000.
- Stop the Run.
Trolley Garage Camera Operation [If something is not working]¶
- Turning on the camera: Enter kerberos credentials if necessary. Run
ssh -nNT -L 2080:g2field-fe1-priv:2080 -L 2081:g2field-fe1-priv:2081 -o GSSAPIAuthentication=yes G2Muon@g2gateway01.fnal.gov. The cameras can be accessed on localhost:2080. Klick on the pictures to focus. If motion is detected, the video starts to update.
- If the webpage doesn't load the backend might have crashed. To start that log into g2field-fe1 and cd to the folder "TrolleyCameraViewer".
motion -c motion.config. This starts the application as a daemon, no screen sessions is required.
- Turning on the LEDs: Press the "Camera Power Cycle" button on the trolley control page OR navigate to TrolleyLEDControl directory (cd ~/TrolleyLEDControl). Execute the command ./trolleyLEDcontrol 1 to turn on the LED. Using an argument of 0 turns off the LED. Argument r or R resets the LED.
* Camera selection: There are two cameras on the trolley garage. The one needed for parking points at the area where the black sharpie mark on the trolley rails is visible (and the arrow on the trolley) for parking. -To select this camera, in the cheese app select Edit > Preferences, which shows a few pull-down menus. From the Device pull-down menu, select /dev/video0 to see the upstream side camera that we need to verify the trolley location before parking. For instruction on how to remotely power cycle the cameras see: https://muon.npl.washington.edu/elog/g2/Trolley+system/722
- There is a "Camera Power Cycle" button on top of the "Trolley Control" page
- Alternatively: The cameras can be power-cycled by typing "TrolleyCameraPowerCycle.sh" anywhere on g2field-fe1 (It lives in ~/bin/). This power-cycles both cameras and also resets and switches on the LEDs.
Garage to Drive CW¶
- Start the Run with comment "Trolley run to Drive CW". After the run starts, click on "Trly. To Drive CW" button
- Check DQM (localhost:3333 on the field station) for FIDs (make sure they are updating), Position, Barcode etc. to make sure everything is fine. Keep an eye on ps-feedback DQM ((localhost:3333 on the field station) and make sure the mean field is with in the green bands.
- It will stop at around 262.4 degrees. Velocity will read zero. Record the azimuthal position of where the trolley stopped, in the UW Elog.
- Once the trolley is within ~2deg of the drive the barcode DQM (localhost:3333) starts to display "Barcode Drive Abs 1" and "Abs 2".
- Check on the trolley barcode DQM (localhost:3333) that you see the structure highlighted in the following picture plus one abs barcode with atleast 4 peaks down in "Barcode Drive Abs 1"
If not move manually closer in very small steps of +0.005 (Manual Motion Control: Trolley (Rel. Deg) ).
- Take a screenshot of this barcode DQM page and attach it to the trolley run ELOG.
- Stop the Run
Drive to Drive CCW
- Start a new Run with comment "Drive to drive CCW" and click on "Trly. To Drive CCW".
- It will take about an hour to reach other side. Check vital DQM plots once in a while, keep an eye on ps-feedback DQM and make sure the mean field is with in the green bands.
- It will stop at around -98 degrees. Velocity will read zero. Record the azimuthal position of where the trolley stopped, in the UW Elog.
- Check on the trolley barcode DQM (http://g2field-be:3335/trolleyBarcode [remote: localhost:3333/trolleyBarcode]) that you see the structure highlighted in the following picture.
If not move manually closer in very small step of -0.005. (Manual Motion Control: Trolley (Rel. Deg))
- Take a screenshot of this barcode DQM page and attach it to the trolley run ELOG.
- Stop the Run
If not move manually closer in very small step of -0.005. (Manual Motion Control: Trolley (Rel. Deg))
Drive to Garage CW¶
- If the checklist told you that the inflector current needs to be adjusted, now is the time to do that. Start a new run only for that operation.
- Start a new Run with comment "Drive to garage CW" and click on "Trly. To Garage".
- It will take about 45 minutes to reach the garage. Check vital DQM plots once in a while, keep an eye on ps-feedback DQM and make sure the mean field is with in the green bands.
- Trolley will go to 179 and come back to 175. Velocity will read zero. Record the azimuthal position of where the trolley stopped, in the UW Elog.
- Stop the Run
- Start a new run with comment "Trolley Parking"
- The black arrow should be approximately in the middle of black mark. If not try to do move the trolley manually in small steps (order of 0.01d eg). (Manual Motion Control: Trolley (Rel. Deg)). A typical trolley alignment for parking will look like this:
- System needs trolley position to be with in 175.00+- 0.5 for the parking to work
- Switch the Garage On
- Click Garage park trolley —> Trolley might slide left a little bit, it is perfectly normal. It takes about 5 min to park
- When fully parked, gargae velocity will read zero and forward limit status for Garage turns to zero.
- Turn Garage off and Trolley off
- Stop the run, go to programs and stop Galil control, trolley interface and trolley DQM
- Start a new run with comment "Normal run"
Check the midas messages. If there were any "Skipping trolley data frame at iteration" messages (red) copy them to the ELOG.
- Perform the "Trolley Parked Checklist" in the ECL
- Let shifters know that you are done with the trolley run.- Remind them to uncheck the "Trolley run in progress" flag in the Quad GUI. Remind them to uncheck the 'Quad/Kicker inhibit' switch in the Manual Motion control on the Trolley Control page.-
- Call the operator at MC-1 to pull out the "beam inhibit" button. _Note: the status in the checklist doesn't change. The status is latched, MCR needs to reset it. They will do that as soon as they switch beam back on. _
Return the control access keys by calling MCR.If beam is needed RC/shifters will let them know. Return other keys to person in charge at the control room.
- Post another fermi elog saying that you finished the trolley run
Trouble Shooting and Emergency Response¶
- These instructions should be familiarized by the operator. In some situations, operators may not have time to read through these instructions and then react.
- At ANY time, if the Galil frontend crashed and CANNOT BE RECONNECTED, push the CRASH BUTTON ON THE WALL.
- If a "Motion Trip" alarm is triggered upon switching on the Galil Frontend, check the Limit Switch Status. If the trolley motor limit switches are "0", it indicates that the Collimator and Fiber Harp controllers are powered. Then go to the ring to power them on and start the slow control frontends for them in the Main DAQ. If they cannot be switched on properly, contact the trolley experts.
- During the motion, if the "Motion Trip" alarm is triggered, check if the motion is stopped and all motors are powered off. If not, use the "abort" button to stop the motion.
- Check the tension. If the tensions are all normal (<3), the alarm may be caused by some transients in the interlock system. Log the incident to the Elog, resume the motion, and watch the tension carefully. If the tension rises sharply towards 4, abort the motion and treat it as the "High Tension Trip"
- If the driving tension is high (>4), turn on the trolley motors, back up by 0.1 deg to release the tension. Log this incident and write down the location. Contact the field on-call and trolley experts for further instructions. DO NOT KEEP OPERATING THE TROLLEY IN THIS CASE!
- Better keep watching the NMR signals repeatedly, and if anyone of them disappeared, contact the field on-call.
A few expert instructions¶
Adjusting the limit of the "Trly. To Drive CW" and "Trly. To Drive CCW"¶
Only the "closer" cable is used for the position determination at the drive. The settings are in the Galil frontend. (newg2@g2field-fe1:gm2fielddaq/frontends/galil-fermi/src/galil-fermi.cxx) called "Limit_CW" and "Limit_CCW". Good to know: 1203 enc/deg.
Change it and compile the frontend in the "gm2fielddaq/frontends/galil-fermi/" folder with "make". No "make install" is required. Next time the frontend is started the changes will be used.
Power Cycle the Trolley Interface¶
- The trolley interface can be power cycles with power switch
ssh -L 8076:192.168.30.72:80 G2Muon@g2gateway01. Log in on localhost:8076 with user name/password: G2Muon/precess2016. The interface is on outlet 2 called "Trolley Interface".
OLD Trolley Check List (Activities in the ring are marked as Bold, which are IBMS-3, Kicker Magnetometer,Plunging Probe, Fiber Harp and RF System)¶
Fill out the electronic trolley check list (in the Fermilab ELog drop down menu). Check each entry in the checklist and attach screenshots (or pictures using phone) for the Kicker GUI, Quad GUI, Collimator control, FiberHarp control, Plunging Probe coordinates, RF units(on the false floor) and Q024 & Q025 after checking all of the respective systems as detailed below:
- Beam off and keys to go into the ring:
- If in controlled access: Call main control room and get 2 keys (1 for yourself and 1 for a shifter/operator, checking out the controlled access key automatically stops the beam). Fill the details in the control access sheet. Get 2 MR keys from person in charge at control room.
- If not in controlled access (Beam is already off in this case, so get keys for ring access): Get 2 High Bay keys and 2 MR keys from person in charge at control room. (1 for yourself and 1 for a shifter)
- Hold on to your keys until you finish the trolley run
- Main DAQ: Ask shifters to stop the run and do the laser study following instructions in the shifter's wiki page
- Quads off: Go to quads GUI. Click on "Set voltage to zero" and pulse it at Burst mode. Make sure the voltage is coming down. Stop pulsing when voltage is at least below 50 Volts(Note: Values are in KV on the GUI, It might take some time to come down to 50 volts so you can proceed with the rest of the checklist and stop pulsing before the trolley run). Take a photo and close the quad GUI ( There will be ./startGUI.sh running on one of the terminals, you can do Ctrl+z to close the GUI)
- Quads RF off: Check the following page: [[http://gm2-01.dhcp.fnal.gov/rf.html]]
- Kicker off and Triggers stopped:
- Turn Off: Slide all 3 kicker voltage sliders down to zero. ** Triggers stopped: After sliding down the voltages to zero, click on enable stop triggers at the bottom of the page and make sure the kicker GUI shows that the triggers are stopped in pink at the bottom of the page. Take a photo. Make sure that the kickers are not pulsing (Enabled trigger stop trigger button is Red)
- Collimator : Main DAQ —> collimator control. On the right side click on retract and wait till status shows retract 1, do one at a time. Make sure the retracted status for all 5 collimators are at 1. Also make sure the demand status is 'retracted' for all collimators. The collimator control front end must be ON. If it cannot be turned on, it indicates that the hardware controller is not powered, and in such case the trolley motion hardware interlock will not be cleared, and a motion trip alarm will be generated when the trolley motion control frontend is switched on. Take a photo of the control page.
If the collimator needs to be power cycled: https://cdcvs.fnal.gov/redmine/projects/g-2/wiki/Power_Cycle_Collimator_
- Fiber harps: Make sure Fiber harp control is running (green) in the Main DAQ status page. Then , Main DAQ —> Fiber harp control, same procedure as collimator. Make sure retracted status for all 4 are at 1. Also make sure the request rectract status is 1 for all 4. The fiber harp control frontend must be ON. If it cannot be turned on, it indicates that the hardware controller is not powered, and in such case the trolley motion hardware interlock will not be cleared, and a motion trip alarm will be generated when the trolley motion control frontend is switched on. Take a photo of the control page. Also double check if the fiber harps are retracted while you are in the ring.
- Q024 and Q025: Go to the power supply room and locate the power supplies labeled Q024 and Q025 (location is shown in the diagram below). Verify that they are ON (Both A and B), with ~400 Amp (420 Amp) on the DC current read back for Q024 (Q025). The display on Q025A may be flaky, but it should be ON. Take a photo of Q024 (A&B) and Q025 (A&B).
- IBMS3: Go into the ring and check that the sliver color screw holding the IBMS3 is in the retracted position and can be seen through the glass window. ( need 2 people present anytime you go into MR region). If retracted, this is what you would see through the IBMS3 glass window.
- Plunging probe: Take the MR flash light from the MR-cabinet and make sure the plunging probe is out of the way for trolley by looking through the glass flange ( replace the aluminum foil on the flange when done).* Also check Plunging Probe Control Page in the field DAQ and make sure (x,y,z) = (-4005±10, 792±10, 543±5 ). Take a photo of the control page.
- Kicker Magnetometer: Make sure the kicker magneometer is in the retrated position and held with rods and cable ties. (The kicker magnetometer is located just to the right of Bazooka 1. Lift up the insulation from the Velcro at the bottom to see the kicker magnetometer, and reattach the insulation after checking )
- RF system is off: Check on the RF system rack (right across from IBMS-3) that RF power amplifier 'power' and 'enable' switches are turned off. There are a total of 12 units. Top 4 units are shown in the image below (Unit between 4 and 5 is not in use). Take a photo, you might need 2 photos to get all 12 *
The "old" quad electronic shows sometimes pulser faults during a trolley run. This is nothing to worry about but annoying. In order to suppress them, we set the checkmark "Trolley run in progress" in the quad QUI whenever we perform a trolley run.