Project

General

Profile

Wiki » History » Version 79

Version 78 (Dominic Barker, 08/23/2018 03:16 AM) → Version 79/81 (Dominic Barker, 08/23/2018 07:15 AM)

!https://cdcvs.fnal.gov/redmine/attachments/download/48302/sbnd-logo-big.png!

---

h2. %{color:red}Quick Info%

* *RUN COORDINATOR* Minerba Betancourt (612-859-9078)
* Google Hangouts ID: sbndshift@gmail.com (password is the same as the sbn-doc-db)
* "Hangouts Group":https://hangouts.google.com/group/cdpWmYQYwS3z31T62
* VST shift: *+1 331 442 4353* (Hangouts Voice Service)
* ROC West x5413
* MCenter Control Room (MC-CR) x3726 or x8871
* Internationally: prefix *+1 630 840* to any extensions (marked with x)
* "Experts list":https://docs.google.com/spreadsheets/d/1_lixBwEmZP8X2hEL4gabbaPx3uZYUMtQdMLQH-5-aQE/edit?usp=sharing

---

{{toc}}

h1. Welcome to the SBND VST Wiki

This is the wiki for the SBND's Vertical Slice Test (VST). This is an integration of the cold and warm electronics with DAQ and online monitoring using LArIAT's cryostat and TPC.

*HUGE DISCLAIMER: IT IS TOO SOON TO GET COMFORTABLE*
*THIS PAGE IS BEING DEVELOPED RIGHT NOW! CHECK IT EVERY TIME SO YOU ARE UPDATED TOO!*

h1. %{color:red}Run Plan for August 20 to September 05%

* At the start of your shift, please end the previous run by stopping the SBND DAQ and then the LArIAT DAQ, and start them both up again (first LArIAT, followed by SBND). Be sure to document this in the elog.
* When the new run starts, the Online monitor show reliable information for the noise correlation matrix and the wires rms. Save a snapshot of

1) the Correlation Matrix
2) the Collection Plane RMS
3) the Collection Plane Pulse Height

and add them to a New Run e-log entry (see entry "297":http://dbweb6.fnal.gov:8080/ECL/sbnd/E/show?e=297 as an example).

There are now 4 types of reports:

Start of Shift Checklist
End of Shift Checklist

New Run
End of Run

The shifter must fill in all of them. Instructions are in the form themselves.

At the end of the shift, shifters need to post an event display image of the data collected during the shift.

h1. Taking VST Shifts

In summary, a VST shift is just like a LArIAT shift, with a few extras. If you didn't do this yet, make sure you are in the "SBND.Shift":https://hangouts.google.com/group/cdpWmYQYwS3z31T62 group at Google Hangouts.

h2(count). %{color:green}Prerequisites%

*Gaining Access*

* First, set up your Fermilab computing account and Kerberos. This can be done by following instructions "here":https://redmine.fnal.gov/redmine/projects/lardbt/wiki/Fermilab_Computing_Access. Check that you can issue a Kerberos ticket with

<pre> kinit <yourKerberosPrincipal>@FNAL.GOV</pre>

and confirm that you have gotten a kerberos ticket by checking the output of the command @klist@.

* If you are at Fermilab:
** Check that you are connected via ethernet, or to the Fermilab @fgz@ Wi-Fi network and not the guest Wi-Fi network. This is required to access the lariat shift machines. The latter may lead to "Connection reset by peer" errors, refusing you access. Register your device to the fgz network "here":https://fermi.service-now.com/kb_view_customer.do?sysparm_article=KB0011206.
** If you find you cannot SSH into lariat-cr-02.fnal.gov, make sure your SSH config file allows X11 forwarding and all GSSAPI options. Further details can be found "here":https://cdcvs.fnal.gov/redmine/projects/lardbt/wiki/Fermilab_Computing_Access#Logging-into-Fermilab-Computers-with-Kerberos.

* If you are outside of the lab:
** %{color:red} The Java dependent Cisco AnyConnect client can no longer run from a browser due to the lack of support for NPAPI%. This means that the old VPN method (https://vpn.fnal.gov) only works in old browsers. Use this as an alternative method, which is browser-independent:

Uninstall Cisco AnyConnect (skip this if you are sure you don't have it):

<pre>
sudo /opt/cisco/vpn/bin/vpn_uninstall.sh
</pre>

Install openconnect (it is an open source vpn client)

if you are using ubuntu...

<pre>
sudo apt-get install openconnect
</pre>

if not, use the appropriate method for your system. You can verify the installation by asking the version:

<pre>
openconnect --version
</pre>

Install a vnc client of your choice. TigerVNC or Remmina work perfectly.

h2(count). When you are about to start

Contact the current shifter in the hangouts group to tell him/her that you are getting ready. This will give the current shifter time to fill in his End Of Run report. Ask about the current plan and status (the shifter might ask you to read some e-log entries).

Don't log into the hangouts yourself just yet. Wait until the current shifter tells you that he/she is ready to give you the controls.

h2(count). Setup your shift station

Either from a remote connection or from ROC-West, follow these instructions.

*Log onto the shift computer*

h3. For a shift at Fermilab:

* From a *ROC-West machine*, such as lariat-cr-02:
** Log in with username = @lariat@.
* From an *MCenter machine*, such as ftbflx18 or ftbflx15:
** Log in with username = @ftbf_user@.
* Password is YOUR kerberos password, then it will ask for YOUR kerberos principal. (Reverse from usual order)

h3. From a remote shift:

* *VPN* - To connect to the Fermilab VPN, open a terminal and execute:

<pre>
sudo openconnect https://vpn.fnal.gov
</pre>

and it is going to ask for your Username and Password (your SERVICES password).

If it works, you will be greeted with this message:

<pre>
| NOTICE TO USERS
| This is a Federal system (and/or it is directly connected to a Fermilab local network) ...
</pre>

That means you are in. Do not close this terminal.

* *SSH Tunnel* - In another terminal, get a Kerberos ticket

<pre>
kinit <username>@FNAL.GOV
</pre>

with your Kerberos ID. Now, open the tunnel with:

<pre>
ssh -C -L 5900:localhost:5900 lariat@lariat-cr-02.fnal.gov
</pre>

and you know it worked if you get another greeting:

<pre>
Welcome to the LArIAT ROCwest Control Room
</pre>

Again, leave this terminal open. If this fails, ask one of the following collaborators (stjohn@fnal.gov or badgett@fnal.gov) to add you to the ~lariat/.k5login list, which will grant you SSH access.

* *VNC* - Finally opening the VNC. You may use a client of you choice. When it asks for a server, use *localhost:5900* which is our tunnel. The password is the same as the *lariat-doc-db* password (*%{color:red}NOT SBND's%*).

** *Please do not interfere with the shifter by VNC!* Contact the shifter on Google Hangouts to confirm a convenient time before you test your VNC.
* *Google Hangouts* from Chrome browser (phone function does not work in Firefox)
** Try to make a free phone call to a US number using the hangouts dialer (may require installing Google Talk plugin first time)
** This is how the Main Control Room will phone you (it will ring when they dial +1 331-442-4353) and it is how you should phone them.

h3. Windows needed on the shifter computer (or by VNC)

See these "snapshots":https://cdcvs.fnal.gov/redmine/projects/lariat-online/wiki/How_to_set_up_your_shift_station#Shifter-desktop-layout of the ROC-West desktop should give you a good idea about what to look for.

---

h2(count). Verify that Synoptic and Acnet are running

You should get familiar with them. Always keep them open and in clear view. If connecting remotely, you should see them in the VNC shared desktop.

https://cdcvs.fnal.gov/redmine/projects/lariat-online/wiki/Using_Synoptic_for_monitoring

https://redmine.fnal.gov/redmine/projects/lariat-online/wiki/Using_ACNET_for_monitoring

h2(count). Connect as the SBND shifter in the hangouts

When the previous shifter had time to call the MCR and declare the end of his/her shift, you may log into the shifter's Hangouts account with:

sbndshift@gmail.com

using the current SBND DocDB password. The SBND Shift chat is used by the collaborators to communicate with the shifters and between themselves about the VST operations.

h2(count). Log into the SBND Electronic Logbook

http://dbweb0.fnal.gov/ECL/sbnd

with your Services user. Make an *VST Start of Shift Cheklist* entry and follow the checklist, which includes calling the Main Control Room (see next step).

h2(count). Calling the Main Control Room (MCR)

Remember to call MCR (x3721) at the beginning and ending of each shift. If you are remote, you can use the hangouts to do so, but you may have to install a plugin first.

Main Control Room (internationally) +1.630.840.3721

Tell the MCR that you are the next LARIAT SHIFTER and if you are local or remote. If you are remote, inform the MCR of your contact number, which is the hangouts voice service:

+1 331 442 4354

You have to call the MCR at the begging and ending of each shift!

h2(count). During your shift

*See the RUN PLAN at the top of the page.*

Either in your machine or in the one you are remotely operating via VNC, you have to keep 3 (three) independent terminals running: LArIAT DAQ, SBND DAQ and SBND DQM.

* LArIAT DAQ: https://cdcvs.fnal.gov/redmine/projects/lariat-online/wiki/LArIAT_DAQ_Instructions

* SBND DAQ and DQM: https://redmine.fnal.gov/redmine/projects/sbnddaq/wiki/Starting_DAQ_Software

If for any reason you have to access those yourself (sometimes it happens when the previous Kerberos ticket expires), here is how:

** With the tunnel open, log in to the lariat-gateway with the sbnd user:

<pre>
kinit <youruser>@FNAL.GOV
ssh -X -Y sbnd@lariat-gateway.fnal.gov
</pre>

once you are in the gateway, access lariat-daq00 if you want to launch the LArIAT DAQ, or lariat-daq04 for the SBND DAQ (you want both, so repeat this procedure twice).

<pre>
kinit <youruser>@FNAL.GOV
ssh -X -Y sbnd@lariat-daq04.fnal.gov
</pre>

again, substitute daq04 for daq00 if you want to launch the LArIAT DAQ.

In the DAQ00 window, execute the LArIAT script:

<pre>
go
</pre>

Do *not* specify any arguments.

In the daq04 machine, excecute the SBND script:

<pre>
startDAQInterface
</pre>

You also need the SBND DQM website (which can only see this via Fermilab VPN or locally through the fgz network): https://sbn-online.fnal.gov/cgi-bin/minargon/minargon.wsgi
If you are in a remote shift, you can open another terminal and make use of the tunnel. Launch your favorite browser from the terminal, for instance

<pre>
google-chrome
</pre>

and it will be able to access the DQM site.

Keep checking the Synoptics alarms and the DQM website. The DQM has several plots that should be updating every few minutes. If you see any of the time stamps getting too old, you may need to restart it (instructions above). Instructions on how the DQM works are in its first page.

*Archiver checks*
Once an hour, check the status of the archiver by following the instructions on the [[Slow Controls Archiving]] page.

*%{color:red}Some Troubleshooting%*

If the SBND DAQ won't start, there are two steps to follow, in this order:

Step 1. Try again: sometimes it just need some persistence.

Step 2. Other times, it might need some clean up. Try running this script:

<pre>
kill_ipcs
</pre>

and go back to Step 1.

Another problem is data corruption. You will see the RMS of the Wires going through the roof. If that happens, try this one first without turning off the daq.

<pre>
source start_bnl_ld.sh
python WIB_nevis_sync.py
</pre>

After about 3 minutes, you should see the rms going back to normal. If not, then follow this recipe:

stop the daq (ctrl+C only once)

Reconfigure the cold electronics
Note: this is the standard CE configuration procedure, so can be used for any time configuration is needed. If starting from state where electronics are off, you can skip the FEMBoff step.
<pre>
ssh lariat-gateway02
source start_bnl_ld.sh
source Local_diagnositc_RUN_FEMBoff.sh
source Local_diagnositc_RUN_FEMBon.sh
source Local_diagnositc_RUN_ext.sh
</pre>

start the daq on lariat-daq04

<pre>
startDAQInterface
</pre>

Report any oddities in the e-log with a *default* form, under the *Vertical Slice Test* category.



h2. Event Display


There are two ways to the run the event display now. If you want to look at the most up to date data. Data which might not have gone through SAM yet you can log into the daq04 machine and use the event display there. Otherwise you can you the event display on the sbnd gpvms.

h3. daq04 Event Display.

To run the event display. From the control room PC log into daq04:
<pre>
ssh -X lariat-gateway -l sbnddqm -t ssh -X lariat-daq04
</pre>

Then run the command replacing the run number you wish to look at and the sub run number:
<pre>
./home/nfs/sbnd/SBND_DAQ2/bin/evd_vst run_number sub_run_number
</pre>

This will bring up the standard event display which you can flick through the events.

h3. SBND GPVM Event Display.

log into the sbndgpv01. Remember to kinit (kinit username@FNAL.GOV).
<pre>
ssh username@sbndgpvm01.fnal.gov.
</pre>

Then got the VST directory in app just running the following command:
<pre>
cd /sbnd/app/users/VST
bash evd_vst.sh
</pre>

This will cycle through the events and automatically pick up and read the latest file. You can pause the cycle by typing PAUSE (upper or lower case) into the command line and pressing enter (you will not see that you are writing pause). You can then save the event display. Type PLAY (upper or lower case) to continue the loop. NOTE: If you press previous on the GUI then when you press play it will loop backwards. So press next once to sort the loop out. Note that I've only just made it so it might be a little buggy. Contact DomBarker for issues.

---

---

---



h1. Advanced Stuff

h2. How to change the DAQ configuration to calibration mode

At lariat-daq04.fnal.gov, open the file /home/nfs/sbnd/SBND_DAQ2/bin/start_daq_interface.sh and comment the line
> CONFIG=NevisTPCXMIT30 # External triggers
and uncomment the line
> #CONFIG=NevisTPCCalib # Internal calibration triggers
Then you can start a new run following the usual procedure. This will run the DAQ using internal triggers and a calibration pulse will be sent to the Magic Blue Box for each trigger.

When you are done, remember to undo the changes to the /home/nfs/sbnd/SBND_DAQ2/bin/start_daq_interface.sh file.

h2. How to run the DAQ for the BNL-Nevis integration test

(Note: this DAQ routine has been superseded by "How to change the DAQ configuration to calibration mode":https://cdcvs.fnal.gov/redmine/projects/vertical-slice-test/wiki#How-to-change-the-DAQ-configuration-to-calibration-mode . If you want to run the DAQ in calibration mode, follow the instructions there)

1. Make sure the Nevis crate and NIM bin are powered. Experts: (on-site) Davio Cianci, (off-site) José I. Crespo (Skype: josei.ca)
2. ssh into the DAQ machine
Log into the SBND gateway (or you can create an FNAL VPN connection and skip sbnd-daq00, not needed on FNAL site)
> ssh -Y sbnd-daq00.fnal.gov
Log into the LArIAT gateway
> ssh -Y lariat-gateway01.fnal.gov
Log into the DAQ server using the common sbnd account (reserved for DAQ)
> ssh -Y sbnd@lariat-daq04
3. Go to the jcrespo/ directory
> cd jcrespo
Run the DAQ-decoder-plotter script
> ./run_and_plot_sbnddaq.sh
and follow the instructions printed on screen. This script runs the DAQ in calibration mode (CALIB pulses are sent to the Magic Blue Box every 1 s) -- once you have collected the desired number of events, it requires you to stop the DAQ using CTRL + C twice --, then runs the binary decoder, and finally runs a ROOT analysis to plot the waveforms.

The data is saved in the directory /home/nfs/sbnd/jcrespo/runs
The script makes a new directory for each run named RunYearMonthDayHourMinutesSecond (e.g. Run20180620113511).
The raw binary data is the file named output_NevisTPCCALIB_generator.dat. For a hexdump, you can use
> hexdump -v -e '8/2 "%04X ""\n"' output_NevisTPCCALIB_generator.dat | less
(it will display the data in 8 columns with 2 bytes each). The ADC data is encoded in the lower 12 bits (3 last hex characters) of the words beginning with "0".

h2. How to configure [[LArIAT Trigger]]

Follow the link [[LArIAT Trigger]]

h2. How to Access Files

All files created for the VST test are initially created on lariat daq01 and daq04. They are despoiled in /daqdata and are quickly moved to /daqdat/dropbox where they last for ~2 days until they are moved to SAM. Files beginning with sbnd are the raw data non-decoded files. Files beginning with digits_sbnd are larsoft friendly files. These have been decoded and raw:digits objects have been created. Files beginning with lariat for the corresponding runs contain the beam-line detector information.

Raw Data files starting with sbnd from runs prior to Thursday 13th July are now temporarily being stored in /pnfs/sbnd/persistent/users/VST/sbnd_files. You can access these from the SBND GPVM. Gray is putting the digits on pnfs as well. Soon there will be the corresponding digits file as well.

All Other files can be found on SAM. All the files come under the lariat experiment name so make sure there is a "-e lariat" in the commands you write. To Quickly access the files here is a set of instructions:

Setup the required stuff.
<pre>
source /grid/fermiapp/products/setups.sh;
setup ifdhc;
setup sam_web_client;
setup fife_utils;
</pre>

To Look at the files available you can use some of the "SAM commands":https://cdcvs.fnal.gov/redmine/projects/sam-main/wiki/Sam_web_client_Command_Reference. Here is a few examples:

To see a list of files from after a date:
<pre>samweb -e lariat list-files "start_time > 02-Jul-2018" </pre>

To see all the files which have sbnd:
<pre>samweb -e lariat list-files "file_name like sbnd_r%.root" </pre>

To use metadata.
<pre>samweb list-files -e lariat "MetaDataName = value"</pre>

Use And to do two or more filters :
<pre>samweb -e lariat list-files "start_time > 02-Jul-2018" AND file_name like sbnd_r%.root </pre>

You can look at the MetaData in the files by running:
<pre>samweb -e lariat get-metadata file.root</pre>

You can locate the file on enstore:
<pre>samweb -e lariat locate-file file.root </pre>

To use the files you have to stage them and copy them into your directory. This "can be done":https://cdcvs.fnal.gov/redmine/projects/sbndcode/wiki/_How_to_create_and_use_a_SAM_definition using "project.py":https://cdcvs.fnal.gov/redmine/projects/larbatch/wiki/User_guide as a wrapper once definitions of the data have been created.
For now you can use ifdh_fetch to fetch one or more files. It works in the same way as SAM and here is a couple examples on using it:

Know the file name:
<pre>ifdh_fetch -e lariat filename.root</pre>

Copy every file with a name that begins with sbnd_r. (NOTE: We have the downloaded so don't do it again. Just run on them. If they have been deleted because it has been a long time and we need the space for MC productions then you can download them but make sure we have the space!):

<pre>ifdh_fetch -e lariat --dims "file_name like sbnd_r%.root"</pre>

For more sbnd specific information you can find another wiki here: https://cdcvs.fnal.gov/redmine/projects/sbndcode/wiki/_How_to_create_and_use_a_SAM_definition.