(DocDB # 9670)¶
The NOVA Power Distribution System (PDS) supplies power to five different electronic components used in the readout:
front-end boards (FEBs), avalanche photodiods (APDs), thermoelectric coolers (TECs), data concentrator modules (DCMs) and time distribution units (TDUs). Two similar systems are required to supply power for the Near and Far Detectors. NOVA electronics channel counts is given in Table 1. The schematic view of the power distribution system is given in Figure 1.
Table 1. NOVA electronics channel counts.
Figure 1. Schematic view of the Power Distribution System.
Table 2. Power requirements for the detector electronics.
The Voltages and currents needed for each electronics component are summarized in Table 2.
The 425V to the PDBs is provided by two Wiener MPOD HV-EX high voltage power supplies System mainframes using 15 ISEG EHS F6-05x 156-F floating 16-channel cards. Each of the 16 channels of the ISEG is individually programmable from 0-500V, and provides a maximum of 15mA of current.
Each Wiener HV channel feeds a single PDB.
1 HV channel -> 1 PDB -> 1 DCM -> 64 APDs
1 HV module (16 channels) -> 1 DB -> 12 DCM
The low voltages (3.3V and 24V) needed by the FEBs, DCMs, and TE coolers, are provided by the Wiener PL506 Power Supply System. Each Wiener PL506 chassis has six floating individually programmable power supply modules: three 2-7V channels, each rated to 115A and 550W, and three 12-30V channels, each rated to 23A and 550W. Hence each PL506 crate feeds power to three adjacent PDBs: the 3.3V via 2AWG cables and the 24V via 6AWG cables.
Both of the Wiener HV and LV power supplies are remotely controllable with Ethernet interfaces, and have programmable output voltages and current limits. For further details regarding the Power Distribution System please refer to Documentation_for_PDB.pdf (DocDB # 9670).
There are 2 HV crates for Far Detector.
1st crate supports DB 1-8,
2nd crate supports DB 9-14.
There are 4 LV crates per DB. Each crate supports 3 DCMs with 3.5 and 24 V (Figure 1).
Crate 1 : 3.5V / 24 V
Crate 2 : 3.5V / 24 V
Crate 3 : 3.5V / 24 V
Crate 4 : 3.5V / 24 V
Relay Rack Labeling Conventions
(R. Tesarek, DocDB # 6814)¶
The naming convention is intended for easy access to cables. Labeling conventions are based on their geographical locations.
The general format of the labels is:
LLxrmRRnnn-sLLx - Building Level
- LL0: ground level
- LL1: -1 level
- LL2: -2 level
- LL3: -3 level
- LL4: -4 level
- CC: Computer Center
- CR: Control Room
- DH: Detector Hall
- DC: Data Closet (LLI)
- Detector Hall:
- top catwalk (LL0) racks located relative to gates
(01-19) N/S (north or south of the gate)
- floor level (LL4) racks located relative to
catwalk support columns (01 - 11 from south to
north) N/S (north or south of the labeled
- top catwalk (LL0) racks located relative to gates
- Computer Center
- East row (01-05 from S to N)
- West row (06-10 from S to N)
- F: Front
- B: Back
Figure 2. Labeling
Monitoring Power Supplies¶
The ACNET system monitors many different variables, which can be viewed anywhere using the synoptic page. Synoptic is a client-server system for graphical data representation.
It is similar to ACNET Lex SA and EPICS EDM applications. The displays can run either within a Java application, or in a conventional web browser as "live" SVG images. In a web page mode internet browser generates .svg images of the displays, so you will not be able to get access to the displays with the control/set capabilities. Live web pages are intended for the quick review of the detector status. Gaining control/set capabilities requires Java Runtime Environment.
Below is the list of pages, from which you can access the synoptic displays for Near and Far Detectors.
- http://synoptic.fnal.gov/ (Need VPN + MIT Kerberos Ticket if you need control/set capabilities and off site) or
- http://novadcs-far-master-01.fnal.gov:8000/synoptic/ (Need VPN + MIT Kerberos Ticket if off site), or
- http://synoptic.fnal.gov/novafar/ for a web display-only for off site (quick view of data without control/set capabilities).
The synoptic pages are always running in the control room, and a DCS expert as well as the shifter should be familiar with the synoptic pages and what is displayed on them. The ACNET server computers are novadcs-far-readout-0X where X is 1, 2, 3, or 4. There is also a data logger for viewing historical data (DocDB # 8900 or the Sensor-log Analysis expert's page here).
If for some reason these pages are not running in the control room or you need to recover after a crash:
1. Go to * http://synoptic.fnal.gov/ address from the web browser (for Near Detector).
2. Click Launch Viewer from the left panel. This will prompt to start a java application.
3. Start the Java application.
4. Hit 'Cancel' if prompted to open displays. Hit the "Download & Open" button in the main Synoptic viewer window.
5. Select appropriate folder: Nova_Near in the browser.
6. Select NearIndex for Near Detector displays.
7. Open all windows by clicking the links.
For the Far Detector DCS monitor recovery:
1. Go to * http://novadcs-far-master-01.fnal.gov:8000/synoptic/ address from the web browser (for FAR Detector only).
Repeat ALL the steps from 2-7 above, obviously replacing Near by Far in the names.
Shifter instructions with screenshots for NOVA power supply controls through DCS can be found in Shifter_instructions_PS_control.pdf (DocDB # 8209 by Athans).
From the main summary display you can pick the PS view you need: LV, HV, by Di-Block, by Channel. To be able to reset power or make any changes to LV or HV, the first thing to do is to "Enable" controls. This is done by clicking the "Enable" button at the lower right corner of the window . Select the time span you need to make changes (10 min, 1 hour etc.). Make sure to "Turn off" the controls once you are done to prevent any accidental changes from the window.
Below are the screenshot images from document DocDB # 8209 showing how to (Re)Start Synoptic displays and enable controls.
Each LV and HV channel is monitored by 18 parameters. The first 12 parameters are just status monitors: Green is 'ON' and Red is 'OFF' four ON/OFF status button. For the rest of the parameters Green means Normal and RED means an 'ALARM'.
Parameters 13-18 have set values that are shown in corresponding boxes. The 'ON/OFF' control buttons are used to recycle the power on a particular channel. For LV the power can be recycled either by 'ON/OFF' control buttons for a single channel or by 'ON/OFF' crate master switch control buttons. Crate switches should be used with Expert's supervision.
This will recycle both 3.5 V and 24 V power on all channels supported by the crate.
Few things you need to know regarding the power supply displays.¶
1. The figure below shows the definitions of acronyms in the alarm tree display for Muon Catcher. The acronyms are the same for all alarm tree displays for Near and Far Detectors.
2. In a single channel view displays you can also see the graphical view of each variable. The 0. s mark on the time axis represents the moment the display was originated. Negative times correspond to the history in the past.
If the channel is turned off properly without problem, than the corresponding values of the variables in the graphs will be 0. On the other hand, if the channel is turned off automatically due to an ALARM, the graphs will retain the very last value of the variable that channel had right before it went into ALARM. It is done intentionally for troubleshooting purposes. What this means is that you will see that the channel is OFF and yet the graphs will show some non zero value for Voltage, Current etc.
3. Sometimes it is necessary to turn the power supplies completely OFF for safety reasons in order to do maintenance or repair work after leaks. This automatically will set ALL voltages to zero. The channel master switches are set up in such away that they will automatically restore the set values for both the LV (3.5V and 24V ), and the HV (425V). If using single channel ON/OFF buttons, you might need to set the Voltage values manually in the pink column in the diblock view displays (FD) or LV & HV details displays (ND). After entering each value, hit enter to set the voltage.
4. While the channel master switches are intended for quickly powering ON/OFF the entire detector (ND) or entire diblock (FD), crate master switches are intended for troubleshooting and expert activity.
If you don't know the reason why the power was cut to the detector, do not power cycle the crates, as it will reset all the inhibit error bits, preventing the experts from diagnosing the problem.
While any of the inhibit error bits is active, channel(s) will not be able to ramp up.
PS expert will power cycle the crates if necessary to clear the error bits, after investigating the issue.
5. There is a small discrepancy between the FD and ND setting of HV "Enable Kill" bit in the synoptic displays. Although discussed at length between the experts, no final decision was reached regarding whether it should be updated or not. Contact Ralph for more details.
6. Alarm displays now have "Alarm Acknowledge" functionality. It was designed to prevent shifters from ignoring the active alarms. Since every single alarm on the lower branches of the alarm displays will cause the entire diblock to go into an alarm, it is not recommended to tell shifters to ignore the alarm for days, even if you know what is causing the alarm, as they will not notice any new alarms in that diblock.
Clicking any of the buttons in the lower branches will pop up an "Alarm Acknowledgement" display, show in the figure below, which will allow to temporarily bypass a particular alarm in the ACNET system. You can select the time span you would like to disable the particular alarm. This will restore the diblock to a normal green condition, allowing shifters to monitor the rest of the diblock for any new alarms. After the specified time lapses, the alarm will return. You can also enable all alarms at any time from the same display. Note: Bypassing the ACNET alarms will not change any of the hardware alarm settings, so if things go wrong, the channel will still shut down if/when the measured values hit the alarm hardware thresholds.
Needless to say, shifter is NOT allowed to DISABLE any of the alarms without expert supervision.
7. Occasionally Rack monitor display may turn pink, showing disconnected while the HV and LV power is actually ON.
SOLUTION: Call Ash River and have them check the Network switch on Network rack and reset NI CompactRIO on both HV racks. After reset display should update automatically.
Rack monitor display monitors the Rack temperatures, so you don’t want to run without it for a along time, but it has NO effect on interlock.
8. If PS Synoptic displays are not responsive and cannot be killed from desktop, you need to manually check for zombie processes and kill them manually.
SOLUTION: In this case java applications need to be killed from the terminal by ’kill -9’ command. Login to CR machine running the PS displays. Run the following commands to find and kill the all running java processes.
> ps aux | grep java
> kill -9 PID
9. Some of Ralph's scripts can be found in DocDB # 16146. These scripts will reset the LV and HV power supplies to their right settings as well as reset error bits ( HV power supplies ONLY ).
Instructions for Checking / Upgrading / Downgrading WIENER Controller Modules.¶
It will require a Windows computer with MUSEControl Software installed.
1. install module into the crate.
2. Open WIENER folded and run MUSE-control. This will check the channels.
An error message “No module found” will appear. Press “Ignore”.
3. Go to System -> Firmware update. Click Browse: This will check what is the current firmware installed.
Upgrade / Downgrade to the version as needed.
You can check the firmware version on NDOS or FarDet via SNMP walk command.
You will be prompted to click the “Cancel” button after the installation is complete.
4. To find the latest firmware versions:
Go to website: file.wiener-d.com to get different firmware versions if they are not already in the computer.
Select firmware version to download.
5. Next step is to setup the network on the module
Go to System -> Configure:
Copy established IP, MASK and gateway info into the window.
Name of the test stand : "NV-TEST-HV " (Jianming's office)
This will set up network info on the controller.
6. Power recycle the crate to finish setup.
7. Connect ethernet cable and try to communicate with the crate via SNMP or ping the IP address.
Try turning the Main Switch ON and OFF with SNMP commands.
8. If you can successfully communicate with the module and you have the proper firmware version installed than you are DONE.
It can be installed into Near or Far Detector. It will require new network setup for corresponding detector, since the module was assigned an IP address
associated with the test stand.
Additional instructions can be found in firmware_upgrade_LV_PS.pdf by Ralf Ehrlich.
NDOS Power Supplies (Obsolete)¶
The low voltages needed by the FEBs (3.5V), DCMs (24V), and TE coolers (24V), are provided by the Wiener PL506 Power Supply System. Each power supply has three 3.5V channels and three 24V channels, so that it can serve 3 power distribution boxes (PDBs), while each PDB gets fed by one 3.5V and 24V channel each. These power distribution boxes distribute the power to the FEBs, TECs, and DCMs. There are currently two low voltage power supplies installed, which are both located at relay rack 1 (see Table 3).
To access the LV power supplies remotely, you will need to know the corresponding IP addresses.
For NDOS the corresponding IP addresses are:
18.104.22.168 (not there any more since muon catcher is removed)
The LV channel numbers are the following:
u0, u2, u4 = 3.5 V
u1, u3, u5 = 24 V
u-LOWER case (important)
The high voltage needed by the APD is provided by the Wiener MPOD HV-EX high voltage power supplies system mainframes using ISEG EHS F6-05x 156-F floating 16-channel cards. Each Wiener HV channel feeds a single PDB. The HV power supply is installed at relay rack 2.
To access the NDOS HV power supplies remotely use IP addresses:
The HV channel numbers are the following:
ONLY for HV:
The channel formatting is: uNMM
u = LOWER case (important)
N =slot number (0-7 ? number of modules)
MM = 00-15 (channel number within the module)
LEADING zeros are scratched. So channel u000 -> u0
FAR Detector Power Supplies¶
Far detector power supplies are accessed by network address instead of IP address.
FAR Detector LV Power Supplies:¶
Network Address: nova-far-lvpower-NN-MM.fnal.gov
NN = Diblock number (01-14)
MM = LV position in the Relay Rack (in other words: crate number: 01-04)
FAR Detector HV Power Supplies:¶
Network Address: nova-far-hv-power-NN.fnal.gov
NN = crate number (01, 02)
The HV channel numbers are the following:
The channel formatting is: uNMM
u = LOWER case (important)
N =slot number (0-5 ?)
MM = 00-15 (channel number within the module)
LEADING zeros are scratched. So channel u000 -> u0
For safety reasons, whenever something needs to be done at the power cables, the plug at the front panel should be pulled to disconnect the power supply AFTER everything is turned off.
NEAR Detector Power Supplies¶
The IP addresses for ND power supplies:
Node Name MAC address Fermilab tag IP
NOVA-NDUG-LVPOWER-1-1 0050-c22d-cbd4 575544 22.214.171.124
NOVA-NDUG-LVPOWER-1-2 0050-c22d-cc12 575564 126.96.36.199
NOVA-NDUG-LVPOWER-2-1 0050-c22d-c9f5 556830 188.8.131.52
NOVA-NDUG-LVPOWER-2-2 0050-c22d-cc26 575583 184.108.40.206
NOVA-NDUG-LVPOWER-2-3 0050-c22d-cbe0 575555 220.127.116.11
NOVA-NDUG-HVPOWER 0050-c22d-cbfb 557752 18.104.22.168
The network mask is 255.255.255.168
The gateway address is 22.214.171.124
Data Logger Plotter (DocDB# 8900-v6)¶
Most of the variables that are being monitored in Synoptic displays, are being logged.
Using the links below, different variables can be plotted to see the variations over different time spans.
Consult the DocDB# 8900-v6 for the list of device names or variables being logged. Device names also can be found in the UpdatedCatalog-of-ACNET-devices-for_data-logger-plotter.xlsx file attached below.
(You will need Kerberos ticket and VPN for off-site access)
Links do not work with Safari browser. Use Firefox, Chrome or Internet Explorer.
ND Data Logger
Command Line Control¶
Both LV and HV power supplies can be controlled via SNMP protocol, which allows the user to access and change almost all options and values of the power supplies. It is suggested that the power supplies have a firmware of version 126.96.36.199 or later. While all values of the power supplies can be read or written via MUSE control, there are few limitations to SNMP: It is not possible to read the power of the module and the power of the load via SNMP, however the load power can be calculated from outputMeasurementTerminalVoltage and outputMeasurementCurrent. Hardware limits for the terminal voltage, sense voltage, and current cannot be read either. It is also not possible to have read or write access to the option “no ramp down at switch off”, and “internal sense connection”. All other values can be accessed with SNMP as well as MUSE control.
One needs to have NetSNMP installed to communicate with the power supply. The file WIENER-CRATE- MIB.txt (which can be found at http://galileo.phys.virginia.edu/research/groups/hep/aag/nova/pds/power_supply_files/WIENER-CRATE-MIB.txt)
is needed as well. At the computer at the ΝΟνΑ surface building, this file is located at /home/nova. In the section below, one can find a list of commands which can be used to control the power supply.1 The option - m +./WIENER-CRATE-MIB.txt assumes that the commands are executed from the same location as the MIB file, i.e. at /home/nova. If the MIB file is located somewhere else (e.g. if run from your own laptop), the path to the file needs to be changed accordingly. If the MIB file is moved to /usr/share/snmp/mibs/, then the option -m +WIENER-CRATE-MIB can be used.
The WIENER-CRATE-MIB.txt file includes ALL commands needed to communicate with the Power Supplies.
The most important command are:
snmpget - reading 1 channel
snmpwalk - reading ALL channels
snmpset - writes / sets 1 channel
Below are some examples of the commands issued via SNMP.
To check whether the crate is on or off:
snmpget -v 2c -m +./WIENER-CRATE-MIB.txt -c public 188.8.131.52 sysMainSwitch.0
To turn the crate on:
snmpset -v 2c -m +./WIENER-CRATE-MIB.txt -c public 184.108.40.206 sysMainSwitch.0 i 1
To turn the crate off:
snmpset -v 2c -m +./WIENER-CRATE-MIB.txt -c public 220.127.116.11 sysMainSwitch.0 i 0
To return a list of possible HV channels:
snmpwalk -v 2c -m +./WIENER-CRATE-MIB.txt -c public 18.104.22.168 outputName
To return the set voltages of all HV channels:
snmpwalk -v 2c -m +./WIENER-CRATE-MIB.txt -c public 22.214.171.124 outputVoltage
To return the set voltages of one HV channel (e.g. u400 – first channel in slot 4):
snmpget -v 2c -m +./WIENER-CRATE-MIB.txt -c public 126.96.36.199 outputVoltage.u400
To set the set voltages for HV channel u400 to 450V:
snmpset -v 2c -m +./WIENER-CRATE-MIB.txt -c guru 188.8.131.52 outputVoltage.u400 F 450
To turn on HV channel u400:
snmpset -v 2c -m +./WIENER-CRATE-MIB.txt -c guru 184.108.40.206 outputSwitch.u400 i 1
To turn off HV channel u400:
snmpset -v 2c -m +./WIENER-CRATE-MIB.txt -c guru 220.127.116.11 outputSwitch.u400 i 0
To find out all generic options used with SNMP commands, type the following command in the terminal window.
% man snmpcmd
In the example commands
-v 2c specifies the protocol version to be used.
-m +./WIENER-CRATE-MIB.txt is the path to MIB file.
-c guru sets the community string for SNMP v2c transactions. It basically gives the administrative privileges.
Next term in the command is the IP / Network address of the Power supply.
IP address is then followed by the system, output entry etc. you want to set, output on a screen or turn ON/OFF. The full list of systems, variables can be found in MIB file.
i - stands for integer (note lower case, important).
i 0 - means turn the Switch "OFF".
i 1 - means turn the Switch "ON".
F 450 - will set the Voltage of u400 channel to 450.0 V Float value.
Two important snmp commands that allows to search and clear inhibit bits without power cycling the HV crate:
Find out the status of the inhibit bits:
snmpwalk -v 2c -m ./WIENER-CRATE-MIB.txt -c guru nova-far-hv-power-01.fnal.gov outputStatus
The following command will reset the inhibit status bits for u104 channel
snmpset -v 2c -m ./WIENER-CRATE-MIB.txt -c guru nova-far-hv-power-01.fnal.gov outputSwitch.u104 i 10
To clear the error bits of HV power supply channels, run the following command:
snmpset -v 2c -m +./WIENER-CRATE-MIB.txt -c guru NNN.NNN.NNN.NNN outputSwitch.u$channelnumber i 10
snmpset -v 2c -m +./WIENER-CRATE-MIB.txt -c guru NNN.NNN.NNN.NNN groupsSwitch.uNNN i 5
Set groupsSwitch variable to 5(enable) or 4 (disable) to enable disable 'Enable kill'
It doesn't work for the LV power supplies. In order to clear LV PS error bits, you need to turn off the channel.
Word of Caution: Please be aware that while on the Fermilab network, executing any of the SET / WRITE commands (or Shell scripts) even by accident WILL change the power supply settings.
Before operating or changing any of the PS settings, PLEASE READ the Operation Manual (Manual_for_PDS.pdf) in its entirety. Although it was written for NDOS it is also applied to NOVA Far Detector Power Supply controls. Also familiarize yourself with the commands and instructions from MIB.txt file.
Editing / Creating Synoptic Displays¶
Detailed instructions can be found in Manual_Synoptic-1_3.pdf below.
The following video was created by the developers as a reference material for editing and creating Synoptic Displays.
The URL of the video is: http://vmsstreamer1.fnal.gov/Training/AD/120302ADSynoptic/index.htm
If you are not an experience Synoptic developer, DO NOT edit displays in the central repository.
Download the entire Display tree into your local disk and edit it there.
To Download the Display tree:
1. Double click on Launch Builder. This will prompt you to run the .jnlp java script. Running the script will open
the builder Synoptic window.
2. From the main menu go to CVS and click Reload Tree. This will create Synoptic directory on you local disk with links to actual displays. It will not download the actual display.
3. To access and edit displays, open the builder window by running java script from the website. Do reload tree.
When building PS displays for ND power supplies, I downloaded several displays from Nova_Far (FarDet displays) by doing File -> Save as ... from builder window.
You can modify the displays and run them to see changes. Dealing with sub-displays is a bit tricky. You need to change the properties to make sure that your sub-display points into the right directory. Synoptic automatically looks for sub-displays in the central repository.
Once you are confident that your changes are correct than you can commit them into the central repository. Again, you need to change the properties of the sub-displays to make sure they are pointing the the right directory before you commit them to CVS central repository. To avoid changing the pointers back and forth, I committed displays to CVS to test the sub-displays.
Synoptic displays get the information from the ACNET system. Every power supply has several ACNET devices associated with it, the names of which were set up by ACNET group and given to me.
The ACNET device names for Near Detector power supplies are set up according to the convention of the FarDet power supplies for consistency.
where bb is the diblock number, pp is the position number. bb can go from 01-03. For muon catcher we use 0M instead of 04. Position numbers for from 01-04. for Muon catcher we have only 2 PDBs (only 2 positions). [MV] is the name of the variable such as Measured Voltage etc.
L:bbppCF is the device from which the 12 configuration parameters are read.
Device names for the ND crates are:
E:0101MS for Crate 1.
E:0102MS for Crate 2.
E:0201MS for Crate 3.
E:0202MS for Crate 4.
E:0203MS for Crate 5.
Full list of Device names can be found in DocDB# 8900-v6, also in the UpdatedCatalog-of-ACNET-devices-for_data-logger-plotter.xlsx file attached below.
Most Relevant Documents¶
On Call Phone List: http://nova-docdb.fnal.gov:8080/cgi-bin/RetrieveFile?docid=8806
Shifter information: https://cdcvs.fnal.gov/redmine/projects/novaoperations/wiki/NOvA_Shifts
Shifter information Latest issues: https://cdcvs.fnal.gov/redmine/projects/novaoperations/wiki/ShiftBulletinBoard
DCS processes: Gennadiy's instructions: https://cdcvs.fnal.gov/redmine/projects/novadcs/wiki/Regional_IOC_and_CSS_GUI_?version=38