SISPI Daily News¶
Quirks and Features - Operational issues you should be aware of¶
- Spurious Glitches in the ICS STATUS Flags (6/13/2014)
We had occasions (very infrequent) where the ICS (cRIO) reports spurious errors for example for the shutter in the ICS STATUS field that gets recorded in the database. This field is tied to the ICS interlock and when this happens the interlock breaks. The glitch goes away with the next update but of course the interlock remains broken. If this occurs please inform an expert right away. The fix requires action on the cRIO that only Marco (or somebody he trained) should do. You can continue observing albeit with reduced efficiency by clearing the interlock using the RESET button and restarting the exposure queue.
- No sound on observer1 (2/2/2014)
Sometimes people mute the sound. Go to monitor Obs1-B, upper left corner, where you should see "Applications Places System". Click on "System", go to Preferences, scroll down to Sound, and click. Make sure that the volume control is unmuted and sound level is up.
When you prepare the system for the night check out the audio system on observer1. This includes the microphone so that you can use skype in case you need to contact an expert during the night. Also make sure that the volume is set to a reasonable level (black, puck like object under the screens). SISPI uses a number of audio alerts: end of exposure, queue empty, alarms and the chat ring. You can test your setting with the chat ring button. If you click that you should here an old fashioned telephone ring.
- VSUB turn on confirmation (10/9/2013)
On the SISPI Observer Console GUI, the "Turn VSUB on" button now includes a confirmation dialog box (but not for turning VSUB off). So turning VSUB on will now require two mouse clicks. This will make it harder to accidentally turn on VSUB at a time where that could damage the CCDs.
- kentools error: Unexpected eof in HdrRead (9/15/2013)
When you try to run kentools on a new exposure and you see this error, it is because you are trying to load the image is KenTools before it has been transferred onto the disk -- if you get this error -- wait a little longer, and try again. (thanks to DJJ for this tip)
- GCS "CONNECTION ERROR" alarm (9/06/2013)
If the GCS breaks an interlock, and the GCS logs mention a "CONNECTION ERROR", there is probably also a log in PANG about a socket error when the command "set roi full" times out. This can be fixed by configuring.
- Obstac and the Queue (9/01/2013)
Obstac is designed to keep the exposure queue filled and running. We have noticed a few incidence where it appears that obstac disabled itself or when it let the queue go empty. The exact cause is still under investigation. For an observer this means that you have to keep an eye on the queue and the GO/STOP button. If this happens it can usually by fixed by re-enabling obstac and if necessary clicking the GO button.
- Menu Bar Dropdown on Observer1 Disappeared (8/31/2013)
There is supposed to be a dropdown (called a "Drawer") in the menu bar (called a "Panel") on observer1 that makes it convenient to run observer_setup and observer_cleanup scripts and start the Scripts Editor. Sometimes, for unknown reasons, this disappears. Compete directions on recreating this can be found here.
- Exposure Requests and Propid (2/20/2013)
Due to some recent incidents we would like to remind you that propid, observers, program and similar quantities are set when you add the exposure request to the queue. Once a request has been added to the queue it can no longer be modified. Your only option is to delete the request and to resubmit. In particular, any changes to the propid using the Observer Console will not affect exposure requests already in the queue.
- Yellow Vsub Indicator (2/8/2013)
The Vsub indicator on the Observer Console can be green (Vsub on), red (Vsub off) or yellow (Vsub mixed). In the mixed state at least one (Monsoon) crate (but not all) has Vsub on. If SISPI can't communicate with a crate (ie the panview process on the corresponding readout machine) it considers Vsub for this crate to be on. This makes this Vsub indicator a great tool to discover startup problems. Occasionally, for reasons we don't yet understand, some of the panview processes don't start properly when the SISPI instance (re-)starts. There are many indicators if this happens (e.g messages in the PSL xterm or missing messages in the green panview xterm). You will also not be able to configure. But most of this is not readily available/visible whereas the Vsub indicator is right there on the GUI. A yellow light after a restart most likely means that one of the panview processes didn't start properly. If you notice this, open the PSL xterm window and it should give you some indication which PAN needs to be restarted. Once you have this information go to the Architect Console, select the appropriate PAN role and click restart. In the system1 vnc session (if that's up) you should see the panview process shutdown and Labview is restarting.
- Test exposures (6/30/2013)
Sometimes it necessary to read out the CCD array and have the image completely ignored by DESDM. One option is to change the proposal id to 2012B-9993. Alternatively, you can manually submit an exposure request. To do this hit the Manual button on the Exposure Control panel and enter the following.
obsType=zero, object=junk, dtsqueue=none
- Default Instance Name (1/30/2013)
In order to avoid confusion with the SISPI instance name we modified the Architect to use a default value when the -i option is not used. For example, the command
architect -c DES.ini
will start a SISPI instance named DECam_<today's date>. On January 30, 2013 this would be DECam_20130130. To use the same name for all instances from a single night (in case SISPI had to be restarted) we set the time to change the date to 10 am local CTIO time. The -i option can still be used to set a special instance name.
- Vsub Details (1/30/2013)
Triggered by some recent events we added a new call to SISPI to make it easy to get detailed vsub information for all Monsoon crates. Typically vsub is either on or off for the entire system (all 6 Monsoon crate (counting each split backplane as a crate). There are several indicators on the SISPI GUIS to show this state. Sometimes the crates are in a mixed state - some on, some off. In this case the vsub indicator on the ObserverConsole GUI turns yellow. To find out which crate and which module (slot) in each crate has vsub on you can now use the new command get vsub details. Go to the Architect Console, select the OCS role and enter the command in the control section. This returns a string like this:
FPANS: SL4_CCD12_VSUB_ENBL 0.000000 SL3_CCD12_VSUB_ENBL 0.000000 << _decam_bkpf SL4_CCD12_VSUB_ENBL 0.000000 SL3_CCD12_VSUB_ENBL 0.000000 << _decam_bkp4 SL4_CCD12_VSUB_ENBL 0.000000 SL5_CCD12_VSUB_ENBL 0.000000 SL6_CCD12_VSUB_ENBL 0.000000 << _decam_bkp5 SL4_CCD12_VSUB_ENBL 0.000000 SL5_CCD12_VSUB_ENBL 0.000000 SL6_CCD12_VSUB_ENBL 0.000000 << _decam_bkp1 SL4_CCD12_VSUB_ENBL 0.000000 SL5_CCD12_VSUB_ENBL 0.000000 SL6_CCD12_VSUB_ENBL 0.000000 << _decam_bkp3 GPAN: 0.000000
GPAN is the guider backplane (bkpg). A 0.000000 indicates that vsub is off for this module. A 1.000000 indicates that vsub is on.
- Hand-Over Procedure (11/16/2012)
SISPI uses the information from the Setup block in the System Control section of the Observer Console GUI to distinguish different observing programs, to set FITS header keywords and to control the flow of image data to NCSA and NOAO. It is important to set this information correctly at the start of your observing period and to clear it when you are done. The agreement between DES and CTIO is that whoever has the first half night clears the setup information at the end of the first half of the night. Clearing this information is very simple:
- Go to the System Control section of the observer console GUI and clear the setup information
- Click on Edit
- Click on Clear
- Click on Save
- Go to the System Control section of the observer console GUI and clear the setup information
- Master Console Holder No Longer Exists (11/16/2012)
Sometimes the GUIs get slow and we restart them, or we've restarted the SISPI instance, logged in to the GUIs, then remember to restart them because that's easier than refreshing each one. When this happens, the new GUIs have a new session, but the old session still has Master Console. To reset this deadlock and to restart the GUIServer use a shell window (sispi account), execute setup SISPI and run the command releaseMasterConsole <instance name>.
- GCS/PANG (11/11/2012)
This GCS/PANG error has been seen too often recently. There is no clear diagnosis yet but in many (but not all to make it more difficult) the problem arises then PANG does not receive (or process) the stop_exposure command at the end of the exposure. In these cases panview doesn’t receive the stop command and the guider continues to run (leading, for example, to the new stripe issue the eye ball squad found)
Here is a quick recipe that often (but again not always) works:
(all in the architect console)
- select GCS
- enter stop_exposure in the command field and submit.
(If you select PANG there should be some indications in the log the EXPOSE is DONE)
(Note for SISPI team - PANG stop does not work from the Architect Console but works fine from the text console)
- select GCS
- enter configure in the command field and submit
On the observer console
- try reset
- if that fails try configure
- if that fails you need to restart the instance
- DECAL fails to prepare for exposure (10/19/2012)
Every once in a while the DECAL software acts up and fails to get ready for the next exposure. You see a message in the log section on the Observer Console. Try to restart DECAL with the architect console and also try to configure. When this doesn't help we assume that the DECAL labview software is in a bad state. If no DECAL expert is available and you want to continue you need to remove the DECAL component from the instance.
Using the architect console enter
OCS OCS remove_component DECAL
Then configure and you should be able to continue.
Once you do this, DECAL is no longer available and the automatic LED control for dome flats will not work (If needed you can turn on the LEDs manually)
- Runaway Guider (10/19/2012)
We had one incident where the shutter didn't close and the guider didn't stop. This blocked any further exposures. In this situation (it might never happen again) go to the ICS gui, open the Shutter controls (not manual) and type in close in the command field.
- Finding the mouse (10/02/2012)
The mouse pointer is really easy to loose track of in those eight monitors. You can now get some help locating it by pressing the ctrl key on the keyboard; this will make some concentric circles radiate out an inch or so from the pointer. This can be enable/disabled through the toolbar's System dropdown -> Preferences -> Mouse. This brings up a new window with two tabs; the "General" tab has a "Locate Pointer" section with a check box to turn this behaviour on and off.
- Fill in the Setup Information (10/02/2012)
Don't forget that you need to fill in the setup information (observer name, proposal ID - the other information can be retrieved for you from the database). You have to do this at the beginning of the night and after restarting the OCS or restarting SISPI.
- Comfort Display Restart (10/02/2012)
Instructions to start (or restart) the comfort display can be found here.
The first thing you should do is to check the status of the DISPIB role (the display image builder that drive the comfort display). If this encountered a problem, for example when trying to build png images it disables itself and needs to be "kicked" to continue. In the Architect Console select the DISPIB role in the left view so that you can see the log messages and then enter: IB DISPIB configure in the component, device and command fields.