SISPI Daily News¶
Quirks and Features - Operational issues you should be aware of¶
- Guider GUI Change (11/5/2014)
The Guider page in the web-based GUI has been updated. It now includes radio buttons to select settings for normal vs dense fields (which does not, at the moment, do anything because the values are identical), radio buttons to select clear or cloudy skies (when cloudy, the guider will be very tolerant of loosing a guide star, and continue to look for it to reappear), and a drop-down list of common guider exposure times with an option to enter a custom value that is not in that list. There is also a button to reset these three settings back to their defaults with one click.
- Architect Startup, especially with the Starfinder (8/26/2013)
Occasionally, the Architect will fail to start a role specified in the ini file. The role STARFINDER seems to have this problem the most often. This appears to be a problem with eups; if a role requires special setup and that setup fails, the role will not start. The architect has been updated to notice this error condition and try again; since the eups problems appear to be a race condition, this often solves the problem. The Architect has also been updated to log more information about this situation (and any PML command sent after the instance is up and running) for better debuging. Note that the Architect waits a few moments before retrying the eups setup; at OSU, I (Ann) have seen the starfinder take long enough to get started that I was able to log in to the Console, get a list of roles from the Architect, see that the starfinder was missing, all before the starfinder successfully started without my intervention. So if a role has not started, look at the Architect logs to see if it's still trying.
- Architect Option Change (2/28/2013)
The optional flags to run the cleanup scripts, and to speed up startup by not waiting on start_role calls have now been made the default. These features can now be turned off (instead of turned on) with command line flags (-k is now keep_processes instead of kill, and the old -n for no_wait became -r for return_after_start_role (in contrast to the default, which is to return immediately)). The cleanup performed by the architect now includes ramdisk cleanup as well as process cleanup.
- Exposure Requests and Propid (2/20/2013)
Due to some recent incidents we would like to remind you that propid, observers, program and similar quantities are set when you add the exposure request to the queue. Once a request has been added to the queue it can no longer be modified. Your only option is to delete the request and to resubmit. In particular, any changes to the propid using the Observer Console will not affect exposure requests already in the queue.
- Yellow Vsub Indicator (2/8/2013)
The Vsub indicator on the Observer Console can be green (Vsub on), red (Vsub off) or yellow (Vsub mixed). In the mixed state at least one (Monsoon) crate (but not all) has Vsub on. If SISPI can't communicate with a crate (ie the panview process on the corresponding readout machine) it considers Vsub for this crate to be on. This makes this Vsub indicator a great tool to discover startup problems. Occasionally, for reasons we don't yet understand, some of the panview processes don't start properly when the SISPI instance (re-)starts. There are many indicators if this happens (e.g messages in the PSL xterm or missing messages in the green panview xterm). You will also not be able to configure. But most of this is not readily available/visible whereas the Vsub indicator is right there on the GUI. A yellow light after a restart most likely means that one of the panview processes didn't start properly. If you notice this, open the PSL xterm window and it should give you some indication which PAN needs to be restarted. Once you have this information go to the Architect Console, select the appropriate PAN role and click restart. In the system1 vnc session (if that's up) you should see the panview process shutdown and Labview is restarting.
- Restarting the GUIDERGUI (2/8/2013)
The GUIDERGUI is different from all the other SISPI GUIs and requires special care if you need to restart it. The GUIDERGUI is not a web-based application. Instead it runs on readout2 in the vnc session using display 15. Typically a vncviewer running on observer1 is connected to this server and shows the Guider GUI. This particular configuration prevents the Architect from stopping the GUIDERGUI application when SISPI shuts down. To cleanup you need to use the cleanup_processes -k command or you need to use the -k option when you run the architect. Failure to do so will leave multiple copies of the GUIDERGUI running on readout2 which typically leads to bad observer confusion.
If you only want to restart the GUIDERGUI you can use the Architect Console but it's a bit more complicated then just restarting other SISPI applications. Note that this procedure will break the interlock and that you will have to run configure to restore the system. In other words don't do this in the middle of an observing sequence.
- On the Architect Console select the GUIDERGUI role and click stop
- You should see some messages about this and some child processes still running in the (terminal) windows where you are running the architect.
- In the readout2 vnc session on observer1 (or if you have to start a new viewer and connect to the vnc server) close the Guider GUI window and close the terminal window
- On the Architect Console select the GUIDERGUI role and click start
- When the GUI is back (readout2 vnc session) you need to run configure to reset the interlocks.
- When the Guider configures it will connect to the GUIDER GUI - no need to push the Connect Guider button.
- Test exposures (6/30/2013)
Sometimes it necessary to read out the CCD array and have the image completely ignored by DESDM. One option is to change the proposal id to 2012B-9993. Alternatively, you can manually submit an exposure request. To do this hit the Manual button on the Exposure Control panel and enter the following.
obsType=zero, object=junk, dtsqueue=none
- Default Instance Name (1/30/2013)
In order to avoid confusion with the SISPI instance name we modified the Architect to use a default value when the -i option is not used. For example, the command
architect -c DES.ini
will start a SISPI instance named DECam_<today's date>. On January 30, 2013 this would be DECam_20130130. To use the same name for all instances from a single night (in case SISPI had to be restarted) we set the time to change the date to 10 am local CTIO time. The -i option can still be used to set a special instance name.
- Vsub Details (1/30/2013)
Triggered by some recent events we added a new call to SISPI to make it easy to get detailed vsub information for all Monsoon crates. Typically vsub is either on or off for the entire system (all 6 Monsoon crate (counting each split backplane as a crate). There are several indicators on the SISPI GUIS to show this state. Sometimes the crates are in a mixed state - some on, some off. In this case the vsub indicator on the ObserverConsole GUI turns yellow. To find out which crate and which module (slot) in each crate has vsub on you can now use the new command get vsub details. Go to the Architect Console, select the OCS role and enter the command in the control section. This returns a string like this:
FPANS: SL4_CCD12_VSUB_ENBL 0.000000 SL3_CCD12_VSUB_ENBL 0.000000 << _decam_bkpf SL4_CCD12_VSUB_ENBL 0.000000 SL3_CCD12_VSUB_ENBL 0.000000 << _decam_bkp4 SL4_CCD12_VSUB_ENBL 0.000000 SL5_CCD12_VSUB_ENBL 0.000000 SL6_CCD12_VSUB_ENBL 0.000000 << _decam_bkp5 SL4_CCD12_VSUB_ENBL 0.000000 SL5_CCD12_VSUB_ENBL 0.000000 SL6_CCD12_VSUB_ENBL 0.000000 << _decam_bkp1 SL4_CCD12_VSUB_ENBL 0.000000 SL5_CCD12_VSUB_ENBL 0.000000 SL6_CCD12_VSUB_ENBL 0.000000 << _decam_bkp3 GPAN: 0.000000
GPAN is the guider backplane (bkpg). A 0.000000 indicates that vsub is off for this module. A 1.000000 indicates that vsub is on.
- Hand-Over Procedure (11/16/2012)
SISPI uses the information from the Setup block in the System Control section of the Observer Console GUI to distinguish different observing programs, to set FITS header keywords and to control the flow of image data to NCSA and NOAO. It is important to set this information correctly at the start of your observing period and to clear it when you are done. The agreement between DES and CTIO is that whoever has the first half night clears the setup information at the end of the first half of the night. Clearing this information is very simple:
- Go to the System Control section of the observer console GUI and clear the setup information
- Click on Edit
- Click on Clear
- Click on Save
- Master Console Holder No Longer Exists (11/16/2012)
Sometimes the GUIs get slow and we restart them, or we've restarted the SISPI instance, logged in to the GUIs, then remember to restart them because that's easier than refreshing each one. When this happens, the new GUIs have a new session, but the old session still has Master Console. To reset this deadlock and to restart the GUIServer use a shell window (sispi account), execute setup SISPI and run the command releaseMasterConsole <instance name>.
- Frozen OCS (11/11/2012)
Occasionally we observe that the OCS is not doing anything even though there are image requests in the queue and no interlocks are broken. You also notice that when you try to stop the queue (Hit the GO/STOP button) the queue will not be disabled (and the Stop label changes to Go). In this case it's likely that one of the exposure loop threads is holding on to a lock (a python process synchronization tool). We are working on preventing this from happening but it still happens. In this situation, use the Architect Console and select the OCS.
In the command field enter get runstate
If this returns RUNNING the OCS is indeed in the state just described. You can try to recover using the clearLocks command. In the Architect Console select the OCS and enter the command clearLocks.
Note The OCS can also look stalled when you are waiting for the TCS to complete the slew. So before you do anything you might want to check the ICS/TCS gui (is the Stop Slew button displayed?) and/or ask the telescope operator
Note 2 on 11/12/2012 This occurred tonight on exposure 150599 when the shutter errored. There were errors in the shutter logs but no alarms and no interlocks were broken. The runstate was "RUNNING" so then we did a clearLocks, which stopped the queue. We were then able to press Go without a reconfigure or a reset, and the rest of the queue completed successfully. Perhaps this means that the shutter should break its interlock when this error happens.
- GCS/PANG (11/11/2012)
This GCS/PANG error has been seen too often recently. There is no clear diagnosis yet but in many (but not all to make it more difficult) the problem arises then PANG does now receive (or process) the stop_exposure command at the end of the exposure. In these cases panview doesn’t receive the stop command and the guider continues to run (leading, for example, to the new stripe issue the eye ball squad found)
Here is a quick recipe that often (but again not always) works:
(all in the architect console)
- select GCS
- enter stop_exposure in the command field and submit.
(If you select PANG there should be some indications in the log the EXPOSE is DONE)
(Note for SISPI team - PANG stop does not work from the Architect Console but works fine from the text console)
- select GCS
- enter configure in the command field and submit
On the observer console
- try reset
- if that fails try configure
- if that fails you need to restart the instance
- Master Console is Activated (11/1/2012)
The master console feature is activated in DES.ini. This means that you have to login as DECamObserver (+ propid password) on one of the GUIs on observer1.
It has to be the observer1 computer and it has to be DECamObserver account, even if you have your own SISPI account. You can use your personal GUI account but you will only have user privileges and cannot change this level unless approved by the observer sitting at the master console. The level of the master console session will be automatically raised to Expert. If you want to authorize some other session (e.g. a remote expert) to raise its level you need to open the control field by clicking on the button with the person icon in the toolbar. This displays a list of active sessions and pending requests. Click on "+" to approve a request.
- Audio Settings (10/31/2012)
When you prepare the system for the night check out the audio system on observer1. This includes the microphone so that you can use skype in case you need to contact an expert during the night. Also make sure that the volume is set to a reasonable level (black, puck like object under the screens). SISPI uses a number of audio alerts: end of exposure, queue empty, alarms and the chat ring. You can test your setting with the chat ring button. If you click that you should here an old fashioned telephone ring.
- Baseline F&A Configuration for SV (10/31/2012)
The default configuration file (sv.ini) for science verification configures the system to use the telescope and filter look up tables, to accumulate tweaks from the AOS system and to initialize the trim z coordinate to z = 1400.0 - 110.0 * temperature where temperature is the average upper truss temperature. The temperature look up table is not used. You should not change any of these settings unless you really know what you are doing. The only thing for you to do is to push the Init Z trim button before you start to observe (once the dome has been opened and the telescope is at ambient temperature). This will clear any tweak value and set the z trim value for the current temperature. Note that the temperature is obtained from the TCS so the TCSInterface has to be running and configured at this time.
- DECAL fails to prepare for exposure (10/19/2012)
Every once in a while the DECAL software acts up and fails to get ready for the next exposure. You see a message in the log section on the Observer Console. Try to restart DECAL with the architect console and also try to configure. When this doesn't help we assume that the DECAL labview software is in a bad state. If no DECAL expert is available and you want to continue you need to remove the DECAL component from the instance.
Using the architect console enter
OCS OCS remove_component DECAL
Then configure and you should be able to continue.
Once you do this, DECAL is no longer available and the automatic LED control for dome flats will not work (If needed you can turn on the LEDs manually)
- Runaway Guider (10/19/2012)
We had one incident where the shutter didn't close and the guider didn't stop. This blocked any further exposures. In this situation (it might never happen again) go to the ICS gui, open the Shutter controls (not manual) and type in close in the command field.
- Finding the mouse (10/02/2012)
The mouse pointer is really easy to loose track of in those eight monitors. You can now get some help locating it by pressing the ctrl key on the keyboard; this will make some concentric circles radiate out an inch or so from the pointer. This can be enable/disabled through the toolbar's System dropdown -> Preferences -> Mouse. This brings up a new window with two tabs; the "General" tab has a "Locate Pointer" section with a check box to turn this behaviour on and off.
- Fill in the Setup Information (10/02/2012)
Don't forget that you need to fill in the setup information (observer name, proposal ID - the other information can be retrieved for you from the database). You have to do this at the beginning of the night and after restarting the OCS or restarting SISPI.
- Comfort Display Restart (10/02/2012)
Instructions to start (or restart) the comfort display can be found here.
The first thing you should do is to check the status of the DISPIB role (the display image builder that drive the comfort display). If this encountered a problem, for example when trying to build png images it disables itself and needs to be "kicked" to continue. In the Architect Console select the DISPIB role in the left view so that you can see the log messages and then enter: IB DISPIB configure in the component, device and command fields.
Resolved Issues (at least we hope so)¶
Notes for SISPI Expert Use¶
- I don't show up in the users list! (1/16/2013)
To help debug some other problem, I logged in to the GUIs as myself and requested expert privs from the master console. They never got alerted, and in fact I did not even show up in the list of users. I was still able to get root access, but even then the observers couldn't see that I was logged in, and I did not see myself in the GUI user list. While debugging the original issue, I discovered SVE errors after a role restart. The SVE errors explain the odd GUI behaviour, and a few other strange things in the GUIs. Once the SVE was discovered to be the problem, the instance was restarted (which is the required fix for SVE problems). So if you see lots of odd behaviour, particularly of things in the GUI not updating, after restarting a role, check for SVE errors.