SISPI Operations

This page outlines SISPI operations during commissioning, science verification and the start of regular observing. Details are still pretty much in flux. For the purpose of this discussion we distinguish four groups: regular observers, DECam experts (e.g. electronics, cooling etc), operations experts (CTIO instrument scientist, DES run manager), and the SISPI team.

SISPI Accounts

SISPI uses several user accounts on the DECam computer cluster (besides the groups mentioned above we also have the the quick reduce team and the NOAO data transfer group)

Account Purpose Users
DECamObserver General purpose account All observers
sispi Start/control SISPI instance operations experts, SISPI team
Configuration management
codemanager software and release management SISPI team
postgres database management SISPI team
quickreduce quick reduce operations QR team
bits E2E data transfer system SDM team

SISPI GUI Accounts

In order to observe with DECam one also needs to have a SISPI user account. The SISPI user account and corresponding password are used by SISPI to authenticate requests to access the graphical user interfaces. A SISPI user account can be obtained by emailing the SISPI team or by completing the sign up form displayed on the login page of every GUI.
Observers do not need to request their own SISPI user account. Instead they access the GUIs using the DECamObserver account and their proposal ID as password.
SISPI accounts support 4 different privilege levels:

Level Typical Users
user Observers, not at the Observer Console
observer Observers at the Observer Console
expert DECam and operations experts
root SISPI team

When connecting to a SISPI GUI the privilege level will be set to user. Privileged users can raise the level using the session management dialog from the SISPI GUI toolbar. With increasing level additional functionality becomes available. The table below shows the current implementation but note that this is still subject to fine tuning. Each higher level includes the functionality of the lower levels.

Authentication Level Allowed Commands
None (no user is logged in) help, ping, all non-PML console commands
user (non privileged) PML get
observer all PML commands but no direct access to Architect, ICS components, IB, PANs (PAN, PSL, FCS, GCS)
expert all commands
root all commands, session management

To allow for remote expert access under controlled circumstances an additional authentication layer was added that requires explicit approval from the observer at the DECam observer console in the Blanco console room for remote users before the authentication level can be changed.
This Master Console concept guarantees that the observer is always aware of who is connected to the current SISPI instance. If a remote expert is contacted to look into a problem, the expert will connect to the GUI system and request to raise the session level. The observer at the Master Console will be notified of the request and can grant the change in privileges. The observer at the Master Console can terminate other user sessions at any time. During regular observing the observer console in the console room will be configured as the Master Console. However, as described in the SISPI Expert guide, other configurations are possible and the DES run manager/CTIO instrument scientist can fine tune the use of the Master Console in the .ini configuration file.
An additional function of the Master Console concept is to automatically raise the privilege level of the observer session from user to observer. This works even for non-privileged user accounts. The Master Console level can be configured in the .ini configuration file.

Operations Plan

Here is a description of the current baseline plan SISPI operations. We use the term operations manager for the DES run manager or CTIO staff who is in charge for operations.

  • SISPI will be (re-)started every afternoon by the operations manager under the sispi account on system1 in a VNC session
  • The Master Console is enabled in the .ini configuration file and configured to use the 8-screen workstation (observer1) in the console room with the DECamObserver account. The Master Console level is set to expert
  • When arriving in the console room, the observer follows the instructions in the SISPI Setup Guide for Observers and starts desktop sessions on the 3 computers in the console room (observer1, observer2, comfort1) using the DECamObserver (linux) account. (Some operations manager might choose to perform these steps for the observers)
  • The observer starts the SISPI GUIs on the observer1 machine and signs in (to the GUIs) using the DECamObserver (SISPI user) account using the proposal ID for authentication. Note that there are two DECamObserver accounts = one for the SISPI GUIs and one for Linux. Both have different passwords.
  • The observer session becomes the Master Console and the privilege level is raised to expert. The observer can verify this by looking at the toolbar on the top of each SISPI GUI. A silver M is displayed for the Master Console session.
  • The observer will be able to configure SISPI, load the exposure queue, manage exposures and control obstac.
  • The observer can restart the GUIs
  • The observer can access any of the components directly. For example, commands like SHUTTER open 25.0 are available.
  • The observer has access to the process control functionality provided by the Architect Console GUI. For example, the observer can stop or restart individual processed. The observer can shutdown the SISPI instance.
  • The observer cannot restart SISPI (requires access to the sispi account)

Instructions about QR need to be added.

The observer does not control RASICAM. This is handled by the telescope operator.