Putting Data Management Results into the Database


After the data is taken on the mountain, Data Management (DM) does a careful quality assessment to determine which exposures are good enough for the survey (or supernova group) and which should be rejected. An exposure that has been processed by DM and not accepted into the survey results will be noticed by ObsTac (the Observation Tactician) and retaken. It is the Run Manager's job to get the assessments from DM in to the mountain top database.

Using the GUI tool to fetch Data Management results and apply them to the SISPI database

Log in as user sispi to any SISPI machine at CTIO. If you are doing so remotely, be sure to have an X11 connection either by X Tunnelling with "ssh -X sispi@system1" (or "ssh -X" to any other SISPI computer) or by running the commands from inside the sispi vnc. The password needed to ssh into the system is the usual sispi password (check the white board or ask the Ops team if you can't find it). If you log into the machine from the DECamObserver account on observer2, you may not need a password at all.
Then run

setup RunManagerTools

When you run apply_data_gui, it will look like this:

Click on the "Fetch and Load" button to fetch any new results from Data Management's DES Operations database and display them in the GUI. (NOTE: There may be a noticeable delay between the moment when the button "releases" and data appear in the GUI -just be patient. You can watch the "File: Loading... (X/Y)" next to the Load File button to see the progress.) Note that it is possible to receive updates for exposures that were taken a while ago for example if DES DM reprocessed the data. The fetched data will be written to a csv file in ~sispi/decam/data_management_logs, and any csv file of results can be opened using the "Load File" button. Note that writing a file is part of the "Fetch" step, so when you just load a file nothing new will be written to that directory. Unless you have a good reason to do otherwise, though, just use the Fetch and Load button. It will now look something like this:

You can use the "Select All" and "Select None" buttons to set everything in the "Accept?" column, or click on individual check boxes for more fine grained control:

The "Accept?" column means that you accept Data Managment's assessment of the exposure, whether good or bad, and that you would like to update the mountain-top database to reflect Data Management's assessment. Most of the time, you will want to accept all of Data Management's assessments. Once you are sure about your selections, press "Apply" to put the changes into the database.

(NOTE: You should record the exposure ID for ANY exposures you choose not to apply so that they can be kept track of -- you can't fetch and load them again later).

After applying the changes to the database, you be shown a confirmation dialog box. When you click "OK", the program will exit.

(NOTE: It is OK to do this in the middle of the night if you didn't do it sooner--it won't mess up the data from that night.)

Command Line Tools

You can query the mountain top database about the Data Management status of particular exposures without using the GUI using the command "check_dm". Log in as user sispi and run

setup RunManagerTools

Here is an example:

[sispi@system1 ~]$ check_dm 213260 213261
Exposure dm_done dm_accept
00213260  False    False
00213261  False    False