Obstac disaster workarounds


These instructions are intended to let DES observers (and those that support them) influence observing when obstac developers are not available. The instructions included here start off with simple tasks, such as forcing the observation of a given SN sequence at a given time, and get increasingly complex, ultimately arriving at what to do if you need to observe for an extended period of time without obstac at all.

These instruction assume some understanding of SISPI. Some should only be followed by experienced observers who understand how to make sure they do not point at the moon, observe at high airmass, and similar. Some require some basic python or SQL skills.

Observe a specific SN sequence by hand (no obstac necessary)

Any of the SN observing sequences can be completed by hand using SISPI scripts. There are on the SISPI host machines, and attached to this page as SN-C1.json, SN-C2.json, SN-C3-g.json, etc.

If these are loaded into SISPI, it will just do them. It is up to the observer to make sure it is sensible to do them, and verify that they are not too close to the moon, at too high an airmass, and similar considerations obstac normally takes care of.

Using these script files will set the appropriate metadata in the SISPI database for obstac to understand that the exposures have been completed.

Observe specific DES wide-survey exposures by hand (no obstac necessary)

Attached to the page is all_wide.json, a SISPI json observing file with every DES wide survey exposure. DO NOT ATTEMPT TO LOAD THIS FILE INTO SISPI. It is never an appropriate set for any particular time, and who knows what will happen if you load 80000+ exposures into the SISPI queue.

Instead, extract the subset of exposures you want in your observing script, in the order you want them, to generate a new observing script. It is up to you to figure out when they are at an appropriate airmass, far enough from the moon, etc. When exposures are extracted from this script rather than entered by hand, the metadata in the SISPI database will be set so that obstac recognizes the exposures as having been completed.

Here is an example script to perform such an extraction. It creates an observing queue with the set of tiling 5 z and Y exposures in a box bounded by RA and declination, ordered by RA.

import json

with open('all_wide.json','r') as f:
    all_wide = json.load(f)

obs_queue = [o for o in all_wide
             if 30 < o['RA'] < 45
             and -55 < o['dec'] < -50
             and o['filter'] in ('z','Y')
             and o['tiling_id'] == 5]

from operator import itemgetter
obs_queue = sorted(obs_queue, key=itemgetter('RA'))

with open('my_wide_queue.json','w') as f:
    json.dump(obs_queue, f, indent=True)

Note that RA is in degrees, runs from 0 to 360, and must do so when the queue is written as well: SISPI will not accept negative values.

Note that this script includes all wide survey exposures that fit your criteria, including those that have been completed already or are declared to be low priority.

Prevent obstac from scheduling SN

This procedure will not prevent SN from being added by hand using the SISPI json scripts described above.

The simplest way to prevent obstac from scheduling SN is to set the deadman gap and the minimum gap to something long (like 3650 days) in the obstac configuration file, such that the section that nominally looks like this:

[Supernova Tactician]
airmass_limit = 1.5
query_limit = 20
max_interval = 7
min_interval = 4

becomes this:
[Supernova Tactician]
airmass_limit = 1.5
query_limit = 20
max_interval = 3651
min_interval = 3650

This change will prevent obstac from scheduling any SN sequences within 10 years of the list time they were observed.

See this wiki page for the location of the configuration file and how to get obstac to pick up changes in it.

Prevent obstac from scheduling specific wide-survey exposures

The obstac.obs_to_ignore table in the SISPI database lists exposures obstac should avoid scheduling. It has three columns: hex_id, tiling_id, and filter_id. Note that filter_id is an integer index for the filter, not the filter name. ('g' is 1, 'r' is 2, 'i' is 3, 'z' is 4, 'Y' is 5.)

The obstac.wide view contains a list of all wide-survey exposures, and so is generally a good starting point from which to select things to add to obstac.obs_to_ignore. Note that obstac.wide has both filter and filter_id columns, so you can select based on the one and insert into obstac.obs_to_ignore from the other.

For example, if you want obstac never to schedule g exposures in tiling 6, in an area with declination between -40 and -50 and RA between 10 and 20 degrees, you can do this in the SISPI database:

INSERT INTO obstac.obs_to_ignore (hex_id, tiling_id, filter_id) (
  SELECT hex_id, tiling_id, filter_id
    FROM obstac.wide
    WHERE filter='g'
      AND tiling_id=6
      AND decl BETWEEN -50 and -40
      AND ra BETWEEN 10 AND 20

Note that you must have write permission to the SISPI database to do this. (Talk to Klaus H., Liz B., or Ann E.)

Prevent obstac from scheduling a specific SN sequence

The obstac.obs_to_ignore table in the SISPI database can also be used to prevent obstac from scheduling specific SN sequences (see above). The obstac.sn_status table plays the same role for SN that the obstac.wide does for wide-survey exposures. For example, to prevent obstac from observing the SN-C3 g sequence:

INSERT INTO obstac.obs_to_ignore (hex_id, tiling_id, filter_id) (
  SELECT hex_id, tiling_id, filter_id
    FROM obstac.sn_status
    WHERE hex_name='SN-C3'
      AND filter='g'

Observing wide exposures that have not been previously observed (without obstac)

The instructions above for observing DES wide exposures without obstac did nothing to avoid re-observing already completed exposures. This variation of it filters out completed exposures.

The first step is to create a file listing the hex_id, tiling_id, and filter of every completed exposure. There are several ways to do this.

Getting completed wide survey exposures, method 1: querying the SISPI database

You can query the state of all hex/tiling/filter combinations in the SISPI database thus:

The SISPI database is how obstac itself knows what it has done already. The query to get a list of completed hex_id, tiling_id, filter tripplets is:

SELECT hex_id, tiling_id, filter, done FROM obstac.wide

Save the results to disk, and then use awk to remove the "done" column and the rows where done is false. The whole process looks like this:

$ psql -h -p 5443 -d decam_prd -U decam_reader \
> -F ' ' -A -t \
> -c 'SELECT hex_id, tiling_id, filter, done FROM obstac.wide' \
> -o wide_status.txt
$ awk '$4~/t/ {print $1, $2, $3}' wide_status.txt > wide_done.txt

If you are at CTIO, you can replace the database host name ( and port with the mountaintop host and port.

You might think you can replace the query with one which eliminates the need for the extra "awk" command, but postgresql sometimes (but not always) optimizes this very badly, and causes the query to take many minutes to finish. (obstac.wide is a view, rather than an actual table, so a lot of joins are going on behind the scenes.)

Getting completed wide survey exposures, method 2: nightsum data files

If there are database problems which prevent the use of method 1, a list of completed hex/tiling/filter combinations can be exctracted from data files linked to from the nightsum page. These are not necessarily "current": they contain the completion state as of the time nightsum was run.

Go to the most recent nightsum page, and scroll down to the section named "Survey Progress". Immediately to the right of the section title is a faint gray link, "json". This links to a json file (a different data schema from the SISPI queue one, though) with the state of all wide survey hexes as of the time nightsum was run on that night. Download the file, which will be named footprint.json.

A short python script can convert this file into a list of completed hex/tiling/filter combinations:

import json
with open('footprint.json', 'r') as f:
    all_wide = json.load(f)

done_wide = [w for w in all_wide if w['mjd'] is not None]

with open('wide_done.txt','w') as f:
    for w in done_wide:
        print >>f, w['hex_id'], w['tiling'], w['filter']

(If the "mjd" key is set in the footprint.json file, it is the night on which the exposure was completed.)

Selecting undone wide survey exposures

The wide_done.txt file (generated by either of the above two methods) can now be used to exclude exposures. The sample python code from the instructions above can now be modified to use this file thus:

import json

with open('all_wide.json', 'r') as f:
    all_wide = json.load(f)

with open('wide_done.txt', 'r') as f:
    done_wide = set()
    for line in f:
        hex_str, tiling_str, filter_name = line.split()
        hex_id = int(hex_str)
        tiling_id = int(tiling_str)
        done_wide.add((hex_id, tiling_id, filter_name))

undone_wide = [w for w in all_wide
               if (w['hex_id'], w['tiling_id'], w['filter']) not in done_wide]

obs_queue = [o for o in undone_wide
             if 30 < o['RA'] < 45
             and -55 < o['dec'] < -50
             and o['filter'] in ('z','Y')
             and o['tiling_id'] == 5]

from operator import itemgetter
obs_queue = sorted(obs_queue, key=itemgetter('RA'))

with open('my_wide_queue.json','w') as f:
    json.dump(obs_queue, f, indent=True)

What if obstac is broken?

Okay, so obstac is just broken, and you need to observe. What now?

  1. Call the experts.
  2. Document everything in the elog
  3. Restart SISPI, and see if that fixes it.
  4. If not, revert to a previous version of obstac or the obstac.conf file, and see if that fixes it.

If none of these work:

Run obscript

obscript is a utility that is part of the obstac product. Instead of running as part of SISPI, it uses the same code to generate a SISPI json script instead. If the problem is in the interaction between obstac and SISPI, this should still run, even if obstac doesn't run as part of SISPI.

To run it:

$ setup Site
$ setup obstac
$ obscript --time 2013-08-30T00:00:01Z --duration 1.0  --config ~/obstac/DES/obstac-nosispi.conf --output somescript.json

The "--time" value should be set for the time (UT) at which you will be starting the queue. The "--duration" is the approximate time it should take to complete the queue, in hours.

In case of fire, break glass

If something is even more profoundly wrong and even obscript doesn't work, a variant of these instructions above can generate a set of SISPI observing scripts that can be run without relying on anything within obstac.

  1. Download all_wide.json
  2. Generate wide_done.txt, a file of complete exposures following one of the two methods here.
  3. Download, attached below, and run it in the same directory as all_wide.json and wide_done.txt.

Running will create a bunch of SISPI json queue files, all of a format that looks like this:


Each of these file contains a queue of undone exposures in a given tiling, in a set of filters, in a spherical quandrangle (area of the celestial sphere bounded by lines of constant RA and Declination) 15 degrees of RA by 5 degrees of declination. Exposures within each are ordered by RA, westernmost first, opposite the direction of the motion of the sky.

The file name shows the minimum RA and minimum declination of the 15 degrees by 5 degrees quadrangle. So, the file
queue_RA075_Dec-45_tiling5_zY.json has pointings between 75 and 90 degrees (5 and 6 hours) of RA and -45 and -40 degrees of declination.

So, if you need to observe wide and obstac isn't doing it, pick one of these, load it into the SISPI queue, and run it. Choose queues based on the following criteria:

  1. The pointings should be at an airmass better than 1.4 and far from the moon.
  2. If the moon is up, observe in zY. If it is down, in gri.
  3. Do not observe in more than one tiling on the same area of sky in the same filters on the same night.
  4. If the Local Sidereal Time is earlier than about 15 degrees (1:00) or greater than 300 degrees (20:00), observe blocks that are near transiting (those where the LST at the start of the queue is about equal to the RA in the file name). If the LST is between 15 and 300 degrees (1:00 and 20:00), observe as close to an RA of 15 degrees as airmass will allow.

If you need to do this on multiple consecutive nights, regenerate wide_done.txt each night, and run again using the new wide_done.txt, to generate new observing queues. Exposures completed like this will get recorded correctly as done in the database, and so wide_done.txt will include exposures completed like this.