Project

General

Profile

Obstac for observers

DES observers should read from here through the "Surprises and gotchas" section

What obstac does

During observing, obstac has two states: autoobs=on and autoobs=off. When autoobs is off, obstac does nothing. When autoobs is on, it monitors the length of the queue and, whenever the duration of the queue falls below the configured targed queue duration, it adds observations to the queue until the duration exceeds target queue duration, and then stops until the next time the queue falls below it.

An indicator above the observing queue shows whether autoobs is enabled:

Currently, the configured queue duration is 900 seconds. This may be modified be setting the min_duration keyword in the queue section of the obstac configuration file, and restarting the obstac SISPI roles.

Turning obstac on and off

Autoobs can be turned of and off using the "Enable Auto" (or "Disable Auto") button in the exposure control tab of the observer console:

Redoing observations

If observers note that an exposure is unambiguously bad, and the observation needs to be repeated, observers can mark the exposure to be redone. Find the offending exposure in the exposure table page of the SISPI interface, and select it to bring up details on that exposure. One of the details is a "redo" checkbox (row 4, column 5 in the gray area); checking this checkbox will mark it as needing to be redone.

For older exposures, enter the exposure id in the "Exposure ID" field in the upper right of the exposure details section (lower panel) of the exposure table page, and hit the "Select" button to its immediate right to bring up the exposure details with the redo checkbox.

Recent tests indicate that only one expid can be processed with this method. To process a second exposure the chrome browser "reload/refresh" button must be pressed, then a new exposure id loaded and redo selected. This may be true only for exposures from previous nights that must be loaded manually.

To confirm that your redo request was successful, go to the DB query window at [[http://system1.ctio.noao.edu:8080/TV/app/Q/execute]], select namespace exposure and use a query like this:

SELECT id, date, discard FROM exposure WHERE id IN(258466,258467, 258468, 258469, 258472, 258473, 258474, 258475, 258478, 258479, 258480, 258485, 258486, 258487, 258488, 258491, 258492, 258493, 258494) ORDER by id

Running the above query as an example should yield discard==true for each each exposure in the list.

Exposures can also be declared "redo" in bulk using the lower-level SQL interface to the database; see the instructions in "Obstac for SISPI experts" for more.

Do this only for object exposures

Do this only for exposures that are bad due to technical reasons, e.g. forgot to turn on Vsub. This should not be used for exposures that are bad due to clouds or seeing as setting this flag will instruct obstac to reschedule this exposure as soon as possible. Usually, clouds won't dissipate that quickly and you would only be wasting time by rescheduling the exposure right away.

The Redo mechanism is only for special cases, for most exposures the decision on image quality will be made by DES DM and it's the run manager's responsibility to handle this.

Surprises and gotchas

Everything here is work-arounds for bugs. The bugs will eventually be fixed.

Filling to completion

A fix for this bug is awaiting testing.

Once obstac starts filling the queue, it will not stop until the queue duration has reached its target duration (900 seconds), even if autoobs is turned off during the fill. It will stop after it reaches the target duration; the only thing to do is wait.

Obstac for SISPI experts

Configuring obstac

Finding the obstac configuration file

The obstac configuration file is set by the obstac_config_file option in the Architect "ini" file used to start the SISPI instance. The current value (as of 2013-09-05) is:

/usr/remote/user/sispi/obstac/DES/obstac.conf

The location of this file should not be changed without a good reason.

Switching to a new configuration

When the configuration file is changed, the obstac-related roles in SISPI must be restarted in order to pick up the changes. In particular, when AUTOOBS starts, the correctly configured version of OBSTACSRVR must already be running. So, the recommended procedure is:

  • Stop the AUTOOBS role,
  • restart the OBSTACSRVR role, and only then
  • start the AUTOOBS role.

Changing airmass limits

The airmass_limit options set the maximum airmass at which obstac is willing to schedule an observation. This option appears twice in the configuration file: once in the Supernova Tactician section, which sets the limit for supernova observations, and once in the Survey Tactician section, which sets the limit for wide survey observations.

The parameter to change the supernova airmass limit occurs around line 130 of the obstac configuration file.

Obstac is limited in the values of the airmass parameter it can take. In particular, it currently must be one between 1.1 and 2.6 inclusive, and bet set in even tenths (that is, 1.4 and 1.5 are both okay, 1.45 is not). (This comes from obstac's use of a lookup table for rise and set times. Other airmasses are not included in the lookup table.)

Note that raising the direct airmass limit may not be sufficient to get it to observe something, even when the airmass is related to why it is excluded. For example, obstac can also exclude observations due to low expected t_eff, which is affected by airmass indirectly.

Changing the tactician

Obstac selects exposure using bits of code (python callable objects) called "tacticians." The Global Tactician section of the configuration file sets the name of the callable object to be used, and the python module in which this object is defined:

[Global Tactician]
module = obstac.tacticians.FirstYearDeepFirstTactician
object = top_tactician

In most tacticians, the object name is top_tactician. The module can be set to any python module in obstac's PYTHONPATH that defines an object that implements the tactician API.

Changing the sky brightness limits

The sky_limit options set the maximum modeled sky brightness at which obstac is willing to schedule an observation. This option appears twice in the configuration file: once in the Supernova Tactician section, which sets the limit for supernova observations, and once in the Survey Tactician section, which sets the limit for wide survey observations. The value of this option should be set to a sequence of five numbers, one for each filter, in the order g, r, i, z, and Y.

Note that obstac selects exposures based on effective exposure time (which maps to S/N on a point source), and the sky brightness is one parameter in this calculation: loostening the sky brightness cutoff directly may not result in obstac being willing to observe a higher sky brightness.

Changing the time for start and end of observing

The sun_min_zd option of the Day section defines the closest the sun can be to zenith for times which obstac will schedule an exposure, effectively setting the times of the beginning and end of observing.

Ignoring observations

One of the easiest ways to change obstac's behavior is to add exposures to the "obs_to_ignore" table.

Here is an example showing how to replace the current contents of this table with all hexes in the minisurvey:

[sispi@observer2 ~]$ psql -U neilsen -h server2 decam_prd
Password for user neilsen: 
psql (9.0.3)
Type "help" for help.

decam_prd=> SET ROLE decam_prd;
SET
decam_prd=> SET SEARCH_PATH TO obstac; 
SET
decam_prd=> DELETE FROM obs_to_ignore;
DELETE 400
decam_prd=>   INSERT INTO obs_to_ignore 
decam_prd->   SELECT hex_id, tiling_id, filter_id 
decam_prd->   FROM hexes
decam_prd->     NATURAL JOIN program_hexes
decam_prd->     NATURAL JOIN program_tilings
decam_prd->     NATURAL JOIN program_filters
decam_prd->     NATURAL JOIN programs
decam_prd->     NATURAL JOIN filters
decam_prd->     WHERE des_program_name IN ('mini1a', 'mini1b', 'mini2');
INSERT 0 1750
decam_prd=> \q

Redoing exposures

There are two mechanisms for telling obstac not to consider a given exposure as completing a needed observation. The first is the "discard" flag in the exposure table, which is set when the observers check the "redo" checkbox in SISPI. The second mechanism uses the "dm_done" and "dm_accept" flags of the exposure table, intended for recording feedback from data management. Tools for setting the later are not yet complete.

To set, for example, exposure 1234567 to be redone using the first method, one can use a query like this:

UPDATE exposure.exposure
SET discard=TRUE
WHERE id=1234567;

You can set multiple exposures, for example 1234567, 1234601, and 1234732, bad, one can use a query like this:

UPDATE exposure.exposure
SET discard=TRUE
WHERE id IN (1234567, 1234567, 1234601);

If you want to use the DM flags rather than the redo flag, you can replace setting the "discard" column with setting the "dm_accept" and "dm_done" columns. For example, the first of the above queries becomes:

UPDATE exposure.exposure
SET dm_done=TRUE, dm_accept=FALSE
WHERE id=1234567;

Using obscirc

obscirc is an obstac command line tool to describing the observing circumstances, either for a time generally, or for a given pointing at a given time.

It uses the same format of configuration file that obstac does, but if it is run on a system where SISPI is not running, some values may need to be different. Because of this, it should currently be regarded as a tool for experts only, unless an expert has provided an appropriate configuration file.

In its most basic usage, it describes the circumstances at a given time, including what obstac would choose at that time (assuming nothing about the queue or database contents changes between now and then):

$ setup obstac
$ obscirc -i 2013-08-30T00:00:01Z -c ~sispi/obstac/DES/obstac-nosispi.conf 
longitude    70.815000    degrees    West
latitude    -30.165278    degrees
elevation    2215.000000    meters
UT    2013-08-30T00:00:01Z
local_time    2013-08-29T19:00:01 CDT
clocktime    1377820801    seconds
MJD    56534.000012    days
LST    267.537338    degrees
sunset    0    2013-08-29T22:22:46Z    2013-08-29T17:22:46 CDT
sunset    -6    2013-08-29T22:50:52Z    2013-08-29T17:50:52 CDT
sunset    -12    2013-08-29T23:18:45Z    2013-08-29T18:18:45 CDT
sunset    -18    2013-08-29T23:46:32Z    2013-08-29T18:46:32 CDT
midnight    2013-08-30T04:43:45Z    2013-08-29T23:43:45 CDT
sunrise    -18    2013-08-30T09:40:57Z    2013-08-30T04:40:57 CDT
sunrise    -12    2013-08-30T10:08:43Z    2013-08-30T05:08:43 CDT
sunrise    -6    2013-08-30T10:36:34Z    2013-08-30T05:36:34 CDT
sunrise    0    2013-08-30T11:04:37Z    2013-08-30T06:04:37 CDT
sun_zd    110.910368    degrees
moon_zd    169.271417    degrees
moon_elongation    72.534895    degrees
moon_brightness    0.046655
tactician    SurveyTactician
program    survey
earliest_start    2013-08-29T23:18:45Z    2013-08-29T18:18:45 CDT
latest_start    2013-08-30T05:55:01Z    2013-08-29T18:18:45 CDT
hex_name    -579-501
hex_id    3004
tiling_id    3
filter    g
ignored    False
attempted    False
done    False
RA    301.198712    degrees
decl    -50.937992    degrees
moon_angle    137.434842    degrees
HA    -33.661375    degrees
after_transit    -2.237964    hours
transit    2013-08-30T02:14:17Z
rise_HA    1.400000    -53.494578    degrees
set_HA    1.400000    53.494578    degrees
rise_LST    1.400000    247.704134    degrees
set_LST    1.400000    354.693291    degrees
after_rise    1.400000    1.318603    hours
before_set    1.400000    5.794532    hours
rise    1.400000    2013-08-29T22:40:54Z    2013-08-29T17:40:54 CDT
set    1.400000    2013-08-30T05:47:41Z    2013-08-30T00:47:41 CDT
zd    32.473178    degrees
airmass    1.184822
skymag    22.082825    mag/asec^2
delta_skymag    -0.146328    mag/asec^2
pred_seeing    1.000000    asec
pred_tau    0.579681

Notice that it begins with a number of parameters describing the night and time as a whole, then what obstac would choose at that time, and then a number of parameters about that pointing and filter, at that time.

Instead of asking it to use the pointing and filter obstac would choose, you can specify a hex, filter, and tiling yourself:

$ obscirc -i 2013-08-30T00:00:01Z -n SN-E2 -f g -t 20 -c ~sispi/obstac/DES/obstac-nosispi.conf

Direct specification of the hex, tiling, and filter yourself is useful for determining why obstac might not have chosen something you thought it should have.

Finally, you can specify a pointing directly with RA and declination, and filter:

$ obscirc -i 2013-08-30T00:00:01Z -r -15 -d -40 -c ~sispi/obstac/DES/obstac-nosispi.conf 

Note that the -r and -d switches specify the R. A. and declination in decimal degrees.

Using obscript

obscript uses obstac to generate an observing script suitable for loading into SISPI. It is designed both to provide a guess about what obstac will choose at some later time, and to serve as a last-ditch fall-back if integration between obstac and SISPI ever fails.

Like obscirc, obscript uses the same format of configuration file that obstac does, but if it is run on a system where SISPI is not running, some values may need to be different. Because of this, it should also be regarded as a tool for experts only, unless an expert has provided an appropriate configuration file.

It can be run thus:

$ obscript --time 2013-08-30T00:00:01Z --duration 1.0  --config ~/obstac/DES/obstac-nosispi.conf --output somescript.json

Obstac configuration for obscirc and obscript

For reference, the difference between the obstac configuration file used by SISPI and the one used by obscirc and obscript should look something like this:

[sispi@observer2 DES]$ diff obstac.conf obstac-nosispi.conf
58,59c58,59
< module = obstac.circumstances.AutoRegSeeingSource 
< object = AutoRegSeeingSource
---
> module = obstac.circumstances.ConstantSeeingSource 
> object = ConstantSeeingSource
106,107c106,107
< module = obstac.tacticians.TopTactician
< object = top_tactician
---
> module = obstac.tacticians.SimTopTactician
> object = sim_top_tactician
168,169c168,169
< module = obstac.ObsQueueOCS
< object = ObsQueueOCS
---
> module = obstac.ObsQueue
> object = ObsQueue
[sispi@observer2 DES]$ 

Whether the change in the seeing source is needed is situation dependent: changing it cause obscirc and obscript to assume a constant seeing of 1 arcsecond. Leaving it as is will cause obstac to try to extrapolate it from data in the database. If the time is a long way in the future, it will do a poor job, and be very slow. If it is in the past, it will do an unrealistically good job.

The wide SQL view in the obstac database

For those comfortable with using SQL and querying the SISPI database (or, preferably when not observing, the mirror of the SISPI database at Fermilab), there is a view designed to simplify examining the wide survey footprint and status. Here is an example of its use:

decam_prd=> SELECT * FROM wide WHERE year=1 LIMIT 10;
 hex_name | tiling_id | filter |        ra        |   decl   | exptime  |     rise_lst      |     set_lst      | year | done | ignore | exposure_id | filter_id | hex_id 
----------+-----------+--------+------------------+----------+----------+-------------------+------------------+------+------+--------+-------------+-----------+--------
 100+16   |         1 | g      |          10.0034 |    1.629 | 00:01:30 | -22.6376311696028 | 42.6444311696028 |    1 | t    | f      |      238946 |         1 |   2521
 100+16   |         1 | i      |          10.0034 |    1.629 | 00:01:30 | -22.6376311696028 | 42.6444311696028 |    1 | f    | f      |             |         3 |   2521
 100+16   |         1 | r      |          10.0034 |    1.629 | 00:01:30 | -22.6376311696028 | 42.6444311696028 |    1 | f    | f      |             |         2 |   2521
 100+16   |         1 | Y      |          10.0034 |    1.629 | 00:00:45 | -22.6376311696028 | 42.6444311696028 |    1 | t    | f      |      235274 |         5 |   2521
 100+16   |         1 | z      |          10.0034 |    1.629 | 00:01:30 | -22.6376311696028 | 42.6444311696028 |    1 | f    | f      |             |         4 |   2521
 100+16   |         2 | g      | 9.23620355604184 | 2.102424 | 00:01:30 | -22.8665834934455 | 41.3389906055292 |    1 | f    | f      |             |         1 |   2521
 100+16   |         2 | i      | 9.23620355604184 | 2.102424 | 00:01:30 | -22.8665834934455 | 41.3389906055292 |    1 | f    | f      |             |         3 |   2521
 100+16   |         2 | r      | 9.23620355604184 | 2.102424 | 00:01:30 | -22.8665834934455 | 41.3389906055292 |    1 | f    | f      |             |         2 |   2521
 100+16   |         2 | Y      | 9.23620355604184 | 2.102424 | 00:00:45 | -22.8665834934455 | 41.3389906055292 |    1 | f    | f      |             |         5 |   2521
 100+16   |         2 | z      | 9.23620355604184 | 2.102424 | 00:01:30 | -22.8665834934455 | 41.3389906055292 |    1 | f    | f      |             |         4 |   2521

The ra is in decimal degrees, running from -180 to 180 degrees so that it varies continuously over the survey footprint.

rise_lst and set_lst are the local sidereal times at which the pointing rises and sets at 1.4 airmasses, in decimal degrees. As with ra, the LST values run from -180 to 180 degrees.

The year designates the year of the survey in which the observation is scheduled.

ignore is a boolean designating whether the observation is in the obstac ignore table.

done marks whether obstac regards the observation as having been completed. Note that this column is redundant with the exposure_id column; it is present only to provide a shorthand for exposure_id IS NOT NULL .

exposure_id shows the exposure id (id in the exposure table), if the observation has been completed successfully. If it has not been completed, the value is NULL.

The sn_status SQL view in the obstac database

For those comfortable with using SQL and querying the SISPI database (or, preferably when not observing, the mirror of the SISPI database at Fermilab), there is a view designed to simplify examining the SN exposure specifications and status. Here is an example of its use:

decam_prd=> SELECT * FROM sn_status ORDER BY set_lst LIMIT 10;;
 hex_name | seqtype | tiling_id | filter |        ra        |    decl    | exptime  |     rise_lst      |     set_lst      | exposure_id |             date              | filter_id | hex_id 
----------+---------+-----------+--------+------------------+------------+----------+-------------------+------------------+-------------+-------------------------------+-----------+--------
 SN-E1    | short   |        22 | z      | 7.87375040415142 | -43.010905 | 00:03:20 | -51.4128272614696 | 67.1603280697724 |      243834 | 2013-10-14 00:12:33.444043+00 |         4 |    345
 SN-E1    | short   |        21 | z      |           7.8744 |   -43.0096 | 00:03:20 | -51.4120186578342 | 67.1608186578342 |      243833 | 2013-10-14 00:08:45.63045+00  |         4 |    345
 SN-E1    | short   |        20 | r      |           7.8744 |   -43.0096 | 00:02:30 | -51.4120186578342 | 67.1608186578342 |      243831 | 2013-10-14 00:01:56.17261+00  |         2 |    345
 SN-E1    | short   |        20 | i      |           7.8744 |   -43.0096 | 00:03:20 | -51.4120186578342 | 67.1608186578342 |      243832 | 2013-10-14 00:04:55.816949+00 |         3 |    345
 SN-E1    | short   |        20 | g      |           7.8744 |   -43.0096 | 00:02:55 | -51.4120186578342 | 67.1608186578342 |      243830 | 2013-10-13 23:58:32.186127+00 |         1 |    345
 SN-E2    | short   |        22 | z      |  9.4993396800291 | -43.999305 | 00:03:20 | -49.8989623111116 | 68.8976416711698 |      243840 | 2013-10-14 00:32:13.877711+00 |         4 |    346
 SN-E2    | short   |        20 | r      |              9.5 |    -43.998 | 00:02:30 | -49.8981661054508 | 68.8981661054508 |      243837 | 2013-10-14 00:21:29.185072+00 |         2 |    346
 SN-E2    | short   |        21 | z      |              9.5 |    -43.998 | 00:03:20 | -49.8981661054508 | 68.8981661054508 |      243839 | 2013-10-14 00:28:22.802916+00 |         4 |    346
 SN-E2    | short   |        20 | g      |              9.5 |    -43.998 | 00:02:55 | -49.8981661054508 | 68.8981661054508 |      243836 | 2013-10-14 00:18:03.824508+00 |         1 |    346
 SN-E2    | short   |        20 | i      |              9.5 |    -43.998 | 00:03:20 | -49.8981661054508 | 68.8981661054508 |      243838 | 2013-10-14 00:24:27.309714+00 |         3 |    346

Most columns have the same meaning as they do in the wide view (described above).

The exposure_id and date columns contain the exposure_id and date of the most recent exposure not declared bad.

Turning on and off second half (am) observing

When DES is not observing a full night, but just the first (am) half, obstac needs to know so that it doesn't try to schedule a SN sequence that goes past the half-way point (local solar midnight). Obstac gets this information from the SISPI database. When the schedule in the database is set such that DES does not have the second (am) half, however, obstac will refuse to schedule exposures then, which might be desirable (for example if we get unexpected engineering time). The following is an example of using blunt hammer to turn off all second half (am) observing:

Last login: Wed Dec 30 13:17:14 2015 from vpnls15.ctio.noao.edu
[sispi@observer2 ~]$ setup Site
[sispi@observer2 ~]$ setup postgresql
[sispi@observer2 ~]$ setenv PGHOST $SISPI_DB_HOST
[sispi@observer2 ~]$ setenv PGPORT $SISPI_DB_PORT
[sispi@observer2 ~]$ setenv PGDATABASE $SISPI_DB_NAME
[sispi@observer2 ~]$ psql -U neilsen
Password for user neilsen: 
psql (9.0.13)
Type "help" for help.

decam_prd=> SET ROLE decam_prd;
SET
decam_prd=> SET SEARCH_PATH TO obstac;
SET
decam_prd=> BEGIN;
BEGIN
decam_prd=> UPDATE observing_periods SET am = FALSE ;
UPDATE 60
decam_prd=> COMMIT;
COMMIT
decam_prd=> ANALYZE observing_periods;
ANALYZE
decam_prd=> \q
[sispi@observer2 ~]$ 

Turning on all second half (am) observing is similar, except the update should set am to TRUE.

Disaster work-arounds

Obstac disaster workarounds can be found here.

Developers documentation

Additional documentation aimed and obstac developers can be found in the doc directory of the obstac subversion product.