Obstac for observers¶
DES observers should read from here through the "Surprises and gotchas" section
What obstac does¶
During observing, obstac has two states: autoobs=on and autoobs=off. When autoobs is off, obstac does nothing. When autoobs is on, it monitors the length of the queue and, whenever the duration of the queue falls below the configured targed queue duration, it adds observations to the queue until the duration exceeds target queue duration, and then stops until the next time the queue falls below it.
An indicator above the observing queue shows whether autoobs is enabled:
Currently, the configured queue duration is 900 seconds. This may be modified be setting the min_duration keyword in the queue section of the obstac configuration file, and restarting the obstac SISPI roles.
Turning obstac on and off¶
Autoobs can be turned of and off using the "Enable Auto" (or "Disable Auto") button in the exposure control tab of the observer console:
Redoing observations¶
If observers note that an exposure is unambiguously bad, and the observation needs to be repeated, observers can mark the exposure to be redone. Find the offending exposure in the exposure table page of the SISPI interface, and select it to bring up details on that exposure. One of the details is a "redo" checkbox (row 4, column 5 in the gray area); checking this checkbox will mark it as needing to be redone.
For older exposures, enter the exposure id in the "Exposure ID" field in the upper right of the exposure details section (lower panel) of the exposure table page, and hit the "Select" button to its immediate right to bring up the exposure details with the redo checkbox.
Recent tests indicate that only one expid can be processed with this method. To process a second exposure the chrome browser "reload/refresh" button must be pressed, then a new exposure id loaded and redo selected. This may be true only for exposures from previous nights that must be loaded manually.
To confirm that your redo request was successful, go to the DB query window at [[http://system1.ctio.noao.edu:8080/TV/app/Q/execute]], select namespace exposure and use a query like this:
SELECT id, date, discard FROM exposure WHERE id IN(258466,258467, 258468, 258469, 258472, 258473, 258474, 258475, 258478, 258479, 258480, 258485, 258486, 258487, 258488, 258491, 258492, 258493, 258494) ORDER by id
Running the above query as an example should yield discard==true for each each exposure in the list.
Exposures can also be declared "redo" in bulk using the lower-level SQL interface to the database; see the instructions in "Obstac for SISPI experts" for more.
Do this only for object exposures
Do this only for exposures that are bad due to technical reasons, e.g. forgot to turn on Vsub. This should not be used for exposures that are bad due to clouds or seeing as setting this flag will instruct obstac to reschedule this exposure as soon as possible. Usually, clouds won't dissipate that quickly and you would only be wasting time by rescheduling the exposure right away.
The Redo mechanism is only for special cases, for most exposures the decision on image quality will be made by DES DM and it's the run manager's responsibility to handle this.
Surprises and gotchas¶
Everything here is work-arounds for bugs. The bugs will eventually be fixed.
Filling to completion¶
A fix for this bug is awaiting testing.
Once obstac starts filling the queue, it will not stop until the queue duration has reached its target duration (900 seconds), even if autoobs is turned off during the fill. It will stop after it reaches the target duration; the only thing to do is wait.
Obstac for SISPI experts¶
Configuring obstac¶
Finding the obstac configuration file¶
The obstac configuration file is set by the obstac_config_file option in the Architect "ini" file used to start the SISPI instance. The current value (as of 2013-09-05) is:
/usr/remote/user/sispi/obstac/DES/obstac.conf
The location of this file should not be changed without a good reason.
Switching to a new configuration¶
When the configuration file is changed, the obstac-related roles in SISPI must be restarted in order to pick up the changes. In particular, when AUTOOBS starts, the correctly configured version of OBSTACSRVR must already be running. So, the recommended procedure is:
- Stop the
AUTOOBSrole, - restart the
OBSTACSRVRrole, and only then - start the
AUTOOBSrole.
Changing airmass limits¶
The airmass_limit options set the maximum airmass at which obstac is willing to schedule an observation. This option appears twice in the configuration file: once in the Supernova Tactician section, which sets the limit for supernova observations, and once in the Survey Tactician section, which sets the limit for wide survey observations.
The parameter to change the supernova airmass limit occurs around line 130 of the obstac configuration file.
Obstac is limited in the values of the airmass parameter it can take. In particular, it currently must be one between 1.1 and 2.6 inclusive, and bet set in even tenths (that is, 1.4 and 1.5 are both okay, 1.45 is not). (This comes from obstac's use of a lookup table for rise and set times. Other airmasses are not included in the lookup table.)
Note that raising the direct airmass limit may not be sufficient to get it to observe something, even when the airmass is related to why it is excluded. For example, obstac can also exclude observations due to low expected t_eff, which is affected by airmass indirectly.
Changing the tactician¶
Obstac selects exposure using bits of code (python callable objects) called "tacticians." The Global Tactician section of the configuration file sets the name of the callable object to be used, and the python module in which this object is defined:
[Global Tactician] module = obstac.tacticians.FirstYearDeepFirstTactician object = top_tactician
In most tacticians, the object name is top_tactician. The module can be set to any python module in obstac's PYTHONPATH that defines an object that implements the tactician API.
Changing the sky brightness limits¶
The sky_limit options set the maximum modeled sky brightness at which obstac is willing to schedule an observation. This option appears twice in the configuration file: once in the Supernova Tactician section, which sets the limit for supernova observations, and once in the Survey Tactician section, which sets the limit for wide survey observations. The value of this option should be set to a sequence of five numbers, one for each filter, in the order g, r, i, z, and Y.
Note that obstac selects exposures based on effective exposure time (which maps to S/N on a point source), and the sky brightness is one parameter in this calculation: loostening the sky brightness cutoff directly may not result in obstac being willing to observe a higher sky brightness.
Changing the time for start and end of observing¶
The sun_min_zd option of the Day section defines the closest the sun can be to zenith for times which obstac will schedule an exposure, effectively setting the times of the beginning and end of observing.
Ignoring observations¶
One of the easiest ways to change obstac's behavior is to add exposures to the "obs_to_ignore" table.
Here is an example showing how to replace the current contents of this table with all hexes in the minisurvey:
[sispi@observer2 ~]$ psql -U neilsen -h server2 decam_prd
Password for user neilsen:
psql (9.0.3)
Type "help" for help.
decam_prd=> SET ROLE decam_prd;
SET
decam_prd=> SET SEARCH_PATH TO obstac;
SET
decam_prd=> DELETE FROM obs_to_ignore;
DELETE 400
decam_prd=> INSERT INTO obs_to_ignore
decam_prd-> SELECT hex_id, tiling_id, filter_id
decam_prd-> FROM hexes
decam_prd-> NATURAL JOIN program_hexes
decam_prd-> NATURAL JOIN program_tilings
decam_prd-> NATURAL JOIN program_filters
decam_prd-> NATURAL JOIN programs
decam_prd-> NATURAL JOIN filters
decam_prd-> WHERE des_program_name IN ('mini1a', 'mini1b', 'mini2');
INSERT 0 1750
decam_prd=> \q
Redoing exposures¶
There are two mechanisms for telling obstac not to consider a given exposure as completing a needed observation. The first is the "discard" flag in the exposure table, which is set when the observers check the "redo" checkbox in SISPI. The second mechanism uses the "dm_done" and "dm_accept" flags of the exposure table, intended for recording feedback from data management. Tools for setting the later are not yet complete.
To set, for example, exposure 1234567 to be redone using the first method, one can use a query like this:
UPDATE exposure.exposure SET discard=TRUE WHERE id=1234567;
You can set multiple exposures, for example 1234567, 1234601, and 1234732, bad, one can use a query like this:
UPDATE exposure.exposure SET discard=TRUE WHERE id IN (1234567, 1234567, 1234601);
If you want to use the DM flags rather than the redo flag, you can replace setting the "discard" column with setting the "dm_accept" and "dm_done" columns. For example, the first of the above queries becomes:
UPDATE exposure.exposure SET dm_done=TRUE, dm_accept=FALSE WHERE id=1234567;
Using obscirc¶
obscirc is an obstac command line tool to describing the observing circumstances, either for a time generally, or for a given pointing at a given time.
It uses the same format of configuration file that obstac does, but if it is run on a system where SISPI is not running, some values may need to be different. Because of this, it should currently be regarded as a tool for experts only, unless an expert has provided an appropriate configuration file.
In its most basic usage, it describes the circumstances at a given time, including what obstac would choose at that time (assuming nothing about the queue or database contents changes between now and then):
$ setup obstac $ obscirc -i 2013-08-30T00:00:01Z -c ~sispi/obstac/DES/obstac-nosispi.conf longitude 70.815000 degrees West latitude -30.165278 degrees elevation 2215.000000 meters UT 2013-08-30T00:00:01Z local_time 2013-08-29T19:00:01 CDT clocktime 1377820801 seconds MJD 56534.000012 days LST 267.537338 degrees sunset 0 2013-08-29T22:22:46Z 2013-08-29T17:22:46 CDT sunset -6 2013-08-29T22:50:52Z 2013-08-29T17:50:52 CDT sunset -12 2013-08-29T23:18:45Z 2013-08-29T18:18:45 CDT sunset -18 2013-08-29T23:46:32Z 2013-08-29T18:46:32 CDT midnight 2013-08-30T04:43:45Z 2013-08-29T23:43:45 CDT sunrise -18 2013-08-30T09:40:57Z 2013-08-30T04:40:57 CDT sunrise -12 2013-08-30T10:08:43Z 2013-08-30T05:08:43 CDT sunrise -6 2013-08-30T10:36:34Z 2013-08-30T05:36:34 CDT sunrise 0 2013-08-30T11:04:37Z 2013-08-30T06:04:37 CDT sun_zd 110.910368 degrees moon_zd 169.271417 degrees moon_elongation 72.534895 degrees moon_brightness 0.046655 tactician SurveyTactician program survey earliest_start 2013-08-29T23:18:45Z 2013-08-29T18:18:45 CDT latest_start 2013-08-30T05:55:01Z 2013-08-29T18:18:45 CDT hex_name -579-501 hex_id 3004 tiling_id 3 filter g ignored False attempted False done False RA 301.198712 degrees decl -50.937992 degrees moon_angle 137.434842 degrees HA -33.661375 degrees after_transit -2.237964 hours transit 2013-08-30T02:14:17Z rise_HA 1.400000 -53.494578 degrees set_HA 1.400000 53.494578 degrees rise_LST 1.400000 247.704134 degrees set_LST 1.400000 354.693291 degrees after_rise 1.400000 1.318603 hours before_set 1.400000 5.794532 hours rise 1.400000 2013-08-29T22:40:54Z 2013-08-29T17:40:54 CDT set 1.400000 2013-08-30T05:47:41Z 2013-08-30T00:47:41 CDT zd 32.473178 degrees airmass 1.184822 skymag 22.082825 mag/asec^2 delta_skymag -0.146328 mag/asec^2 pred_seeing 1.000000 asec pred_tau 0.579681
Notice that it begins with a number of parameters describing the night and time as a whole, then what obstac would choose at that time, and then a number of parameters about that pointing and filter, at that time.
Instead of asking it to use the pointing and filter obstac would choose, you can specify a hex, filter, and tiling yourself:
$ obscirc -i 2013-08-30T00:00:01Z -n SN-E2 -f g -t 20 -c ~sispi/obstac/DES/obstac-nosispi.conf
Direct specification of the hex, tiling, and filter yourself is useful for determining why obstac might not have chosen something you thought it should have.
Finally, you can specify a pointing directly with RA and declination, and filter:
$ obscirc -i 2013-08-30T00:00:01Z -r -15 -d -40 -c ~sispi/obstac/DES/obstac-nosispi.conf
Note that the -r and -d switches specify the R. A. and declination in decimal degrees.
Using obscript¶
obscript uses obstac to generate an observing script suitable for loading into SISPI. It is designed both to provide a guess about what obstac will choose at some later time, and to serve as a last-ditch fall-back if integration between obstac and SISPI ever fails.
Like obscirc, obscript uses the same format of configuration file that obstac does, but if it is run on a system where SISPI is not running, some values may need to be different. Because of this, it should also be regarded as a tool for experts only, unless an expert has provided an appropriate configuration file.
It can be run thus:
$ obscript --time 2013-08-30T00:00:01Z --duration 1.0 --config ~/obstac/DES/obstac-nosispi.conf --output somescript.json
Obstac configuration for obscirc and obscript¶
For reference, the difference between the obstac configuration file used by SISPI and the one used by obscirc and obscript should look something like this:
[sispi@observer2 DES]$ diff obstac.conf obstac-nosispi.conf 58,59c58,59 < module = obstac.circumstances.AutoRegSeeingSource < object = AutoRegSeeingSource --- > module = obstac.circumstances.ConstantSeeingSource > object = ConstantSeeingSource 106,107c106,107 < module = obstac.tacticians.TopTactician < object = top_tactician --- > module = obstac.tacticians.SimTopTactician > object = sim_top_tactician 168,169c168,169 < module = obstac.ObsQueueOCS < object = ObsQueueOCS --- > module = obstac.ObsQueue > object = ObsQueue [sispi@observer2 DES]$
Whether the change in the seeing source is needed is situation dependent: changing it cause obscirc and obscript to assume a constant seeing of 1 arcsecond. Leaving it as is will cause obstac to try to extrapolate it from data in the database. If the time is a long way in the future, it will do a poor job, and be very slow. If it is in the past, it will do an unrealistically good job.
The wide SQL view in the obstac database¶
For those comfortable with using SQL and querying the SISPI database (or, preferably when not observing, the mirror of the SISPI database at Fermilab), there is a view designed to simplify examining the wide survey footprint and status. Here is an example of its use:
decam_prd=> SELECT * FROM wide WHERE year=1 LIMIT 10; hex_name | tiling_id | filter | ra | decl | exptime | rise_lst | set_lst | year | done | ignore | exposure_id | filter_id | hex_id ----------+-----------+--------+------------------+----------+----------+-------------------+------------------+------+------+--------+-------------+-----------+-------- 100+16 | 1 | g | 10.0034 | 1.629 | 00:01:30 | -22.6376311696028 | 42.6444311696028 | 1 | t | f | 238946 | 1 | 2521 100+16 | 1 | i | 10.0034 | 1.629 | 00:01:30 | -22.6376311696028 | 42.6444311696028 | 1 | f | f | | 3 | 2521 100+16 | 1 | r | 10.0034 | 1.629 | 00:01:30 | -22.6376311696028 | 42.6444311696028 | 1 | f | f | | 2 | 2521 100+16 | 1 | Y | 10.0034 | 1.629 | 00:00:45 | -22.6376311696028 | 42.6444311696028 | 1 | t | f | 235274 | 5 | 2521 100+16 | 1 | z | 10.0034 | 1.629 | 00:01:30 | -22.6376311696028 | 42.6444311696028 | 1 | f | f | | 4 | 2521 100+16 | 2 | g | 9.23620355604184 | 2.102424 | 00:01:30 | -22.8665834934455 | 41.3389906055292 | 1 | f | f | | 1 | 2521 100+16 | 2 | i | 9.23620355604184 | 2.102424 | 00:01:30 | -22.8665834934455 | 41.3389906055292 | 1 | f | f | | 3 | 2521 100+16 | 2 | r | 9.23620355604184 | 2.102424 | 00:01:30 | -22.8665834934455 | 41.3389906055292 | 1 | f | f | | 2 | 2521 100+16 | 2 | Y | 9.23620355604184 | 2.102424 | 00:00:45 | -22.8665834934455 | 41.3389906055292 | 1 | f | f | | 5 | 2521 100+16 | 2 | z | 9.23620355604184 | 2.102424 | 00:01:30 | -22.8665834934455 | 41.3389906055292 | 1 | f | f | | 4 | 2521
The ra is in decimal degrees, running from -180 to 180 degrees so that it varies continuously over the survey footprint.
rise_lst and set_lst are the local sidereal times at which the pointing rises and sets at 1.4 airmasses, in decimal degrees. As with ra, the LST values run from -180 to 180 degrees.
The year designates the year of the survey in which the observation is scheduled.
ignore is a boolean designating whether the observation is in the obstac ignore table.
done marks whether obstac regards the observation as having been completed. Note that this column is redundant with the exposure_id column; it is present only to provide a shorthand for exposure_id IS NOT NULL .
exposure_id shows the exposure id (id in the exposure table), if the observation has been completed successfully. If it has not been completed, the value is NULL.
The sn_status SQL view in the obstac database¶
For those comfortable with using SQL and querying the SISPI database (or, preferably when not observing, the mirror of the SISPI database at Fermilab), there is a view designed to simplify examining the SN exposure specifications and status. Here is an example of its use:
decam_prd=> SELECT * FROM sn_status ORDER BY set_lst LIMIT 10;; hex_name | seqtype | tiling_id | filter | ra | decl | exptime | rise_lst | set_lst | exposure_id | date | filter_id | hex_id ----------+---------+-----------+--------+------------------+------------+----------+-------------------+------------------+-------------+-------------------------------+-----------+-------- SN-E1 | short | 22 | z | 7.87375040415142 | -43.010905 | 00:03:20 | -51.4128272614696 | 67.1603280697724 | 243834 | 2013-10-14 00:12:33.444043+00 | 4 | 345 SN-E1 | short | 21 | z | 7.8744 | -43.0096 | 00:03:20 | -51.4120186578342 | 67.1608186578342 | 243833 | 2013-10-14 00:08:45.63045+00 | 4 | 345 SN-E1 | short | 20 | r | 7.8744 | -43.0096 | 00:02:30 | -51.4120186578342 | 67.1608186578342 | 243831 | 2013-10-14 00:01:56.17261+00 | 2 | 345 SN-E1 | short | 20 | i | 7.8744 | -43.0096 | 00:03:20 | -51.4120186578342 | 67.1608186578342 | 243832 | 2013-10-14 00:04:55.816949+00 | 3 | 345 SN-E1 | short | 20 | g | 7.8744 | -43.0096 | 00:02:55 | -51.4120186578342 | 67.1608186578342 | 243830 | 2013-10-13 23:58:32.186127+00 | 1 | 345 SN-E2 | short | 22 | z | 9.4993396800291 | -43.999305 | 00:03:20 | -49.8989623111116 | 68.8976416711698 | 243840 | 2013-10-14 00:32:13.877711+00 | 4 | 346 SN-E2 | short | 20 | r | 9.5 | -43.998 | 00:02:30 | -49.8981661054508 | 68.8981661054508 | 243837 | 2013-10-14 00:21:29.185072+00 | 2 | 346 SN-E2 | short | 21 | z | 9.5 | -43.998 | 00:03:20 | -49.8981661054508 | 68.8981661054508 | 243839 | 2013-10-14 00:28:22.802916+00 | 4 | 346 SN-E2 | short | 20 | g | 9.5 | -43.998 | 00:02:55 | -49.8981661054508 | 68.8981661054508 | 243836 | 2013-10-14 00:18:03.824508+00 | 1 | 346 SN-E2 | short | 20 | i | 9.5 | -43.998 | 00:03:20 | -49.8981661054508 | 68.8981661054508 | 243838 | 2013-10-14 00:24:27.309714+00 | 3 | 346
Most columns have the same meaning as they do in the wide view (described above).
The exposure_id and date columns contain the exposure_id and date of the most recent exposure not declared bad.
Turning on and off second half (am) observing¶
When DES is not observing a full night, but just the first (am) half, obstac needs to know so that it doesn't try to schedule a SN sequence that goes past the half-way point (local solar midnight). Obstac gets this information from the SISPI database. When the schedule in the database is set such that DES does not have the second (am) half, however, obstac will refuse to schedule exposures then, which might be desirable (for example if we get unexpected engineering time). The following is an example of using blunt hammer to turn off all second half (am) observing:
Last login: Wed Dec 30 13:17:14 2015 from vpnls15.ctio.noao.edu [sispi@observer2 ~]$ setup Site [sispi@observer2 ~]$ setup postgresql [sispi@observer2 ~]$ setenv PGHOST $SISPI_DB_HOST [sispi@observer2 ~]$ setenv PGPORT $SISPI_DB_PORT [sispi@observer2 ~]$ setenv PGDATABASE $SISPI_DB_NAME [sispi@observer2 ~]$ psql -U neilsen Password for user neilsen: psql (9.0.13) Type "help" for help. decam_prd=> SET ROLE decam_prd; SET decam_prd=> SET SEARCH_PATH TO obstac; SET decam_prd=> BEGIN; BEGIN decam_prd=> UPDATE observing_periods SET am = FALSE ; UPDATE 60 decam_prd=> COMMIT; COMMIT decam_prd=> ANALYZE observing_periods; ANALYZE decam_prd=> \q [sispi@observer2 ~]$
Turning on all second half (am) observing is similar, except the update should set am to TRUE.
Disaster work-arounds¶
Obstac disaster workarounds can be found here.
Developers documentation¶
Additional documentation aimed and obstac developers can be found in the doc directory of the obstac subversion product.