Introduction to the nightsum » History » Version 1

Version 1/14 - Next ยป - Current version
Marc testing Mengel, 08/26/2013 11:08 AM

Introduction to the nightsum

The nightsum reports are automatically generated summaries taken from the electronic logbook and other sources at the end of every night. It is meant to provide an overview of the DES activities at the mountain top. Every morning, a report will be posted to the "nightsum web page": by the observers. The reports are archived by MJD.

We encourage people in the collaboration to include the nightsum in their morning reading routine (username is "DES", password is the same as for docdb). In the nightsum you will find information about the observing team, the night's plan, accomplishments, and problems. There is also a list of all the exposures taken that night, and links to relevant logbook entries containing in-depth information. Here is one "example":

Electronic Logbook and nightsum

For the automatic nighsum tool to work properly, the logbook entries have to be a little bit organized. One observer has to take the role of the 'logger' and make sure that the entries that go into the nightsum are meaningful. There are 3 special logbook Categories (they are called Synopsis_*) and a special tag (summary) that the nightsum listen to. Other observers and users of the logbook should not make entries in those categories without informing the 'logger'. Here is a description of the categories/tags:

  • Synopsis_Night: Only one entry of this category can be included per night. The entry should include information about the observing team, the plan for the night and the accomplishments (here is an "example": Here is a template that observers have found useful. Login as DECamObserver on observer2 and see /usr/remote/user/DECamObserver/annis/log_template.txt.
  • Synopsis_Engineering: All entries in this category will be included. They should inform briefly about any planned engineering/debugging activity, saying what was done, and what results were obtained. (here is an "example":
  • Synopsis_Problem: All entries in this category will be included. This should inform briefly about a problem that disturbed the night's progress in either engineering or science activity (here is an "example":
  • summary: This is a tag to be attached to any other entry that doesn't fall in any of the above. Entries flagged with the summary tag will show up in a section called "Other Issues".

Detailed descriptions of problems and procedures taken during the night are better handled via links, ie, you make a short entry in the relevant Synopsis category with a link to a longer detailed entry in another category.

The nightsum can handle html/wiki-like markups and attached images the same way as the logbook. You must click the "Textile formatted" checkbox on the new entries window for the logbook to recognize the wiki formatting. You should be able to preview the formatted entry using new entries window button.

Instructions on how to create the nightsum (for observers)

This currently requires having a FNAL kerberos account and permissions to log into decam03 as codemanager. This process is not automated yet, but the steps are simple:

  • log into observer2 as user sispi
  • set up the environment:
    $ setup nightsum
  • run the makesum script:
    $ cd ~sispi/nightsum-reports
    $ makesum
  • The step above will create a dir named nightsum-<MJD>. Check the results by pointong your browser to
  • Go to the logbook and make changes as needed. Repeat the makesum step until you are happy with the results.
  • Then, publish the results so it shows up on the web server on

    $ cd ~sispi/nightsum-reports
    $ pubsum

Monitoring a draft of the night summary

A web server can be started that shows a current draft of the night summary, as follows:

  • log into observer2 as user DECamObserver
  • set up the environment:
    $ setup nightsum
  • start the server:
    $ NightSummarizer ~DECamObserver/nightsum-config

The draft will appear in [[]]

If you want to change the port number or host on which it is run, you can edit ~DECamObserver/nightsum-config/NightSummarizerServer.config
The host name in this configuration file must match the name of the host on which the server is being started.

Advice on level of detail

The best way to understand what content should be included in the summary is to consider the purpose and audience of the summary as a whole, and of each section.

The following are some primary uses of the log:
  • Any collaborator might follow the summary to see how DES data collection is going.
  • DES observers might monitor the summary for observing procedure changes, problems to look out for, and other things they might need to know to prepare them for their next observing run.
  • Data processing staff will read the summary to see if there is anything they need to be aware of in processing that night's data. For example, if there are images with bad headers that need to be fixed by hand before processing, there should be some indication of it in the summary.
  • DES developers, engineers, and managment will read the summary to see what tests and other requested non-science activities were done.
  • If anomalies in the data are discovered by anyone (scientists, data processing staff, et al.), the night summary should tell them if the observers knew anything that might be a likely cause.

In each case, the summary is not expected to contain all information needed to explore any given issue. It should, however, contain enough information to let anyone reading the summary know what issues were relevant to that night, and links and other references to any additional resources. A typical "test" entry might include two sentences describing the test, a link to a document in docdb that describes the test, links to any electronic logbook entries related with more details, and a list of related exposure IDs.

Remember, the summary is intended to be quickly scanned by many people who might be checking for specific things. Too much detail is almost as bad as too little. It should, however, contain enough information that "absence of evidence" can be taken as "evidence of absence." If someone looks in the summary and doesn't see any mention of a given test, he or she should be confident that the test was not performed. If someone looks in the summary and doesn't see any problems related to a given system, he or she should be confident that the observers didn't notice any problems with that system.

Questions asked (if not frequently, at least once)

What is the difference between category and tags?

Each entry has at most one (and perhaps exactly one) category, but any entry can have any number of tags, and it doesn't mater what category they are in. So, from nightsum's point of view, entries in a nightsum-related category were written with the specific purpose of going into a specific section of a night summary. The summary tag is a way the observer can label any other entry, written for whatever purpose, "hay, this is something I think everyone should see- put this in the summary!" I expect it will also be used a bit like an "other" category from the night summary point of view; it's a way to add stuff that doesn't fit anywhere else.

What is the difference between problems and engineering?

An engineering task or test is an activity that is not routine science observing, but which is planned. It probably gets done because someone wanted it done.

A problem is an unexpected occurrence that prevents you from following your plan.

So, testing a planned new obstac tactician would be engineering. On the night when it is tested, finding and dealing with bugs in it is engineering. If some other system breaks while you are doing the tests, that's a problem.

Three days later, when you were planning to science, if bugs in that code prevent you from observing, that's a problem.

Again, the easiest way to clarify things is to take the readers perspective. If I've requested that you do something unusual, I will keep an eye on the engineering section to see if you did it, and what the results were. If I'm just following the logs to see if you ran into problems with my software that I expect to be running smoothly, I will look for it in the problems section.

If this is too confusing, perhaps we should merge the sections.

Known issues

Formatting of plain text entires comes out corrupted.

The XML interface the nightsum tool uses doesn't provide the flag whether the entry uses the "textile" wiki-like markup, so the summary tool has to guess. The latest version of the electronic logbook includes a fix for this, but this version is not yet installed on the mountain. When it is, modifying nightsum to honor the flag should be straightforward.

Nightsum sometimes fails when comments have special characters in them (particularly "#" and "&")

This is another electronic logbook bug. I believe this is also fixed in the latest version of the logbook (not yet installed on at the observatory), but this needs to be verified.

Observers currently must run the tool from the "sispi" account

Publishing the summary requires access to files only sispi should be able to read. This functionality can eventually be made part of a web interface to the draft, which can be run by the sispi account.

Documentation and procedures need expansion

Planned feature enhancements

  • The sectioning and formatting should better worked out.
  • The TASCA all-sky movie should be included in the summary.
  • The thumbnail images should be copied into the nightsum directory and thence to the webserver, so they can be seen form outside the observatory.
  • Links to data files (of conveniently machine-readable format, such a tsv, json, or xml) should be included in the summary.
  • The "live" draft of the nightsum should be started by the Architect as part of SISPI startup, and perhaps some screen space found for it.
  • The "live" draft of the nightsum should have a "publish" button, which copies the summary to the appropriate web server (at Fermilab or, eventually, NCSA).
  • The "comments" from the electronic logbook (or at least links to the comments) should be part of the summary.
  • Automatically generated plots of exposure pointings on the survey footprint should be part of the summary.
  • Automatically generated seeing statistics should be part of the summary.
  • Automatically generated cloud statistics should be part of the summary.
  • Automatically generated open-shutter-time, slew time, night duration, and other efficiency measures should be part of the summary.
  • Automatically generated links (and/or summaries) connecting exposures to relavant weather information (DIMM, RASICAM, wind, etc.) should be added to the summary.
  • The summary should be better integrated with Data Mangement's Quality Assessment tools.