Update: July 2013¶
On request from the batch group, we've changed to hourly records from the TimeGoblin, copied to the minos-nearline, and uploaded to the database.
The TimeGoblin closes out files on or near the end of the hour. At XX:05, the files are copied, and at XX:10 the files are uploaded to the DB.
Spills are tagged with an incoming MIBS $74 signal from beams division, which marks the start of the kicker fire. The Master Clock Controller at the Near detector receives a copy of this signal, adjusts it for the known MI latencies, and may trigger the detector. A copy of this pulse is also sent to the Near Detector GPS unit. Note that MIBS $74 signals are frequently sent even when there is no MI beam, so presence of a "spill" signal does not indicate any real beam; the beam monitoring database must be correlated with this time to tell if there were actually any protons on target. Also note that this system does NOT require the ND to be running in order to tag spills, but the timing rack and DAQ computers must be powered and running.
Within a second or so, the timestamp is read out of the GPS by the Near Detector timing PC (daqtpc-nd.fnal.gov), which collates the timestamps into log files on the DAQ cluster, saved in $DAQ_LOG_DIR/spilllogs/spilltime.<date>.<sequence>.dat.
The timestamps are in <seconds> <nanoseconds> format, using the standard UNIX convention of seconds-since-the-epoch (i.e. 1970-01-01 12:00:00). This avoid any need for tracking time zones or the like. However, note that the time stamps are NOT strictly UTC. The GPS unit records time in "GPS" time, which is an integer number of seconds away from UTC (currently UTC + 16 seconds). This changes every time there is a leap second inserted in the UTC timescale. Leap second corrections are done by the TimeGoblin a the ND and FD, so the files are corrected for this.
When a leap second occurs, a timing expert (Nathaniel, usually) calls up the control room near midnight, and has them stop runs at both detectors. Both detector offsets are adjusted, and then runs are restarted. This ensures that our ND and FD timestamps are always consistent.
Files are closed out and new ones started every 6 hours. (Configured via tpc.config)
From DAQ to DB¶
The spilllog files, when completed, are copied via a DAQ logger process to bluearc. (NATHANIEL: I've never known what process was responsible for this step; ask Donatella or Art.) They are copied to /nusoft/app/web/htdoc/minos/controlroom/daqlogs/NEAR/timing/spilllogs.
These files are processed on minos-nearline.fnal.gov. The generic 'minos' user has a crontab set up to check these files at 00:10, 06:10, 12:10, and 18:10 local time. This matches the copy times from the DAQ copy system and the requested Batch schedule. Cron starts a job as minos user on these machines from ~minos/SpillTimeLooter/loot, which looks for unprocessed files in the log directory, and then creates the relevant SPILLTIMEND database sequences in the DB. This process is fairly fast, so spilltimes are available within minutes of this job.
This system is designed not only to robustly keep a complete record of spills, but also to track times when there were definitely no spills. It is common that the SPILLTIMENDVLD table will be updated with zero rows in the corresponding SPILLTIMEND table, which indicates that no beam spills were seen during that period: useful for atmospheric studies, for example.
Written by Nathaniel Tagg, July 2013. Current timing experts are Nathaniel and Giles Barr.