Project

General

Profile

Supernova protocol for the MicroBooNE SN stream

MicroBooNE relies on an SNEWS alert to initiate a manual transfer of the MicroBooNE SN stream runs surrounding the supernova core-collapse event. For this, we have several experts subscribed to the SNEWS mailing list

https://snews.bnl.gov/mailinglists.html

However, to maximize redundancy, I suggest Run Coordinators subscribe too (and you will be among the first ones to know that a supernova happened!). In case of an alert, please contact Jose Crespo (https://microboone-exp.fnal.gov/at_work/expert_call_list.html).

The crucial step is that within 12 hours (see note below) of the time of the event (the detection time indicated by the SNEWS email, not the time of the email reception), the numbers of the runs surrounding the event must be appended to the non-deletion list (note the possible time zone conversion needed). For this you need access as uboonepro or root user and follow the link below

https://cdcvs.fnal.gov/redmine/projects/uboone-operations/wiki/DM_-_Expert_Documentation#Collaborator-has-asked-me-the-DM-expert-to-prevent-the-deletion-of-one-or-more-SN-runs

Nevertheless, the non-deletion list may fail to protect these runs in case of losing the ubdaq-prod-evb.fnal.gov machine (which hosts the /home directories). For this reason, upon an SNEWS alert, the Data Management expert on-call should be contacted to stop the automated deletion of SN files (shutting down the daemon running on ubdaq-prod-ws02) and coordinate a backup copy (not a transfer) of the files to other machines.

Note: the window for responding to an SNEWS alert is given by the data rates of the SN stream. In normal conditions, all SEBs write less than 60 MB/s, giving > 48 h to respond to the alert. But in special circumstances (HV bursts, ASIC misconfiguration), data rates can reach 260 MB/s, reducing the window to 12 h.

In addition, KamLAND has an SN monitor looking for IBD candidates in real time which can be used to alert the community about a nearby massive star in the pre-SN stages.

https://www.awa.tohoku.ac.jp/kamland/SNmonitor/regist/index.html

Run Coordinators may want to subscribe to that alert too. If a KamLAND alert is received, the only required action is to ensure that the MicroBooNE SN stream is running. This can be checked by confirming the stability of the last-24 h data rates of the SN stream (requires connection to the Fermilab VPN or tunneling)

http://ubdaq-prod-evb.fnal.gov:8080/gweb/graph.php?r=day&z=xlarge&title=SN+Disk+Write+Rate&vl=B%2Fs&x=&n=&hreg[]=ubdaq-prod-seb-priv%5B0-1%5D%5B0-9%5D&mreg[]=SN-DiskWriteRate_Asebappseb%5B0-1%5D%5B0-9%5D&gtype=line&glegend=show&aggregate=1

If any of them (except seb10, the PMT server) is currently below 30 MB/s, the associated crate has a problem. This is usually fixed by restarting the run.