Project

General

Profile

Monitoring Information

Log files are generated by almost all DAQ subsystems. These logfiles go into the /daqlogs directory on each of the NOvA DAQ clusters. This volume is served off of the message logger machines (i.e. novadaq-far-msglogger.fnal.gov) for each cluster.

The general hierarchy of the /daqlogs area is:

daqlogs/
├── DCM
│   └── dcm-2-06-05
├── FarDet
│   ├── CSSApplications
│   ├── DCM
│   ├── DCSDatabase
│   ├── DCSMessaging
│   ├── NullPartition
│   ├── Partition-
│   ├── Partition0
│   ├── Partition-0
│   ├── Partition1
│   ├── Partition11
│   ├── Partition2
│   ├── Partition23
│   ├── Partition3
│   ├── Partition4
│   ├── Partition90
│   ├── Partition91
│   ├── Partition92
│   ├── Partition99
│   ├── Partition--opt
│   ├── Partitionpart
│   ├── ResourceManager
│   ├── TDUDelayCalibrationConstants
│   └── TDUDelayMonitor
├── NDOS
│   └── DCM
├── NearDet
│   └── Partition0
├── NovaSpillServer
│   ├── NssSpillForwarder
│   └── NssSpillReceiver
└── SNEWSMessageForwarder
    ├── FarGateway
    └── Gateway

Logs will live in this structure for approximately 15 days before the File Transfer System bundles them up and copies them to tape archives at FNAL.

Retrieving Log files

If the log file you are interested in is no longer on disk, it can be easily retrieved from tape using the SAM data handling system. Here is the state of the art in getting log files back from tape (step by step).

  • Determine the name of the log you are interested in.

Example:

I want a dcm log file from december 1st 2014 (but I don't know exactly which one so I need a list of the names)

Use:

samweb list-files "file_name like dcm%_20141201%.log and availability: virutal" 

This will give a list of file names:

dcm-1-02-01_20141201_142048.log
dcm-1-02-01_20141201_112610.log
dcm-1-01-02_20141201_105927.log
…
dcm-1-01-02_20141201_110038.log
dcm-1-01-03_20141201_112306.log

  • Locate the tar file that has been stored in (since we bundle them up)
samweb file-lineage descendants dcm-3-02-03_20120101_135454.log

log_merge_ids_3552422_3552514_1325735655.tar   <-- This is the tar file that your log is in
  • Retrieve the tar file using IFDH fetch
ifdh_fetch log_merge_ids_34578517_38914559_1418153002.tar

(buch of diagnostic stuff spits out)
…

Your file is copied to the local directory:

[anorman@novagpvm06 anorman]$ ls -l log*
-rw-r--r-- 1 anorman nova 6236354560 Dec 18 13:34 log_merge_ids_34578517_38914559_1418153002.tar
  • Untar the tarbal and examine your log. (there will be more than just your one log in there)