Support #3752

Migrate DBU DataQuality table filling process to near-online

Added by Robert Hatcher almost 8 years ago. Updated almost 8 years ago.

Start date:
Due date:
% Done:


Estimated time:


Andy Blake's jobs (previously run on Near/Far minos-om machines) need to be migrated to central site.

In particular at least the tables: DBUFARRUNQUALITY[VLD] and DBUHVFROMSINGLES[VLD] need to be filled for CALDRIFT calibration processing.


#1 Updated by Andrew Blake almost 8 years ago

I've reviewed the Data Quality software at the near and far detector.

The four modules appear to be up to date in CVS:
CandMorgue/FillNearRunQuality (Near Detector)
CandMorgue/FillFarRunQuality (Far Detector)
CandMorgue/FillHvFromSingles (Far Detector)
SpillTiming/FillSpillServerMon (Far Detector)
(A cvs diff reveals that Robert has fixed some of my bad programming...
sorry and thanks...)

I've committed the ND and FD job control macros to the CandMorgue package:
runDataQualityND.C (Near Detector)
runDataQualityFD.C (Far Detector)

The Near Detector code ought to work! The Far Detector code has never been
run over non-physics runs, so this will need testing.

The Near Detector hot/warm channel finder will also need a home.

#2 Updated by Andrew Blake almost 8 years ago

All Database-related Data Quality modules are now up to date, and committed to the CandMorgue package.
This includes the refurbished hot channel finder, which is committed to the CandMorgue package.

The modules should work on Near/Far and Physics/NonPhysics data (I've tested this on some sample data).
For Non-Physics data, the default mode is to just to skip over the data.

The CandMorgue package also contains job control macros: 'runDataQualityND.C' for the Near Detector;
and 'runDataQualityFD.C' for the Far Detector. These macros run all the necessary modules, including
the hot-channel finding.

The next step should be to set up something on the minos-nearline machine (for initial testing,
the DB writing should be disabled).

#3 Updated by Robert Hatcher almost 8 years ago

Andy Blake wrote on May 29, 2013 12:12,
I just tried logging on to the minos-nearline machine and running some software.

Things didn't work first time for me. If I source the development version of minossoft, and try running root or loon, I see errors relating to various missing libraries (libcrypto, libssl, libXpm...).

I get similar results at Cambridge, when I run my SLC5 minossoft on one of our SLC6 machines.

Any suggestions ?

So what's probably happened is that the code is built against these system-level libraries; at executable link time the exact version used is scribbled into the executable and the code won't run with newer libraries (i.e. vs. The way we've dealt with this sort of thing in the past is by putting a copy of the older libraries in one of the directories in the $LD_LIBRARY_PATH (perhaps the MINOS_EXTERN area, or some special SLF5 area that would get added to the path by the setup script).

What would need to happen it to determine exactly which libraries (and versions) are missing. One tool for find them is:

ldd `which loon`

and looking for which libraries didn't resolve.

#4 Updated by Robert Hatcher almost 8 years ago

Ack, I see while I was writing this up it looks like the system has been downgraded to SLF5.

Okay, we should continue this SLF6 effort on a separate SLF6 VM.

#5 Updated by Rashid Mehdiyev almost 8 years ago

Exactly. The SLF6 efforts need to be pursued further on a separate machine, but right now we are simply running out of time to
restart DQ processes. So, we had to make this choice.

#6 Updated by Arthur Kreymer almost 8 years ago

Sorry, I thought that I have included everyone on the watch list
of the request to reinstall minos-nearline with SLF 5 matching minos50.

Apparently that registration got lost.

minos-nearline should now look just like minos50.
I have run a simple loon process successfully.

#7 Updated by Arthur Kreymer almost 8 years ago

I have moved some recent detector files to new directories


I will update the predator script to update this automatically,
later today.

I will also eventually purge older files.

For now, enjoy !

#8 Updated by Arthur Kreymer almost 8 years ago

Starting at 14:13, new mdaq files are being copied every 2 hours by predator
to /minos/data/neardet_data
and /minos/data/fardet_data

#9 Updated by Andrew Blake almost 8 years ago

Far Detector DQ processing is now running on the minos-nearline machine.

Last subrun of old processing was 50105/12 (2012-10-01)
First subrun of new processing is 51598/13 (2013-06-03)

(Everything in between will need to be recovered by running the FD DQ
'runDataQualityFD.C' macro over the raw data).

The processing is running from cron, under the mindata account. Things
look to be stable, and the DB entries look okay.

#10 Updated by Andrew Blake almost 8 years ago

I've just activated the ND DQ processing, on the nearline machine.

Currently, the DB write permissions are switched off. The processing will
instead write out Andy P's ROOT ntuples, which I'm storing here:


There look to be several bad channels in the Near Detector today - a good
testing ground for the bad channel finder!

#11 Updated by Andrew Blake almost 8 years ago

My DQ webpages are also active in their new locations:

But they're still under construction!

Also available in: Atom PDF