cet-is » art

We have long surmised that the reason the Mu2e last-stage jobs take 30 minutes to open the art input file is because of massive bloat in the parameter set registry ( and therefore on the RootFileDB inside the art event-data disk file ).

I can show that, for typical mixed files, the size of the disk file is 2.2 GB and that the size on-disk of the RootFileDB is 1.2 GB! The registry contains over 1,000,000 parameter sets. So we have two problems: we have a huge hit on disk space and it takes forever the open the input files.

Andrei has long believed that the bloat in the ParameterSetRegistry is caused by three variables:

firstRun
firstSubRun
baseSeed

I bet he is right. When we run a grid process we set

firstRun:grid_cluster_number
firstSubRun:grid_process_number
baseSeed:random_number

Our script that scrubs the grid output for failed jobs also looks for duplicate base seeds. This algorithm effectively guarantees unqiue event IDs (until the grid starts recycling cluster numbers ).

I have done some homework that will allow us to test the above hypotheses. A quick look suggests that we can easily confirm them but doing a proper test is beyond by regexp ability.

On mu2egpvm*, in the directory

/mu2e/data/users/kutschke/ParameterSetBloat

There are 4 files:
ls -s *.txt
512 beamFlashOnMothers.txt
985056 dataMerged.txt
197408 detectorBeamFlash_f3.txt
197408 detectorBeamFlash.txt

These files contain a text image of the contents of the ParameterSetRegistry found in 4 of Andrei's files.
Each line in ech file is one parameter set, in the one-liner string representation.

If someone has the regexp ability please try the following on all 4 files. Try it on the small file first and
when it works, try it on the bigger files:

1)
substitute "firstRun:* " to "firstRun: " everywhere in the file
What I mean is simply erase the value of the parameter.
same for the other two variables

2)
wc original_file
sort -u modified_file | wc

I hope that dataMerged.txt will drop from over 1,000,000 lines to as few as a few hundred.

If this proves to be true we can discuss a strategy for fixing these files: I think this involves compressing the ParameterSets, rebuilding the registry and then remapping all of the parameterset Ids in all of the Provenance objects.

And it is time to revist the question about controlled and uncontrolled information in configurations. I don't know what the right answer is but the present solution is not viable.