Bug #486 files with __END_OF_FILE_STREAM__ parents

Added by Marc Mengel almost 11 years ago. Updated almost 11 years ago.

Start date:
Due date:
% Done:


Estimated time:


e've found a new corner of phase space.

In the good_lbns directory we have accumulated a bunch of root files (empty) and files with
'parents' : NameOrIdList(['__END_OF_FILE_STREAM__']),
I'd seen that before when I was doing that cleanup from recocert crashing. It is caused by worker jobs that received no files from SAM. The jobs themselves didn't crash or anything. They just never received anything to do. So they exited and wrote out empty root files with files with the above (totally correct) parentage.

Your cron, finds nothing wrong, so copies them to good_lbns. Your copy_to_sam, however, sees them as:
Error: Bad metadata on %s cert_261188_p20.16.08_4370065.root
parentage files don't exist. This isn't fatal or anything. But it is rather annoying. The log files are just full of these, over and over and over again.

The current set of them would have happened because of the SAM problems this week. These things will accumulate over time.

When I found them before, I wrote a little script that looked for the above error message and moved both the py and it's root file to "bad_lbns/ENDOFFILE/".

There are three options for what to do with them, depending on what you had envisioned for bad_lbns/ itself. They could be moved to either bad_lbns/ itself, or bad_lbns/ENDOFFILE or they could be deleted.

I have no strong preference. I think the signature is very clear and they are unrecoverable. Depending on why the files weren't delivered, either all of the files where handled by other processes or they will be rerun due to another of your conditions. So I think that the files are pretty worthless and I lean toward just nuking them.

This cleanup could be added to the worker code (not copy them back at all) or to LBN check or the copy to sam part could do it or we could make it part of the cleanup. But SOMETHING needs to get rid of those things.


#1 Updated by Marc Mengel almost 11 years ago

Rough draft of code is in r8

Need to flesh out test routine to write sort of metadata files and validate check.

Also available in: Atom PDF