root.metadata.py files with __END_OF_FILE_STREAM__ parents
e've found a new corner of phase space.
In the good_lbns directory we have accumulated a bunch of root files (empty) and root.metadata.py files with
'parents' : NameOrIdList(['__END_OF_FILE_STREAM__']),
I'd seen that before when I was doing that cleanup from recocert crashing. It is caused by worker jobs that received no files from SAM. The jobs themselves didn't crash or anything. They just never received anything to do. So they exited and wrote out empty root files with metadata.py files with the above (totally correct) parentage.
Your reco_framework.py cron, finds nothing wrong, so copies them to good_lbns. Your copy_to_sam, however, sees them as:
Error: Bad metadata on %s cert_261188_p20.16.08_4370065.root
parentage files don't exist. This isn't fatal or anything. But it is rather annoying. The log files are just full of these, over and over and over again.
The current set of them would have happened because of the SAM problems this week. These things will accumulate over time.
When I found them before, I wrote a little script that looked for the above error message and moved both the py and it's root file to "bad_lbns/ENDOFFILE/".
There are three options for what to do with them, depending on what you had envisioned for bad_lbns/ itself. They could be moved to either bad_lbns/ itself, or bad_lbns/ENDOFFILE or they could be deleted.
I have no strong preference. I think the signature is very clear and they are unrecoverable. Depending on why the files weren't delivered, either all of the files where handled by other processes or they will be rerun due to another of your conditions. So I think that the files are pretty worthless and I lean toward just nuking them.
This cleanup could be added to the worker code (not copy them back at all) or to LBN check or the copy to sam part could do it or we could make it part of the cleanup. But SOMETHING needs to get rid of those things.