Archiving Tasks, Jobs and possibly job logs
This ticket is to track discussions of archiving jobs info, (which is a long term requiremnt 'cause we do not have database space to keep detailed job info indefinitely)
Current rough proposal:
- Periodically Submissions (Tasks) of a certain age would have all the Task /Job / JobHistories / JobFiles etc associated with them dumped off to a database dump of some sort (text dump, or maybe sqlite tables)
- this would be archived in an experiment provided dcache persistent area with small file aggregation.
- at the same time, we could jobsub_fetchlog the associated logs to the same SFA area
- Tasks thus archived would be replaced with entries in a new ArchivedTasks table, which would include summary counts, etc of jobs, etc as needed to continue to give total Campaign statistics.