job reporters should split updates into multiple queues by task_id (odd/even, modulo 3, etc).
Basically the database code is locking Task rows; so if we have multiple bulk updates that involve the same task_id's, they get serialized. But if we have multiple bulk updates with distinct task_id's, then we can run them in parallel. Since most of our updates have task_id's in them (except currently Completed updates -- which could, if we got task_id's with the job_ids when we get active_jobs) then we could really improve our throughput.So:
- update active_jobs code to report jobsub_job_id's and task_ids
- update jobsub_q_scraper to pass those task_id's in when marking jobs completed
- add a number of queues (nqueues) to the job_reporter setup and assorted reporters config files
- file job updates in (task_id % nqueues) queues
- queues each get bulk reporting thread.
Then we can have actual parallel updates going reporting jobs that don't get serialized.