Project

General

Profile

Feature #9738

speed up jobsub_history

Added by Dennis Box over 5 years ago. Updated over 5 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Parag Mhashilkar
Category:
-
Target version:
Start date:
07/27/2015
Due date:
% Done:

0%

Estimated time:
Stakeholders:
Duration:

Description

jobsub_history calls condor_history and is agonizingly for both the user and the server. It is pretty much needed for users to check if their jobs have completed or died.

I did some testing that shows that properly formatted condor_history output stored in in an sqllite database is much much faster. Implement this with the following features:

  • A process which periodically runs condor_history on the tail of the history log and puts these into the sqllite database
  • All the jobsub_history (and jobsub_fetchlog file lookup) using this sqllite database
  • Addtional functionality in jobsub_preen: when sandboxes removed, remove entries from sqllite database

History

#1 Updated by Parag Mhashilkar over 5 years ago

  • Target version changed from v1.1.5 to v1.2

#2 Updated by Dennis Box over 5 years ago

  • Target version changed from v1.2 to v1.1.6

#3 Updated by Dennis Box over 5 years ago

  • Status changed from New to Feedback
  • Assignee changed from Dennis Box to Parag Mhashilkar

there may be some issues with the cron script fill_jobsub_history.sh/py but they are minor and fixable and good enough for rc1.

#4 Updated by Dennis Box over 5 years ago

  • Status changed from Feedback to Resolved

#5 Updated by Dennis Box over 5 years ago

  • Status changed from Resolved to Closed

Also available in: Atom PDF