Idea #20970

Add metrics and log collection to jobsub server

Added by Kevin Retzke over 2 years ago. Updated over 1 year ago.

Target version:
Start date:
Due date:
% Done:


Estimated time:


We (FIFE/landscape) currently collect the Jobsub submit log, and recently started a test process collecting the access log, but really this should be something packaged with jobsub I think. It's just a filebeat executable and a config file that need to be deployed. I've attached an example simple filebeat.yml config that collects several jobsub logs and ships them to our Kafka cluster (we'll probably set up an alias to use soon instead of lssrvXX), from where we can process them and store them in Elasticsearch.

Also, I'd love to see the Jobsub server collect metrics, on things like HTTP request counts and timing, errors, etc. I'd highly recommend the Prometheus client library, which is very easy to use and allows for quickly adding metrics. You can then have us collect the metrics with Prometheus, and/or push metrics directly to our graphite, either of which would make them available for dashboards and alerts in Grafana.

Let me know what assistance you need!

filebeat-jobsub.yml (454 Bytes) filebeat-jobsub.yml Kevin Retzke, 09/27/2018 02:45 PM


#1 Updated by Dennis Box over 2 years ago

  • Status changed from New to Assigned
  • Assignee set to Dennis Box
  • Target version set to v1.2.9

#2 Updated by Dennis Box over 2 years ago

  • Target version changed from v1.2.9 to v1.2.9.rc_x

#3 Updated by Dennis Box about 2 years ago

  • Target version changed from v1.2.9.rc_x to v1.3

#4 Updated by Shreyas Bhat about 2 years ago

  • Assignee changed from Dennis Box to Shreyas Bhat

#5 Updated by Dennis Box almost 2 years ago

  • Target version changed from v1.3 to v1.3.1

#6 Updated by Dennis Box over 1 year ago

  • Target version changed from v1.3.1 to v1.3.2

Also available in: Atom PDF