Project

General

Profile

Idea #20970

Add metrics and log collection to jobsub server

Added by Kevin Retzke 12 months ago. Updated 4 months ago.

Status:
Assigned
Priority:
Normal
Assignee:
Category:
-
Target version:
Start date:
09/27/2018
Due date:
% Done:

0%

Estimated time:
Stakeholders:
Duration:

Description

We (FIFE/landscape) currently collect the Jobsub submit log, and recently started a test process collecting the access log, but really this should be something packaged with jobsub I think. It's just a filebeat executable and a config file that need to be deployed. I've attached an example simple filebeat.yml config that collects several jobsub logs and ships them to our Kafka cluster (we'll probably set up an alias to use soon instead of lssrvXX), from where we can process them and store them in Elasticsearch.

Also, I'd love to see the Jobsub server collect metrics, on things like HTTP request counts and timing, errors, etc. I'd highly recommend the Prometheus client library, which is very easy to use and allows for quickly adding metrics. You can then have us collect the metrics with Prometheus, and/or push metrics directly to our graphite, either of which would make them available for dashboards and alerts in Grafana.

Let me know what assistance you need!

filebeat-jobsub.yml (454 Bytes) filebeat-jobsub.yml Kevin Retzke, 09/27/2018 02:45 PM

History

#1 Updated by Dennis Box 10 months ago

  • Status changed from New to Assigned
  • Assignee set to Dennis Box
  • Target version set to v1.2.9

#2 Updated by Dennis Box 9 months ago

  • Target version changed from v1.2.9 to v1.2.9.rc_x

#3 Updated by Dennis Box 7 months ago

  • Target version changed from v1.2.9.rc_x to v1.3

#4 Updated by Shreyas Bhat 5 months ago

  • Assignee changed from Dennis Box to Shreyas Bhat

#5 Updated by Dennis Box 4 months ago

  • Target version changed from v1.3 to v1.3.1


Also available in: Atom PDF