Add metrics and log collection to jobsub server
We (FIFE/landscape) currently collect the Jobsub submit log, and recently started a test process collecting the access log, but really this should be something packaged with jobsub I think. It's just a filebeat executable and a config file that need to be deployed. I've attached an example simple filebeat.yml config that collects several jobsub logs and ships them to our Kafka cluster (we'll probably set up an alias to use soon instead of lssrvXX), from where we can process them and store them in Elasticsearch.
Also, I'd love to see the Jobsub server collect metrics, on things like HTTP request counts and timing, errors, etc. I'd highly recommend the Prometheus client library, which is very easy to use and allows for quickly adding metrics. You can then have us collect the metrics with Prometheus, and/or push metrics directly to our graphite, either of which would make them available for dashboards and alerts in Grafana.
Let me know what assistance you need!