Project

General

Profile

Feature #23545

Milestone #23538: DAQ Software Prepared for ICARUS Cooldown and Filling

Install Graphite and Grafana for metric monitoring on SBN-FD cluster

Added by Wesley Ketchum 3 months ago. Updated 3 months ago.

Status:
Work in progress
Priority:
High
Assignee:
Gennadiy Lukhanin
Start date:
11/06/2019
Due date:
% Done:

90%

Estimated time:
Co-Assignees:
Duration:

History

#1 Updated by Wesley Ketchum 3 months ago

  • Assignee set to Gennadiy Lukhanin

#2 Updated by Gennadiy Lukhanin 3 months ago

  • % Done changed from 0 to 90
  • Status changed from New to Work in progress

Grafana is running on icarus-db02 in a screen session under the artdaq user account.
Artdaq processes should send metrics to 192.168.191.18:2003. The web interface is accessible on 192.168.191.18:10080 and 127.0.0.1:10080 or can be port-forwared over SSH.

[artdaq@icarus-db02 grafana]$ ksu
Authenticated lukhanin@FNAL.GOV
Account root: authorization for lukhanin@FNAL.GOV successful
Changing uid to root (0)
[root@icarus-db02 grafana]#  netstat -apn |grep tcp |grep -E "(python2|httpd|grafana)" 
tcp        0      0 192.168.191.18:7002     0.0.0.0:*               LISTEN      41649/python2
tcp        0      0 192.168.191.18:10080    0.0.0.0:*               LISTEN      41659/httpd
tcp        0      0 127.0.0.1:10080         0.0.0.0:*               LISTEN      41659/httpd
tcp        0      0 127.0.0.1:10081         0.0.0.0:*               LISTEN      41659/httpd
tcp        0      0 127.0.0.1:10082         0.0.0.0:*               LISTEN      41657/grafana-serve
tcp        0      0 192.168.191.18:2003     0.0.0.0:*               LISTEN      41649/python2
tcp        0      0 192.168.191.18:2004     0.0.0.0:*               LISTEN      41649/python2
tcp        0      0 127.0.0.1:38728         127.0.0.1:10082         ESTABLISHED 41736/httpd
tcp        0      0 127.0.0.1:38730         127.0.0.1:10082         ESTABLISHED 41735/httpd
tcp        0      0 127.0.0.1:38698         127.0.0.1:10082         ESTABLISHED 41743/httpd
tcp        0      0 127.0.0.1:10082         127.0.0.1:38702         ESTABLISHED 41657/grafana-serve
tcp        0      0 127.0.0.1:10082         127.0.0.1:38670         ESTABLISHED 41657/grafana-serve
tcp        0      0 127.0.0.1:10082         127.0.0.1:38728         ESTABLISHED 41657/grafana-serve
tcp        0      0 127.0.0.1:38672         127.0.0.1:10082         ESTABLISHED 41662/httpd
tcp        0      0 127.0.0.1:10082         127.0.0.1:38704         ESTABLISHED 41657/grafana-serve
tcp        0      0 127.0.0.1:10082         127.0.0.1:38730         ESTABLISHED 41657/grafana-serve
tcp        0      0 127.0.0.1:38668         127.0.0.1:10082         ESTABLISHED 41665/httpd
tcp        0      0 127.0.0.1:38700         127.0.0.1:10082         ESTABLISHED 41744/httpd
tcp        0      0 127.0.0.1:10082         127.0.0.1:38672         ESTABLISHED 41657/grafana-serve
tcp        0      0 127.0.0.1:10082         127.0.0.1:38700         ESTABLISHED 41657/grafana-serve
tcp        0      0 127.0.0.1:10082         127.0.0.1:38666         ESTABLISHED 41657/grafana-serve
tcp        0      0 127.0.0.1:38670         127.0.0.1:10082         ESTABLISHED 41663/httpd
tcp        0      0 127.0.0.1:38666         127.0.0.1:10082         ESTABLISHED 41666/httpd
tcp        0      0 127.0.0.1:38702         127.0.0.1:10082         ESTABLISHED 41742/httpd
tcp        0      0 127.0.0.1:10082         127.0.0.1:38698         ESTABLISHED 41657/grafana-serve
tcp        0      0 127.0.0.1:38704         127.0.0.1:10082         ESTABLISHED 41745/httpd
tcp        0      0 127.0.0.1:10082         127.0.0.1:38668         ESTABLISHED 41657/grafana-serve


[root@icarus-db02 grafana]#  ps -ef  |grep -E "(python2|httpd|grafana)" 
root       3197      1  0 12:28 ?        00:00:02 /usr/bin/python2 -Es /usr/sbin/tuned -l -P
artdaq    41511  41190  0 16:51 pts/1    00:00:00 /bin/bash ./grafana-ctrl.sh
artdaq    41649  41529  0 16:51 ?        00:00:00 /usr/bin/python2 -s /usr/bin/carbon-cache --config=/mnt/work/container/carbon/config/carbon.conf --pidfile=/mnt/work/container/pids/carbon.pid --logdir=/mnt/work/container/logs/carbon --instance=1 start
artdaq    41657  41529  0 16:51 pts/1    00:00:00 /usr/sbin/grafana-server --config=/mnt/work/container/grafana/config/grafana.ini --pidfile=/mnt/work/container/pids/grafana.pid --packaging=rpm --homepath=/usr/share/grafana cfg:default.paths.logs=/mnt/work/container/logs/grafana cfg:default.paths.data=/mnt/work/container/grafana/data cfg:default.paths.plugins=/var/lib/grafana/plugins cfg:default.paths.provisioning=/mnt/work/container/grafana/config/provisioning
artdaq    41659  41529  0 16:51 ?        00:00:00 httpd -f /mnt/work/container/httpd/config/conf/httpd.conf
artdaq    41662  41659  0 16:51 ?        00:00:00 httpd -f /mnt/work/container/httpd/config/conf/httpd.conf
artdaq    41663  41659  0 16:51 ?        00:00:00 httpd -f /mnt/work/container/httpd/config/conf/httpd.conf
artdaq    41665  41659  0 16:51 ?        00:00:00 httpd -f /mnt/work/container/httpd/config/conf/httpd.conf
artdaq    41666  41659  0 16:51 ?        00:00:00 httpd -f /mnt/work/container/httpd/config/conf/httpd.conf
artdaq    41735  41659  0 16:51 ?        00:00:00 httpd -f /mnt/work/container/httpd/config/conf/httpd.conf
artdaq    41736  41659  0 16:51 ?        00:00:00 httpd -f /mnt/work/container/httpd/config/conf/httpd.conf
artdaq    41742  41659  0 16:51 ?        00:00:00 httpd -f /mnt/work/container/httpd/config/conf/httpd.conf
artdaq    41743  41659  0 16:51 ?        00:00:00 httpd -f /mnt/work/container/httpd/config/conf/httpd.conf
artdaq    41744  41659  0 16:51 ?        00:00:00 httpd -f /mnt/work/container/httpd/config/conf/httpd.conf
artdaq    41745  41659  0 16:51 ?        00:00:00 httpd -f /mnt/work/container/httpd/config/conf/httpd.conf
root      42280  42147  0 16:58 pts/0    00:00:00 grep --color=auto -E (python2|httpd|grafana)

Also available in: Atom PDF