Feature #23545: Install Graphite and Grafana for metric monitoring on SBN-FD cluster - sbndaq - Fermilab Redmine

Feature #23545

Milestone #23538: DAQ Software Prepared for ICARUS Cooldown and Filling

Install Graphite and Grafana for metric monitoring on SBN-FD cluster

Added by Wesley Ketchum about 1 year ago. Updated 12 months ago.

Status:

Resolved

Priority:

High

Assignee:

Gennadiy Lukhanin

Start date:

11/06/2019

Due date:

% Done:

100%

Estimated time:

Co-Assignees:

Duration:

History

#1 Updated by Wesley Ketchum about 1 year ago

Assignee set to Gennadiy Lukhanin

#2 Updated by Gennadiy Lukhanin about 1 year ago

% Done changed from 0 to 90
Status changed from New to Work in progress

Grafana is running on icarus-db02 in a screen session under the artdaq user account.
Artdaq processes should send metrics to 192.168.191.18:2003. The web interface is accessible on 192.168.191.18:10080 and 127.0.0.1:10080 or can be port-forwared over SSH.

[artdaq@icarus-db02 grafana]$ ksu
Authenticated lukhanin@FNAL.GOV
Account root: authorization for lukhanin@FNAL.GOV successful
Changing uid to root (0)
[root@icarus-db02 grafana]#  netstat -apn |grep tcp |grep -E "(python2|httpd|grafana)" 
tcp        0      0 192.168.191.18:7002     0.0.0.0:*               LISTEN      41649/python2
tcp        0      0 192.168.191.18:10080    0.0.0.0:*               LISTEN      41659/httpd
tcp        0      0 127.0.0.1:10080         0.0.0.0:*               LISTEN      41659/httpd
tcp        0      0 127.0.0.1:10081         0.0.0.0:*               LISTEN      41659/httpd
tcp        0      0 127.0.0.1:10082         0.0.0.0:*               LISTEN      41657/grafana-serve
tcp        0      0 192.168.191.18:2003     0.0.0.0:*               LISTEN      41649/python2
tcp        0      0 192.168.191.18:2004     0.0.0.0:*               LISTEN      41649/python2
tcp        0      0 127.0.0.1:38728         127.0.0.1:10082         ESTABLISHED 41736/httpd
tcp        0      0 127.0.0.1:38730         127.0.0.1:10082         ESTABLISHED 41735/httpd
tcp        0      0 127.0.0.1:38698         127.0.0.1:10082         ESTABLISHED 41743/httpd
tcp        0      0 127.0.0.1:10082         127.0.0.1:38702         ESTABLISHED 41657/grafana-serve
tcp        0      0 127.0.0.1:10082         127.0.0.1:38670         ESTABLISHED 41657/grafana-serve
tcp        0      0 127.0.0.1:10082         127.0.0.1:38728         ESTABLISHED 41657/grafana-serve
tcp        0      0 127.0.0.1:38672         127.0.0.1:10082         ESTABLISHED 41662/httpd
tcp        0      0 127.0.0.1:10082         127.0.0.1:38704         ESTABLISHED 41657/grafana-serve
tcp        0      0 127.0.0.1:10082         127.0.0.1:38730         ESTABLISHED 41657/grafana-serve
tcp        0      0 127.0.0.1:38668         127.0.0.1:10082         ESTABLISHED 41665/httpd
tcp        0      0 127.0.0.1:38700         127.0.0.1:10082         ESTABLISHED 41744/httpd
tcp        0      0 127.0.0.1:10082         127.0.0.1:38672         ESTABLISHED 41657/grafana-serve
tcp        0      0 127.0.0.1:10082         127.0.0.1:38700         ESTABLISHED 41657/grafana-serve
tcp        0      0 127.0.0.1:10082         127.0.0.1:38666         ESTABLISHED 41657/grafana-serve
tcp        0      0 127.0.0.1:38670         127.0.0.1:10082         ESTABLISHED 41663/httpd
tcp        0      0 127.0.0.1:38666         127.0.0.1:10082         ESTABLISHED 41666/httpd
tcp        0      0 127.0.0.1:38702         127.0.0.1:10082         ESTABLISHED 41742/httpd
tcp        0      0 127.0.0.1:10082         127.0.0.1:38698         ESTABLISHED 41657/grafana-serve
tcp        0      0 127.0.0.1:38704         127.0.0.1:10082         ESTABLISHED 41745/httpd
tcp        0      0 127.0.0.1:10082         127.0.0.1:38668         ESTABLISHED 41657/grafana-serve

[root@icarus-db02 grafana]#  ps -ef  |grep -E "(python2|httpd|grafana)" 
root       3197      1  0 12:28 ?        00:00:02 /usr/bin/python2 -Es /usr/sbin/tuned -l -P
artdaq    41511  41190  0 16:51 pts/1    00:00:00 /bin/bash ./grafana-ctrl.sh
artdaq    41649  41529  0 16:51 ?        00:00:00 /usr/bin/python2 -s /usr/bin/carbon-cache --config=/mnt/work/container/carbon/config/carbon.conf --pidfile=/mnt/work/container/pids/carbon.pid --logdir=/mnt/work/container/logs/carbon --instance=1 start
artdaq    41657  41529  0 16:51 pts/1    00:00:00 /usr/sbin/grafana-server --config=/mnt/work/container/grafana/config/grafana.ini --pidfile=/mnt/work/container/pids/grafana.pid --packaging=rpm --homepath=/usr/share/grafana cfg:default.paths.logs=/mnt/work/container/logs/grafana cfg:default.paths.data=/mnt/work/container/grafana/data cfg:default.paths.plugins=/var/lib/grafana/plugins cfg:default.paths.provisioning=/mnt/work/container/grafana/config/provisioning
artdaq    41659  41529  0 16:51 ?        00:00:00 httpd -f /mnt/work/container/httpd/config/conf/httpd.conf
artdaq    41662  41659  0 16:51 ?        00:00:00 httpd -f /mnt/work/container/httpd/config/conf/httpd.conf
artdaq    41663  41659  0 16:51 ?        00:00:00 httpd -f /mnt/work/container/httpd/config/conf/httpd.conf
artdaq    41665  41659  0 16:51 ?        00:00:00 httpd -f /mnt/work/container/httpd/config/conf/httpd.conf
artdaq    41666  41659  0 16:51 ?        00:00:00 httpd -f /mnt/work/container/httpd/config/conf/httpd.conf
artdaq    41735  41659  0 16:51 ?        00:00:00 httpd -f /mnt/work/container/httpd/config/conf/httpd.conf
artdaq    41736  41659  0 16:51 ?        00:00:00 httpd -f /mnt/work/container/httpd/config/conf/httpd.conf
artdaq    41742  41659  0 16:51 ?        00:00:00 httpd -f /mnt/work/container/httpd/config/conf/httpd.conf
artdaq    41743  41659  0 16:51 ?        00:00:00 httpd -f /mnt/work/container/httpd/config/conf/httpd.conf
artdaq    41744  41659  0 16:51 ?        00:00:00 httpd -f /mnt/work/container/httpd/config/conf/httpd.conf
artdaq    41745  41659  0 16:51 ?        00:00:00 httpd -f /mnt/work/container/httpd/config/conf/httpd.conf
root      42280  42147  0 16:58 pts/0    00:00:00 grep --color=auto -E (python2|httpd|grafana)

#3 Updated by Gennadiy Lukhanin 12 months ago

% Done changed from 90 to 100
Status changed from Work in progress to Resolved

Configured Grafana to start on boot with SystemD.

[root@icarus-db02 grafana]# systemctl status  grafana.service
● grafana.service - Grafana service
   Loaded: loaded (/etc/systemd/system/grafana.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2020-01-30 17:00:34 CST; 4min 13s ago
  Process: 47815 ExecStop=/usr/bin/rm -f /data/grafana/data/container/terminate-grafana.sh (code=exited, status=0/SUCCESS)
 Main PID: 47891 (grafana-ctrl.sh)
    Tasks: 57
   CGroup: /system.slice/grafana.service
           ├─47891 /bin/bash /data/grafana/grafana-ctrl.sh
           ├─47894 Singularity runtime parent
           ├─47915 sinit
           ├─47937 /bin/bash /run.sh /mnt/work/container
           ├─48037 /usr/bin/python2 -s /usr/bin/carbon-cache --config=/mnt/work/container/carbon/config/carbon.conf --pidfile=/mnt/work/c...
           ├─48047 /usr/sbin/grafana-server --config=/mnt/work/container/grafana/config/grafana.ini --pidfile=/mnt/work/container/pids/gr...
           ├─48049 httpd -f /mnt/work/container/httpd/config/conf/httpd.conf
           ├─48056 httpd -f /mnt/work/container/httpd/config/conf/httpd.conf
           ├─48140 httpd -f /mnt/work/container/httpd/config/conf/httpd.conf
           ├─48144 httpd -f /mnt/work/container/httpd/config/conf/httpd.conf
           ├─48156 httpd -f /mnt/work/container/httpd/config/conf/httpd.conf
           ├─48157 httpd -f /mnt/work/container/httpd/config/conf/httpd.conf
           ├─48159 httpd -f /mnt/work/container/httpd/config/conf/httpd.conf
           ├─48180 httpd -f /mnt/work/container/httpd/config/conf/httpd.conf
           ├─48181 httpd -f /mnt/work/container/httpd/config/conf/httpd.conf
           ├─48182 httpd -f /mnt/work/container/httpd/config/conf/httpd.conf
           ├─48185 httpd -f /mnt/work/container/httpd/config/conf/httpd.conf
           └─49156 sleep 2

Jan 30 17:04:40 icarus-db02.fnal.gov grafana-ctrl.sh[47891]: + '[' -f /mnt/work/container/terminate-grafana.sh ']'
Jan 30 17:04:40 icarus-db02.fnal.gov grafana-ctrl.sh[47891]: + sleep 2
Jan 30 17:04:42 icarus-db02.fnal.gov grafana-ctrl.sh[47891]: + '[' -f /mnt/work/container/terminate-grafana.sh ']'
Jan 30 17:04:42 icarus-db02.fnal.gov grafana-ctrl.sh[47891]: + sleep 2
Jan 30 17:04:44 icarus-db02.fnal.gov grafana-ctrl.sh[47891]: + '[' -f /mnt/work/container/terminate-grafana.sh ']'
Jan 30 17:04:44 icarus-db02.fnal.gov grafana-ctrl.sh[47891]: + sleep 2
Jan 30 17:04:46 icarus-db02.fnal.gov grafana-ctrl.sh[47891]: + '[' -f /mnt/work/container/terminate-grafana.sh ']'
Jan 30 17:04:46 icarus-db02.fnal.gov grafana-ctrl.sh[47891]: + sleep 2
Jan 30 17:04:48 icarus-db02.fnal.gov grafana-ctrl.sh[47891]: + '[' -f /mnt/work/container/terminate-grafana.sh ']'
Jan 30 17:04:48 icarus-db02.fnal.gov grafana-ctrl.sh[47891]: + sleep 2

[artdaq@icarus-db02 grafana]$ more grafana.service
# Quick HowTo:
# 1. Copy this file to /etc/systemd/system/grafana.service
# 2. Run "systemctl daemon-reload" 
# 3. Run "systemctl enable grafana.service" 
# 5. Run "systemctl start grafana.service" 
# 
# 6. Check status "systemctl status grafana.service" 
# 7. Stop "systemctl stop grafana.service" 

[Unit]
Description=Grafana service
After=syslog.target network.target

[Service]
Type=simple
User=artdaq
Group=E-1052
WorkingDirectory=/data/grafana
ExecStart=/data/grafana/grafana-ctrl.sh
ExecStop=/usr/bin/rm -f /data/grafana/data/container/terminate-grafana.sh
#RemainAfterExit=true
KillMode=process
TimeoutSec=60
Restart=always
RestartSec=120
StartLimitInterval=43200
StartLimitBurst=3
[Install]
WantedBy=multi-user.target

[artdaq@icarus-db02 grafana]$ more grafana-ctrl.sh
#!/bin/bash
export PATH=/usr/local/go/bin:$PATH
rm -rf /data/grafana/data/container/httpd
export GRAFANA_CONTAINER_HOME=/data/grafana/data/container
/usr/local/bin/singularity run -B /data/grafana/data:/mnt/work /data/grafana/image/grafana2.simg /mnt/work/container

Also available in: Atom PDF

Project

General

Profile

SBN Online » sbndaq

Issues