Feature #23545
Milestone #23538: DAQ Software Prepared for ICARUS Cooldown and Filling
Install Graphite and Grafana for metric monitoring on SBN-FD cluster
Start date:
11/06/2019
Due date:
% Done:
90%
Estimated time:
Co-Assignees:
History
#1 Updated by Wesley Ketchum about 1 month ago
- Assignee set to Gennadiy Lukhanin
#2 Updated by Gennadiy Lukhanin about 1 month ago
- % Done changed from 0 to 90
- Status changed from New to Work in progress
Grafana is running on icarus-db02 in a screen session under the artdaq user account.
Artdaq processes should send metrics to 192.168.191.18:2003. The web interface is accessible on 192.168.191.18:10080 and 127.0.0.1:10080 or can be port-forwared over SSH.
[artdaq@icarus-db02 grafana]$ ksu Authenticated lukhanin@FNAL.GOV Account root: authorization for lukhanin@FNAL.GOV successful Changing uid to root (0) [root@icarus-db02 grafana]# netstat -apn |grep tcp |grep -E "(python2|httpd|grafana)" tcp 0 0 192.168.191.18:7002 0.0.0.0:* LISTEN 41649/python2 tcp 0 0 192.168.191.18:10080 0.0.0.0:* LISTEN 41659/httpd tcp 0 0 127.0.0.1:10080 0.0.0.0:* LISTEN 41659/httpd tcp 0 0 127.0.0.1:10081 0.0.0.0:* LISTEN 41659/httpd tcp 0 0 127.0.0.1:10082 0.0.0.0:* LISTEN 41657/grafana-serve tcp 0 0 192.168.191.18:2003 0.0.0.0:* LISTEN 41649/python2 tcp 0 0 192.168.191.18:2004 0.0.0.0:* LISTEN 41649/python2 tcp 0 0 127.0.0.1:38728 127.0.0.1:10082 ESTABLISHED 41736/httpd tcp 0 0 127.0.0.1:38730 127.0.0.1:10082 ESTABLISHED 41735/httpd tcp 0 0 127.0.0.1:38698 127.0.0.1:10082 ESTABLISHED 41743/httpd tcp 0 0 127.0.0.1:10082 127.0.0.1:38702 ESTABLISHED 41657/grafana-serve tcp 0 0 127.0.0.1:10082 127.0.0.1:38670 ESTABLISHED 41657/grafana-serve tcp 0 0 127.0.0.1:10082 127.0.0.1:38728 ESTABLISHED 41657/grafana-serve tcp 0 0 127.0.0.1:38672 127.0.0.1:10082 ESTABLISHED 41662/httpd tcp 0 0 127.0.0.1:10082 127.0.0.1:38704 ESTABLISHED 41657/grafana-serve tcp 0 0 127.0.0.1:10082 127.0.0.1:38730 ESTABLISHED 41657/grafana-serve tcp 0 0 127.0.0.1:38668 127.0.0.1:10082 ESTABLISHED 41665/httpd tcp 0 0 127.0.0.1:38700 127.0.0.1:10082 ESTABLISHED 41744/httpd tcp 0 0 127.0.0.1:10082 127.0.0.1:38672 ESTABLISHED 41657/grafana-serve tcp 0 0 127.0.0.1:10082 127.0.0.1:38700 ESTABLISHED 41657/grafana-serve tcp 0 0 127.0.0.1:10082 127.0.0.1:38666 ESTABLISHED 41657/grafana-serve tcp 0 0 127.0.0.1:38670 127.0.0.1:10082 ESTABLISHED 41663/httpd tcp 0 0 127.0.0.1:38666 127.0.0.1:10082 ESTABLISHED 41666/httpd tcp 0 0 127.0.0.1:38702 127.0.0.1:10082 ESTABLISHED 41742/httpd tcp 0 0 127.0.0.1:10082 127.0.0.1:38698 ESTABLISHED 41657/grafana-serve tcp 0 0 127.0.0.1:38704 127.0.0.1:10082 ESTABLISHED 41745/httpd tcp 0 0 127.0.0.1:10082 127.0.0.1:38668 ESTABLISHED 41657/grafana-serve
[root@icarus-db02 grafana]# ps -ef |grep -E "(python2|httpd|grafana)" root 3197 1 0 12:28 ? 00:00:02 /usr/bin/python2 -Es /usr/sbin/tuned -l -P artdaq 41511 41190 0 16:51 pts/1 00:00:00 /bin/bash ./grafana-ctrl.sh artdaq 41649 41529 0 16:51 ? 00:00:00 /usr/bin/python2 -s /usr/bin/carbon-cache --config=/mnt/work/container/carbon/config/carbon.conf --pidfile=/mnt/work/container/pids/carbon.pid --logdir=/mnt/work/container/logs/carbon --instance=1 start artdaq 41657 41529 0 16:51 pts/1 00:00:00 /usr/sbin/grafana-server --config=/mnt/work/container/grafana/config/grafana.ini --pidfile=/mnt/work/container/pids/grafana.pid --packaging=rpm --homepath=/usr/share/grafana cfg:default.paths.logs=/mnt/work/container/logs/grafana cfg:default.paths.data=/mnt/work/container/grafana/data cfg:default.paths.plugins=/var/lib/grafana/plugins cfg:default.paths.provisioning=/mnt/work/container/grafana/config/provisioning artdaq 41659 41529 0 16:51 ? 00:00:00 httpd -f /mnt/work/container/httpd/config/conf/httpd.conf artdaq 41662 41659 0 16:51 ? 00:00:00 httpd -f /mnt/work/container/httpd/config/conf/httpd.conf artdaq 41663 41659 0 16:51 ? 00:00:00 httpd -f /mnt/work/container/httpd/config/conf/httpd.conf artdaq 41665 41659 0 16:51 ? 00:00:00 httpd -f /mnt/work/container/httpd/config/conf/httpd.conf artdaq 41666 41659 0 16:51 ? 00:00:00 httpd -f /mnt/work/container/httpd/config/conf/httpd.conf artdaq 41735 41659 0 16:51 ? 00:00:00 httpd -f /mnt/work/container/httpd/config/conf/httpd.conf artdaq 41736 41659 0 16:51 ? 00:00:00 httpd -f /mnt/work/container/httpd/config/conf/httpd.conf artdaq 41742 41659 0 16:51 ? 00:00:00 httpd -f /mnt/work/container/httpd/config/conf/httpd.conf artdaq 41743 41659 0 16:51 ? 00:00:00 httpd -f /mnt/work/container/httpd/config/conf/httpd.conf artdaq 41744 41659 0 16:51 ? 00:00:00 httpd -f /mnt/work/container/httpd/config/conf/httpd.conf artdaq 41745 41659 0 16:51 ? 00:00:00 httpd -f /mnt/work/container/httpd/config/conf/httpd.conf root 42280 42147 0 16:58 pts/0 00:00:00 grep --color=auto -E (python2|httpd|grafana)