Project

General

Profile

Instructions for Extracting Data from the Ganglia Database

The database used by Ganglia is a Round-Robin Database (RRD) supported
by a 3rd party package RRDtool.

The structure of the database is hierarchical, with the root for the database visible for NDOS from novadaq-ctrl-datamon.fnal.gov
as /var/lib/ganglia/rrds and the hierarchical structure being cluster/nodename after that.

For example, the databases storing monitored information for the dcm-06 node is stored in the directory:
/var/lib/ganglia/rrds/DCM/dcm-06.fnal.gov
and each monitored metric has its own file within that directory. For example, MilliSlice rates are stored as:
'/var/lib/ganglia/rrds/DCM/dcm-06.fnal.gov/MilliSlice Rate.rrd'

The ganglia web displays are produced from the data in this database. The data can also be accessed directly using a command
line application "rrdtool". A help site for rrdtool can be found here:

http://oss.oetiker.ch/rrdtool/doc/index.en.html

rrdtool -h
gives a list of the rrdtool commands, and:

rrdtool command
gives help on specific rrdtool command options.
For example, "rrdtool fetch" gives help on the fetch command.

There are two rrdtool commands especially useful for getting at data stored in the database:
  • "rrdtool fetch" can be used to extract data from the database to be printed at the console.
  • "rrdtool graph" can be used to create a graph from data in the database.
rrdtool fetch
Examples:
  • rrdtool fetch -s -3600 '/ganglia_database/rrds/DCM/dcm-06.fnal.gov/MilliSlice Rate.rrd' AVERAGE
    at NDOS, and:
  • rrdtool fetch -s -3600 '/var/lib/ganglia/rrds/DCM/dcm-02.fnal.gov/MilliSlice Rate.rrd' AVERAGE
    on the test stand.

This command will print the most recent hour of data (-s -3600 => start data fetch minus 3600 seconds from Now)
from the "MilliSlice Rate" table compiled from dcm-06. The data is printed in two columns, the first
is "seconds since RRD epoch (1970-Jan-01 00:00:00)", and the second column contains the MilliSlice Rate.

rrdtool graph
Examples:
  • rrdtool graph millirate.png -s -3600 DEF:millirate='/ganglia_database/rrds/DCM/dcm-06.fnal.gov/MilliSlice Rate.rrd':sum:AVERAGE LINE2:millirate#FF0000
    at NDOS, and:
  • rrdtool graph millirate.png -s -3600 DEF:millirate='/var/lib/ganglia/rrds/DCM/dcm-02.fnal.gov/MilliSlice Rate.rrd':sum:AVERAGE LINE2:millirate#FF0000
    on the test stand.

This command will generate a graph for the most recent hour of MilliSlice Rate data from dcm-06 and produce a png file "millirate.png" containing a red (#FF0000) line graph as a result.