Project

General

Profile

Metric Management

The sbndqm repository provides analysis modules with the facility to average metrics over a set of time and send them to a database. An example of analyzer using these facilities is at sbndqm/dqmAnalysis/example. A step-by-step walkthrough is also included here.

Configuring a Redis Connection

All metric data is stored in a Redis database. In order to use metrics, you must setup a connection to a redis database. The redis connection is handled as an art service. To setup the redis connection with default configuration, include the file redis_connection.fcl at the top of your fhicl file. Then, in the services block, include:

services: {
  RedisConnection: @local::redis_connection
}

This will setup a connection to redis on localhost on port 6379 without using a password. These settings are over-writeable. Use “host” to change the hostname, “port” to change the port number, and “password” to specify a password. It is also possible to specify a password located in some file. To do this, specify passfile: </path/to/passfile>.

Configuring up the Metric Manager

To start out, in your fhicl file for the EdAnalyzer you should have a block which configures the metric manager. Example of such a block is included below:


      metrics: {
        redis_metric_fast: { // can be named anything
          level: 3
          metricPluginType: "redis" 
          // period in seconds to send metric to database
          reporting_interval: 2.0
          // need to add postfix otherwise this metric sender and the other one will collide
          redis_key_postfix: ":fast" 
        }
        redis_metric_slow: { // can be named anything
          level: 3
          metricPluginType: "redis" 
          reporting_interval: 10.0
          // need to add postfix otherwise this metric sender and the other one will collide
          redis_key_postfix: ":slow" 
        }
      }

This configuration block sets up two instances of a Metric Manager. Each time you send a metric (as will be shown below) it will be sent to both managers. Both metric managers here will connect to a redis database. One will average metrics for 2 seconds in between sending values, and the other will average for 10 seconds. The first metric manager will append the string “:fast” to each metric, while the second one will append “:slow”. Note that this is necessary so that the names of each metric manager don’t collide in the database.

Additional Configuration

There are a few other parameters (not shown above) which can configure the metric manager. For each stream, only a set number of values are stored at a time in the database. By default, this number is set to 1000. You can specify this number with the “maxlen” parameter.

Initializing the Metric Manager

The metric manager should be initialized in the constructor of the analyzer. In the case of our metric configuration above, you would call:

sbndaq::InitializeMetricManager(pset.get<fhicl::ParameterSet>("metrics"));

where pset is the name of the input fhicl::ParameterSet.

Sending a Metric

Let’s say you have a metric ready to send -- it could be the RMS of a readout from the wire, or the purity of the detector, or any single value -- then you would call sendMetric(). Let’s say we have a metric in “value”, then we would call:

  // level of importance of metric
  int level = 3;
  // The mode to accumulate the metrics. Here, the manager will report the average of 
  // of all metrics per reporting interval
  artdaq::MetricMode mode = artdaq::MetricMode::Average;
  // The name of the stream in redis. In redis, this metric will be stored as a Stream
  // object with a key of: redis_name + redis_key_postfix, where redis_key_postfix is 
  // as configured in the fhicl file
  const char *name = "example_metric";
  // send a metric 
  sbndaq::sendMetric(name, value, level, mode);

This will store the metric in each configured metric manager. After the reporting_interval configured in each metric manager, the manager will accumulate the metric (in the mode specified in sendMetric) and will store the value in a database. Note that sendMetric does not have to be called once per event. It can be called at any time, any number of times in each event. It is also safe to use sendMetric in a multithreaded context.

Viewing the Metric in Minargon

If you are using a redis database integrated with the online monitoring frontend Minargon (e.g. on sbnd-daq33) then whenever running the metric manager you can view them on the online monitoring website.

In the case of our example here, you would visit: sbn-online.fnal.gov/cgi-bin/minargon/minargon.wsgi/single_stream/example_metric:slow or sbn-online.fnal.gov/cgi-bin/minargon/minargon.wsgi/single_stream/example_metric:fast. For a general metric that you write yourself, visit: sbn-online.fnal.gov/cgi-bin/minargon/minargon.wsgi/single_stream/<stream_name>.

Note that this website is only accessible behind the fermilab VPN. Also note that the website will not work on some older browsers, including internet explorer.

Groups of Metrics

In many cases, an online monitoring process will have a set of metrics that can be logically grouped together. For example, this grouping could be the values of the RMS of the ADC values for a group of readout channels. The metric manager setup in sbndqm provides facilities to organize these groups of metrics and display them in minargon. There is also an interface to configure how groups of metrics will be displayed in the front end.

Configuring a Group of Metrics

The fhicl configuration here builds on our earlier “metric” configuration. In this case, we are setting values like the names of metrics, the display range, the names of units, and the range of a metric value that would cause an alarm. Here is an example configuration for the RMS of a set of 10 example readout channels, named 1 to 10:


      metric_config: {
        // Each group name, mapping to the list of instances for this group
        // The list of instances are the set of instances that you send metrics
        // to in this group. To make the config more concise, if using instance
        // names which are sequential numbers you can also provide a pair
        // of [lo, hi) and it will count as instance names lo,lo+1,...,hi-1
        groups: {
          example: [ [0, 10] ]
          // could also be
          // example: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
          // or
          // example: [[0,8], 8, 9]
        }

        // The names of the streams __must__ be the names of the redis_key_postfix's
        // for each instance in 'metrics' with the initial colon (':') stripped
        streams: [slow, fast]

        metrics: {
          rms: {
            // NOTE: all configs for each metric are optional

            // default range over which to display metric
            display_range: [0., 10.]
            // if the metric leaves this range, issue a warning
            warning_range: [1., 5.]
            // if the metric leaves this range, issue an alarm
            alarm_range: [0.5, 7.5]
            // units of metric
            units: ADC
            // title of the metric to be shown in displays

            // The title of the metric will be formatted in python
            // passed the parameters "group" and "instance" as strings
            // So, or this title, given the group "example" and instance
            // "1", the title would be: "example Wire 1 RMS" 
            titie: "%(group)s Wire %(instance)s RMS" 
          }
        }
      }

This configuration sets up a new metric “group” -- in this case named “example”. The group has a set of instances, in this case the 10 readout channels, named “1”, 2”, ..., ”10”. It also sets up a number of metrics, which will be kept track of for each instance in this group. In this case we just set up one metric -- the “RMS”.

Initializing the Group Metric Config

The group metric config should be initialized in the constructor on the Analyzer. For our configuration, call:

  sbndaq::GenerateMetricConfig(pset.get<fhicl::ParameterSet>("metric_config"));

where pset is the name of the passed in fhicl::ParameterSet. Note that calling “GenerateMetricConfig” is idempotent -- calling it multiple times with the same configuration in a single process (or across running the process multiple times) will have no additional effect after the first time.

Sending a Group of Metrics

To now send metrics in a group, you will use a slightly modified version of the sendMetric interface from earlier. Here is an example:

  // send metrics for a number of "wires" 

  // level and aggregation mode
  int level = 3;
  artdaq::MetricMode mode = artdaq::MetricMode::Average;

  // name of the metric -- should match the name we assigned in metric_config
  const char *metric_name = "rms";
  // name of the group -- should match the name we assigned in metric_config
  const char *group_name = "example";

  // make metrics for 10 "wires" 
  for (unsigned i = 0; i < 10; i++) {
    // name of this instance
    std::string instance = std::to_string(i);
    // metric value
    double value = (double) i;
    sbndaq::sendMetric(group_name, instance, metric_name, value, level, mode);
  }

In this case, we have just set the RMS of each wire to be the same as the wire index. In a real analysis context, you would set the RMS to whatever value you computed.

Viewing the Group of Metrics in Minargon

If you are using a redis database integrated with the online monitoring frontend Minargon (e.g. on sbnd-daq33) then whenever running the metric manager you can view them on the online monitoring website.

In the case of our example here, you would visit: sbn-online.fnal.gov/cgi-bin/minargon/minargon.wsgi/online_group/example. For a general group of metrics, you would visit: sbn-online.fnal.gov/cgi-bin/minargon/minargon.wsgi/online_group/<group_name>.

Note that this website is only accessible behind the fermilab VPN. Also note that the website will not work on some older browsers, including internet explorer.