Bug #24320: Run Number metric times out and reports 0 - artdaq - Fermilab Redmine

Bug #24320

Run Number metric times out and reports 0

Added by Eric Flumerfelt 9 months ago. Updated 6 months ago.

Status:

Closed

Priority:

Normal

Assignee:

Category:

Target version:

artdaq v3_09_00

Start date:

04/15/2020

Due date:

% Done:

Estimated time:

Experiment:

Co-Assignees:

Duration:

Description

There are some metrics which are sent very infrequently, and which should persist for the length of a run. We may want to add a new mode flag to indicate persistent metrics.

History

#1 Updated by Eric Flumerfelt 9 months ago

This can currently be worked around by setting "send_zeros" to false in the metric configuration, but this also inhibits the reset of all metric values to 0 at the end of the run.

#2 Updated by Eric Flumerfelt 9 months ago

Status changed from New to Resolved

Added MetricMode::Persist flag in artdaq-utilities:bugfix/24320_PersistentMetrics, used in artdaq:bugfix/24320_RunNumberMetric_Persist.

#3 Updated by Ron Rechenmacher 9 months ago

I see before the changes:
artdaq_utilities:bugfix/24320_PersistentMetrics
artdaq:bugfix/24320_RunNumberMetric_Persist

/home/ron/work/artdaqPrj/demo1-develop/daqlogs/metrics/eventbuilder
ron@mu2edaq13 :^) grep -n 'Run Number' *
evb_30180_metrics.log:9:Mon Apr 27 16:10:21 2020: FileMetric: EventBuilder1.Run Number: 1 Run.
evb_30180_metrics.log:46:Mon Apr 27 16:10:36 2020: FileMetric: EventBuilder1.Run Number: 0 Run.
evb_30180_metrics.log:83:Mon Apr 27 16:10:51 2020: FileMetric: EventBuilder1.Run Number: 0 Run.
evb_30180_metrics.log:120:Mon Apr 27 16:11:06 2020: FileMetric: EventBuilder1.Run Number: 0 Run.
evb_30180_metrics.log:157:Mon Apr 27 16:11:21 2020: FileMetric: EventBuilder1.Run Number: 0 Run.
evb_30180_metrics.log:194:Mon Apr 27 16:11:36 2020: FileMetric: EventBuilder1.Run Number: 0 Run.
evb_30180_metrics.log:231:Mon Apr 27 16:11:52 2020: FileMetric: EventBuilder1.Run Number: 0 Run.
evb_30180_metrics.log:268:Mon Apr 27 16:12:07 2020: FileMetric: EventBuilder1.Run Number: 0 Run.
evb_30180_metrics.log:305:Mon Apr 27 16:12:22 2020: FileMetric: EventBuilder1.Run Number: 0 Run.
evb_30180_metrics.log:342:Mon Apr 27 16:12:37 2020: FileMetric: EventBuilder1.Run Number: 0 Run.
evb_30180_metrics.log:379:Mon Apr 27 16:12:52 2020: FileMetric: EventBuilder1.Run Number: 0 Run.
evb_30180_metrics.log:416:Mon Apr 27 16:13:07 2020: FileMetric: EventBuilder1.Run Number: 0 Run.
evb_30180_metrics.log:453:Mon Apr 27 16:13:22 2020: FileMetric: EventBuilder1.Run Number: 0 Run.
evb_30180_metrics.log:490:Mon Apr 27 16:13:37 2020: FileMetric: EventBuilder1.Run Number: 0 Run.
evb_30180_metrics.log:537:Mon Apr 27 16:13:53 2020: FileMetric: EventBuilder1.Run Number: 0 Run.
evb_30181_metrics.log:9:Mon Apr 27 16:10:21 2020: FileMetric: EventBuilder2.Run Number: 1 Run.
evb_30181_metrics.log:46:Mon Apr 27 16:10:36 2020: FileMetric: EventBuilder2.Run Number: 0 Run.
evb_30181_metrics.log:83:Mon Apr 27 16:10:51 2020: FileMetric: EventBuilder2.Run Number: 0 Run.
evb_30181_metrics.log:120:Mon Apr 27 16:11:06 2020: FileMetric: EventBuilder2.Run Number: 0 Run.
evb_30181_metrics.log:157:Mon Apr 27 16:11:21 2020: FileMetric: EventBuilder2.Run Number: 0 Run.
evb_30181_metrics.log:194:Mon Apr 27 16:11:36 2020: FileMetric: EventBuilder2.Run Number: 0 Run.
evb_30181_metrics.log:231:Mon Apr 27 16:11:52 2020: FileMetric: EventBuilder2.Run Number: 0 Run.
evb_30181_metrics.log:268:Mon Apr 27 16:12:07 2020: FileMetric: EventBuilder2.Run Number: 0 Run.
evb_30181_metrics.log:305:Mon Apr 27 16:12:22 2020: FileMetric: EventBuilder2.Run Number: 0 Run.
evb_30181_metrics.log:342:Mon Apr 27 16:12:37 2020: FileMetric: EventBuilder2.Run Number: 0 Run.
evb_30181_metrics.log:379:Mon Apr 27 16:12:52 2020: FileMetric: EventBuilder2.Run Number: 0 Run.
evb_30181_metrics.log:416:Mon Apr 27 16:13:07 2020: FileMetric: EventBuilder2.Run Number: 0 Run.
evb_30181_metrics.log:453:Mon Apr 27 16:13:22 2020: FileMetric: EventBuilder2.Run Number: 0 Run.
evb_30181_metrics.log:490:Mon Apr 27 16:13:37 2020: FileMetric: EventBuilder2.Run Number: 0 Run.
evb_30181_metrics.log:546:Mon Apr 27 16:13:53 2020: FileMetric: EventBuilder2.Run Number: 0 Run.
evb_59234_metrics.log:9:Mon Apr 27 16:22:42 2020: FileMetric: EventBuilder1.Run Number: 2 Run.
evb_59234_metrics.log:46:Mon Apr 27 16:22:57 2020: FileMetric: EventBuilder1.Run Number: 0 Run.
evb_59234_metrics.log:83:Mon Apr 27 16:23:12 2020: FileMetric: EventBuilder1.Run Number: 0 Run.
evb_59234_metrics.log:120:Mon Apr 27 16:23:27 2020: FileMetric: EventBuilder1.Run Number: 0 Run.
evb_59234_metrics.log:157:Mon Apr 27 16:23:42 2020: FileMetric: EventBuilder1.Run Number: 0 Run.
evb_59234_metrics.log:194:Mon Apr 27 16:23:57 2020: FileMetric: EventBuilder1.Run Number: 0 Run.
evb_59234_metrics.log:231:Mon Apr 27 16:24:12 2020: FileMetric: EventBuilder1.Run Number: 0 Run.
evb_59234_metrics.log:268:Mon Apr 27 16:24:27 2020: FileMetric: EventBuilder1.Run Number: 0 Run.
evb_59234_metrics.log:305:Mon Apr 27 16:24:42 2020: FileMetric: EventBuilder1.Run Number: 0 Run.
evb_59234_metrics.log:342:Mon Apr 27 16:24:57 2020: FileMetric: EventBuilder1.Run Number: 0 Run.
evb_59234_metrics.log:379:Mon Apr 27 16:25:12 2020: FileMetric: EventBuilder1.Run Number: 0 Run.
evb_59235_metrics.log:9:Mon Apr 27 16:22:42 2020: FileMetric: EventBuilder2.Run Number: 2 Run.
evb_59235_metrics.log:46:Mon Apr 27 16:22:57 2020: FileMetric: EventBuilder2.Run Number: 0 Run.
evb_59235_metrics.log:83:Mon Apr 27 16:23:12 2020: FileMetric: EventBuilder2.Run Number: 0 Run.
evb_59235_metrics.log:120:Mon Apr 27 16:23:27 2020: FileMetric: EventBuilder2.Run Number: 0 Run.
evb_59235_metrics.log:157:Mon Apr 27 16:23:42 2020: FileMetric: EventBuilder2.Run Number: 0 Run.
evb_59235_metrics.log:194:Mon Apr 27 16:23:57 2020: FileMetric: EventBuilder2.Run Number: 0 Run.
evb_59235_metrics.log:231:Mon Apr 27 16:24:12 2020: FileMetric: EventBuilder2.Run Number: 0 Run.
evb_59235_metrics.log:268:Mon Apr 27 16:24:27 2020: FileMetric: EventBuilder2.Run Number: 0 Run.
evb_59235_metrics.log:305:Mon Apr 27 16:24:42 2020: FileMetric: EventBuilder2.Run Number: 0 Run.
evb_59235_metrics.log:342:Mon Apr 27 16:24:57 2020: FileMetric: EventBuilder2.Run Number: 0 Run.
evb_59235_metrics.log:379:Mon Apr 27 16:25:12 2020: FileMetric: EventBuilder2.Run Number: 0 Run.
--2020-04-27_16:25:26--

After the change:

/home/ron/work/artdaqPrj/demo1-develop/daqlogs/metrics/eventbuilder
ron@mu2edaq13 :^) grep -n 'Run Number' evb_9*
evb_97810_metrics.log:9:Mon Apr 27 16:40:36 2020: FileMetric: EventBuilder1.Run Number - Last: 3 Run.
evb_97810_metrics.log:46:Mon Apr 27 16:40:51 2020: FileMetric: EventBuilder1.Run Number - Last: 3 Run.
evb_97810_metrics.log:83:Mon Apr 27 16:41:06 2020: FileMetric: EventBuilder1.Run Number - Last: 3 Run.
evb_97810_metrics.log:120:Mon Apr 27 16:41:21 2020: FileMetric: EventBuilder1.Run Number - Last: 3 Run.
evb_97810_metrics.log:157:Mon Apr 27 16:41:36 2020: FileMetric: EventBuilder1.Run Number - Last: 3 Run.
evb_97810_metrics.log:194:Mon Apr 27 16:41:51 2020: FileMetric: EventBuilder1.Run Number - Last: 3 Run.
evb_97810_metrics.log:231:Mon Apr 27 16:42:06 2020: FileMetric: EventBuilder1.Run Number - Last: 3 Run.
evb_97810_metrics.log:268:Mon Apr 27 16:42:21 2020: FileMetric: EventBuilder1.Run Number - Last: 3 Run.
evb_97810_metrics.log:305:Mon Apr 27 16:42:36 2020: FileMetric: EventBuilder1.Run Number - Last: 3 Run.
evb_97810_metrics.log:342:Mon Apr 27 16:42:51 2020: FileMetric: EventBuilder1.Run Number - Last: 3 Run.
evb_97810_metrics.log:379:Mon Apr 27 16:43:06 2020: FileMetric: EventBuilder1.Run Number - Last: 3 Run.
evb_97810_metrics.log:416:Mon Apr 27 16:43:21 2020: FileMetric: EventBuilder1.Run Number - Last: 3 Run.
evb_97810_metrics.log:453:Mon Apr 27 16:43:36 2020: FileMetric: EventBuilder1.Run Number - Last: 3 Run.
evb_97810_metrics.log:490:Mon Apr 27 16:43:51 2020: FileMetric: EventBuilder1.Run Number - Last: 3 Run.
evb_97810_metrics.log:540:Mon Apr 27 16:44:08 2020: FileMetric: EventBuilder1.Run Number - Last: 3 Run.
evb_97811_metrics.log:9:Mon Apr 27 16:40:36 2020: FileMetric: EventBuilder2.Run Number - Last: 3 Run.
evb_97811_metrics.log:46:Mon Apr 27 16:40:51 2020: FileMetric: EventBuilder2.Run Number - Last: 3 Run.
evb_97811_metrics.log:83:Mon Apr 27 16:41:06 2020: FileMetric: EventBuilder2.Run Number - Last: 3 Run.
evb_97811_metrics.log:120:Mon Apr 27 16:41:21 2020: FileMetric: EventBuilder2.Run Number - Last: 3 Run.
evb_97811_metrics.log:157:Mon Apr 27 16:41:36 2020: FileMetric: EventBuilder2.Run Number - Last: 3 Run.
evb_97811_metrics.log:194:Mon Apr 27 16:41:51 2020: FileMetric: EventBuilder2.Run Number - Last: 3 Run.
evb_97811_metrics.log:231:Mon Apr 27 16:42:06 2020: FileMetric: EventBuilder2.Run Number - Last: 3 Run.
evb_97811_metrics.log:268:Mon Apr 27 16:42:21 2020: FileMetric: EventBuilder2.Run Number - Last: 3 Run.
evb_97811_metrics.log:305:Mon Apr 27 16:42:36 2020: FileMetric: EventBuilder2.Run Number - Last: 3 Run.
evb_97811_metrics.log:342:Mon Apr 27 16:42:51 2020: FileMetric: EventBuilder2.Run Number - Last: 3 Run.
evb_97811_metrics.log:379:Mon Apr 27 16:43:06 2020: FileMetric: EventBuilder2.Run Number - Last: 3 Run.
evb_97811_metrics.log:416:Mon Apr 27 16:43:21 2020: FileMetric: EventBuilder2.Run Number - Last: 3 Run.
evb_97811_metrics.log:453:Mon Apr 27 16:43:36 2020: FileMetric: EventBuilder2.Run Number - Last: 3 Run.
evb_97811_metrics.log:490:Mon Apr 27 16:43:51 2020: FileMetric: EventBuilder2.Run Number - Last: 3 Run.
evb_97811_metrics.log:546:Mon Apr 27 16:44:08 2020: FileMetric: EventBuilder2.Run Number - Last: 3 Run.
--2020-04-27_16:44:52--

Additionally, after implementing a pause in the ToyHardwareInterface and a 33 second pause every 35 seconds,
I get:

/home/ron/work/artdaqPrj/demo1-develop/srcs/artdaq_demo
ron@mu2edaq13 :^) tshow | egrep 'pausing|resuming' | tdelta -d 1 -ct 1
  597 04-28 06:03:18.452095           0 277645 284240  42                    ToyHardwareInterface d16 . resuming after pause of 33 seconds
  609 04-28 06:03:18.450885        1210 277646 284238  32                    ToyHardwareInterface d16 . resuming after pause of 33 seconds
  620 04-28 06:02:45.451686    32999199 277645 284240  42                    ToyHardwareInterface d16 . pausing 33 seconds
  621 04-28 06:02:45.450629        1057 277646 284238  32                    ToyHardwareInterface d16 . pausing 33 seconds
 1203 04-28 06:02:43.447400     2003229 277645 284240  42                    ToyHardwareInterface d16 . resuming after pause of 33 seconds
 1205 04-28 06:02:43.446985         415 277646 284238  32                    ToyHardwareInterface d16 . resuming after pause of 33 seconds
 1210 04-28 06:02:10.447023    32999962 277645 284240  42                    ToyHardwareInterface d16 . pausing 33 seconds
 1211 04-28 06:02:10.446730         293 277646 284238  32                    ToyHardwareInterface d16 . pausing 33 seconds
 1793 04-28 06:02:08.443313     2003417 277646 284238  32                    ToyHardwareInterface d16 . resuming after pause of 33 seconds
 1795 04-28 06:02:08.442881         432 277645 284240  42                    ToyHardwareInterface d16 . resuming after pause of 33 seconds
 1800 04-28 06:01:35.443080    32999801 277646 284238  32                    ToyHardwareInterface d16 . pausing 33 seconds
 1801 04-28 06:01:35.442507         573 277645 284240  42                    ToyHardwareInterface d16 . pausing 33 seconds
 2384 04-28 06:01:33.439609     2002898 277646 284238  32                    ToyHardwareInterface d16 . resuming after pause of 33 seconds
 2385 04-28 06:01:33.439549          60 277645 284240  42                    ToyHardwareInterface d16 . resuming after pause of 33 seconds
 2390 04-28 06:01:00.439339    33000210 277646 284238  32                    ToyHardwareInterface d16 . pausing 33 seconds
 2391 04-28 06:01:00.439201         138 277645 284240  42                    ToyHardwareInterface d16 . pausing 33 seconds
 2973 04-28 06:00:58.435643     2003558 277646 284238  32                    ToyHardwareInterface d16 . resuming after pause of 33 seconds
 2975 04-28 06:00:58.435200         443 277645 284240  42                    ToyHardwareInterface d16 . resuming after pause of 33 seconds
 2980 04-28 06:00:25.435251    32999949 277646 284238  32                    ToyHardwareInterface d16 . pausing 33 seconds
 2981 04-28 06:00:25.434714         537 277645 284240  42                    ToyHardwareInterface d16 . pausing 33 seconds
--2020-04-28_08:33:42--

/home/ron/work/artdaqPrj/demo1-develop/daqlogs/metrics/eventbuilder
ron@mu2edaq13 :^) grep 'Average Event Size' evb_27764[78]_metrics.log
evb_277647_metrics.log:Tue Apr 28 05:59:50 2020: FileMetric: EventBuilder1.Average Event Size: 288.000000 Bytes.
evb_277647_metrics.log:Tue Apr 28 06:00:05 2020: FileMetric: EventBuilder1.Average Event Size: 288.000000 Bytes.
evb_277647_metrics.log:Tue Apr 28 06:00:20 2020: FileMetric: EventBuilder1.Average Event Size: 288.000000 Bytes.
evb_277647_metrics.log:Tue Apr 28 06:00:35 2020: FileMetric: EventBuilder1.Average Event Size: 288.000000 Bytes.
evb_277647_metrics.log:Tue Apr 28 06:00:50 2020: FileMetric: EventBuilder1.Average Event Size: 0.000000 Bytes.
evb_277647_metrics.log:Tue Apr 28 06:01:05 2020: FileMetric: EventBuilder1.Average Event Size: 288.000000 Bytes.
evb_277647_metrics.log:Tue Apr 28 06:01:20 2020: FileMetric: EventBuilder1.Average Event Size: 0.000000 Bytes.
evb_277647_metrics.log:Tue Apr 28 06:01:35 2020: FileMetric: EventBuilder1.Average Event Size: 288.000000 Bytes.
evb_277647_metrics.log:Tue Apr 28 06:01:50 2020: FileMetric: EventBuilder1.Average Event Size: 0.000000 Bytes.
evb_277647_metrics.log:Tue Apr 28 06:02:05 2020: FileMetric: EventBuilder1.Average Event Size: 0.000000 Bytes.
evb_277647_metrics.log:Tue Apr 28 06:02:20 2020: FileMetric: EventBuilder1.Average Event Size: 288.000000 Bytes.
evb_277647_metrics.log:Tue Apr 28 06:02:36 2020: FileMetric: EventBuilder1.Average Event Size: 0.000000 Bytes.
evb_277647_metrics.log:Tue Apr 28 06:02:51 2020: FileMetric: EventBuilder1.Average Event Size: 288.000000 Bytes.
evb_277647_metrics.log:Tue Apr 28 06:03:06 2020: FileMetric: EventBuilder1.Average Event Size: 0.000000 Bytes.
evb_277647_metrics.log:Tue Apr 28 06:03:27 2020: FileMetric: EventBuilder1.Average Event Size: 0.000000 Bytes.
evb_277648_metrics.log:Tue Apr 28 05:59:50 2020: FileMetric: EventBuilder2.Average Event Size: 288.000000 Bytes.
evb_277648_metrics.log:Tue Apr 28 06:00:05 2020: FileMetric: EventBuilder2.Average Event Size: 288.000000 Bytes.
evb_277648_metrics.log:Tue Apr 28 06:00:20 2020: FileMetric: EventBuilder2.Average Event Size: 288.000000 Bytes.
evb_277648_metrics.log:Tue Apr 28 06:00:35 2020: FileMetric: EventBuilder2.Average Event Size: 288.000000 Bytes.
evb_277648_metrics.log:Tue Apr 28 06:00:50 2020: FileMetric: EventBuilder2.Average Event Size: 0.000000 Bytes.
evb_277648_metrics.log:Tue Apr 28 06:01:05 2020: FileMetric: EventBuilder2.Average Event Size: 288.000000 Bytes.
evb_277648_metrics.log:Tue Apr 28 06:01:20 2020: FileMetric: EventBuilder2.Average Event Size: 0.000000 Bytes.
evb_277648_metrics.log:Tue Apr 28 06:01:35 2020: FileMetric: EventBuilder2.Average Event Size: 288.000000 Bytes.
evb_277648_metrics.log:Tue Apr 28 06:01:50 2020: FileMetric: EventBuilder2.Average Event Size: 0.000000 Bytes.
evb_277648_metrics.log:Tue Apr 28 06:02:05 2020: FileMetric: EventBuilder2.Average Event Size: 0.000000 Bytes.
evb_277648_metrics.log:Tue Apr 28 06:02:20 2020: FileMetric: EventBuilder2.Average Event Size: 288.000000 Bytes.
evb_277648_metrics.log:Tue Apr 28 06:02:35 2020: FileMetric: EventBuilder2.Average Event Size: 0.000000 Bytes.
evb_277648_metrics.log:Tue Apr 28 06:02:51 2020: FileMetric: EventBuilder2.Average Event Size: 288.000000 Bytes.
evb_277648_metrics.log:Tue Apr 28 06:03:06 2020: FileMetric: EventBuilder2.Average Event Size: 0.000000 Bytes.
evb_277648_metrics.log:Tue Apr 28 06:03:27 2020: FileMetric: EventBuilder2.Average Event Size: 288.000000 Bytes.
--2020-04-28_06:10:41--

This all seems good.

#4 Updated by Ron Rechenmacher 9 months ago

Status changed from Resolved to Reviewed

merged branches into develop; change status to reviewed

#5 Updated by Eric Flumerfelt 6 months ago

Target version set to artdaq v3_09_00
Status changed from Reviewed to Closed
Project changed from artdaq Utilities to artdaq

Also available in: Atom PDF

Project

General

Profile

artdaq

Issues

Custom queries