Bug #24320
Run Number metric times out and reports 0
0%
Description
There are some metrics which are sent very infrequently, and which should persist for the length of a run. We may want to add a new mode flag to indicate persistent metrics.
History
#1 Updated by Eric Flumerfelt 9 months ago
This can currently be worked around by setting "send_zeros" to false in the metric configuration, but this also inhibits the reset of all metric values to 0 at the end of the run.
#2 Updated by Eric Flumerfelt 9 months ago
- Status changed from New to Resolved
Added MetricMode::Persist flag in artdaq-utilities:bugfix/24320_PersistentMetrics, used in artdaq:bugfix/24320_RunNumberMetric_Persist.
#3 Updated by Ron Rechenmacher 9 months ago
I see before the changes:
artdaq_utilities:bugfix/24320_PersistentMetrics
artdaq:bugfix/24320_RunNumberMetric_Persist
/home/ron/work/artdaqPrj/demo1-develop/daqlogs/metrics/eventbuilder ron@mu2edaq13 :^) grep -n 'Run Number' * evb_30180_metrics.log:9:Mon Apr 27 16:10:21 2020: FileMetric: EventBuilder1.Run Number: 1 Run. evb_30180_metrics.log:46:Mon Apr 27 16:10:36 2020: FileMetric: EventBuilder1.Run Number: 0 Run. evb_30180_metrics.log:83:Mon Apr 27 16:10:51 2020: FileMetric: EventBuilder1.Run Number: 0 Run. evb_30180_metrics.log:120:Mon Apr 27 16:11:06 2020: FileMetric: EventBuilder1.Run Number: 0 Run. evb_30180_metrics.log:157:Mon Apr 27 16:11:21 2020: FileMetric: EventBuilder1.Run Number: 0 Run. evb_30180_metrics.log:194:Mon Apr 27 16:11:36 2020: FileMetric: EventBuilder1.Run Number: 0 Run. evb_30180_metrics.log:231:Mon Apr 27 16:11:52 2020: FileMetric: EventBuilder1.Run Number: 0 Run. evb_30180_metrics.log:268:Mon Apr 27 16:12:07 2020: FileMetric: EventBuilder1.Run Number: 0 Run. evb_30180_metrics.log:305:Mon Apr 27 16:12:22 2020: FileMetric: EventBuilder1.Run Number: 0 Run. evb_30180_metrics.log:342:Mon Apr 27 16:12:37 2020: FileMetric: EventBuilder1.Run Number: 0 Run. evb_30180_metrics.log:379:Mon Apr 27 16:12:52 2020: FileMetric: EventBuilder1.Run Number: 0 Run. evb_30180_metrics.log:416:Mon Apr 27 16:13:07 2020: FileMetric: EventBuilder1.Run Number: 0 Run. evb_30180_metrics.log:453:Mon Apr 27 16:13:22 2020: FileMetric: EventBuilder1.Run Number: 0 Run. evb_30180_metrics.log:490:Mon Apr 27 16:13:37 2020: FileMetric: EventBuilder1.Run Number: 0 Run. evb_30180_metrics.log:537:Mon Apr 27 16:13:53 2020: FileMetric: EventBuilder1.Run Number: 0 Run. evb_30181_metrics.log:9:Mon Apr 27 16:10:21 2020: FileMetric: EventBuilder2.Run Number: 1 Run. evb_30181_metrics.log:46:Mon Apr 27 16:10:36 2020: FileMetric: EventBuilder2.Run Number: 0 Run. evb_30181_metrics.log:83:Mon Apr 27 16:10:51 2020: FileMetric: EventBuilder2.Run Number: 0 Run. evb_30181_metrics.log:120:Mon Apr 27 16:11:06 2020: FileMetric: EventBuilder2.Run Number: 0 Run. evb_30181_metrics.log:157:Mon Apr 27 16:11:21 2020: FileMetric: EventBuilder2.Run Number: 0 Run. evb_30181_metrics.log:194:Mon Apr 27 16:11:36 2020: FileMetric: EventBuilder2.Run Number: 0 Run. evb_30181_metrics.log:231:Mon Apr 27 16:11:52 2020: FileMetric: EventBuilder2.Run Number: 0 Run. evb_30181_metrics.log:268:Mon Apr 27 16:12:07 2020: FileMetric: EventBuilder2.Run Number: 0 Run. evb_30181_metrics.log:305:Mon Apr 27 16:12:22 2020: FileMetric: EventBuilder2.Run Number: 0 Run. evb_30181_metrics.log:342:Mon Apr 27 16:12:37 2020: FileMetric: EventBuilder2.Run Number: 0 Run. evb_30181_metrics.log:379:Mon Apr 27 16:12:52 2020: FileMetric: EventBuilder2.Run Number: 0 Run. evb_30181_metrics.log:416:Mon Apr 27 16:13:07 2020: FileMetric: EventBuilder2.Run Number: 0 Run. evb_30181_metrics.log:453:Mon Apr 27 16:13:22 2020: FileMetric: EventBuilder2.Run Number: 0 Run. evb_30181_metrics.log:490:Mon Apr 27 16:13:37 2020: FileMetric: EventBuilder2.Run Number: 0 Run. evb_30181_metrics.log:546:Mon Apr 27 16:13:53 2020: FileMetric: EventBuilder2.Run Number: 0 Run. evb_59234_metrics.log:9:Mon Apr 27 16:22:42 2020: FileMetric: EventBuilder1.Run Number: 2 Run. evb_59234_metrics.log:46:Mon Apr 27 16:22:57 2020: FileMetric: EventBuilder1.Run Number: 0 Run. evb_59234_metrics.log:83:Mon Apr 27 16:23:12 2020: FileMetric: EventBuilder1.Run Number: 0 Run. evb_59234_metrics.log:120:Mon Apr 27 16:23:27 2020: FileMetric: EventBuilder1.Run Number: 0 Run. evb_59234_metrics.log:157:Mon Apr 27 16:23:42 2020: FileMetric: EventBuilder1.Run Number: 0 Run. evb_59234_metrics.log:194:Mon Apr 27 16:23:57 2020: FileMetric: EventBuilder1.Run Number: 0 Run. evb_59234_metrics.log:231:Mon Apr 27 16:24:12 2020: FileMetric: EventBuilder1.Run Number: 0 Run. evb_59234_metrics.log:268:Mon Apr 27 16:24:27 2020: FileMetric: EventBuilder1.Run Number: 0 Run. evb_59234_metrics.log:305:Mon Apr 27 16:24:42 2020: FileMetric: EventBuilder1.Run Number: 0 Run. evb_59234_metrics.log:342:Mon Apr 27 16:24:57 2020: FileMetric: EventBuilder1.Run Number: 0 Run. evb_59234_metrics.log:379:Mon Apr 27 16:25:12 2020: FileMetric: EventBuilder1.Run Number: 0 Run. evb_59235_metrics.log:9:Mon Apr 27 16:22:42 2020: FileMetric: EventBuilder2.Run Number: 2 Run. evb_59235_metrics.log:46:Mon Apr 27 16:22:57 2020: FileMetric: EventBuilder2.Run Number: 0 Run. evb_59235_metrics.log:83:Mon Apr 27 16:23:12 2020: FileMetric: EventBuilder2.Run Number: 0 Run. evb_59235_metrics.log:120:Mon Apr 27 16:23:27 2020: FileMetric: EventBuilder2.Run Number: 0 Run. evb_59235_metrics.log:157:Mon Apr 27 16:23:42 2020: FileMetric: EventBuilder2.Run Number: 0 Run. evb_59235_metrics.log:194:Mon Apr 27 16:23:57 2020: FileMetric: EventBuilder2.Run Number: 0 Run. evb_59235_metrics.log:231:Mon Apr 27 16:24:12 2020: FileMetric: EventBuilder2.Run Number: 0 Run. evb_59235_metrics.log:268:Mon Apr 27 16:24:27 2020: FileMetric: EventBuilder2.Run Number: 0 Run. evb_59235_metrics.log:305:Mon Apr 27 16:24:42 2020: FileMetric: EventBuilder2.Run Number: 0 Run. evb_59235_metrics.log:342:Mon Apr 27 16:24:57 2020: FileMetric: EventBuilder2.Run Number: 0 Run. evb_59235_metrics.log:379:Mon Apr 27 16:25:12 2020: FileMetric: EventBuilder2.Run Number: 0 Run. --2020-04-27_16:25:26--
After the change:
/home/ron/work/artdaqPrj/demo1-develop/daqlogs/metrics/eventbuilder ron@mu2edaq13 :^) grep -n 'Run Number' evb_9* evb_97810_metrics.log:9:Mon Apr 27 16:40:36 2020: FileMetric: EventBuilder1.Run Number - Last: 3 Run. evb_97810_metrics.log:46:Mon Apr 27 16:40:51 2020: FileMetric: EventBuilder1.Run Number - Last: 3 Run. evb_97810_metrics.log:83:Mon Apr 27 16:41:06 2020: FileMetric: EventBuilder1.Run Number - Last: 3 Run. evb_97810_metrics.log:120:Mon Apr 27 16:41:21 2020: FileMetric: EventBuilder1.Run Number - Last: 3 Run. evb_97810_metrics.log:157:Mon Apr 27 16:41:36 2020: FileMetric: EventBuilder1.Run Number - Last: 3 Run. evb_97810_metrics.log:194:Mon Apr 27 16:41:51 2020: FileMetric: EventBuilder1.Run Number - Last: 3 Run. evb_97810_metrics.log:231:Mon Apr 27 16:42:06 2020: FileMetric: EventBuilder1.Run Number - Last: 3 Run. evb_97810_metrics.log:268:Mon Apr 27 16:42:21 2020: FileMetric: EventBuilder1.Run Number - Last: 3 Run. evb_97810_metrics.log:305:Mon Apr 27 16:42:36 2020: FileMetric: EventBuilder1.Run Number - Last: 3 Run. evb_97810_metrics.log:342:Mon Apr 27 16:42:51 2020: FileMetric: EventBuilder1.Run Number - Last: 3 Run. evb_97810_metrics.log:379:Mon Apr 27 16:43:06 2020: FileMetric: EventBuilder1.Run Number - Last: 3 Run. evb_97810_metrics.log:416:Mon Apr 27 16:43:21 2020: FileMetric: EventBuilder1.Run Number - Last: 3 Run. evb_97810_metrics.log:453:Mon Apr 27 16:43:36 2020: FileMetric: EventBuilder1.Run Number - Last: 3 Run. evb_97810_metrics.log:490:Mon Apr 27 16:43:51 2020: FileMetric: EventBuilder1.Run Number - Last: 3 Run. evb_97810_metrics.log:540:Mon Apr 27 16:44:08 2020: FileMetric: EventBuilder1.Run Number - Last: 3 Run. evb_97811_metrics.log:9:Mon Apr 27 16:40:36 2020: FileMetric: EventBuilder2.Run Number - Last: 3 Run. evb_97811_metrics.log:46:Mon Apr 27 16:40:51 2020: FileMetric: EventBuilder2.Run Number - Last: 3 Run. evb_97811_metrics.log:83:Mon Apr 27 16:41:06 2020: FileMetric: EventBuilder2.Run Number - Last: 3 Run. evb_97811_metrics.log:120:Mon Apr 27 16:41:21 2020: FileMetric: EventBuilder2.Run Number - Last: 3 Run. evb_97811_metrics.log:157:Mon Apr 27 16:41:36 2020: FileMetric: EventBuilder2.Run Number - Last: 3 Run. evb_97811_metrics.log:194:Mon Apr 27 16:41:51 2020: FileMetric: EventBuilder2.Run Number - Last: 3 Run. evb_97811_metrics.log:231:Mon Apr 27 16:42:06 2020: FileMetric: EventBuilder2.Run Number - Last: 3 Run. evb_97811_metrics.log:268:Mon Apr 27 16:42:21 2020: FileMetric: EventBuilder2.Run Number - Last: 3 Run. evb_97811_metrics.log:305:Mon Apr 27 16:42:36 2020: FileMetric: EventBuilder2.Run Number - Last: 3 Run. evb_97811_metrics.log:342:Mon Apr 27 16:42:51 2020: FileMetric: EventBuilder2.Run Number - Last: 3 Run. evb_97811_metrics.log:379:Mon Apr 27 16:43:06 2020: FileMetric: EventBuilder2.Run Number - Last: 3 Run. evb_97811_metrics.log:416:Mon Apr 27 16:43:21 2020: FileMetric: EventBuilder2.Run Number - Last: 3 Run. evb_97811_metrics.log:453:Mon Apr 27 16:43:36 2020: FileMetric: EventBuilder2.Run Number - Last: 3 Run. evb_97811_metrics.log:490:Mon Apr 27 16:43:51 2020: FileMetric: EventBuilder2.Run Number - Last: 3 Run. evb_97811_metrics.log:546:Mon Apr 27 16:44:08 2020: FileMetric: EventBuilder2.Run Number - Last: 3 Run. --2020-04-27_16:44:52--
Additionally, after implementing a pause in the ToyHardwareInterface and a 33 second pause every 35 seconds,
I get:
/home/ron/work/artdaqPrj/demo1-develop/srcs/artdaq_demo ron@mu2edaq13 :^) tshow | egrep 'pausing|resuming' | tdelta -d 1 -ct 1 597 04-28 06:03:18.452095 0 277645 284240 42 ToyHardwareInterface d16 . resuming after pause of 33 seconds 609 04-28 06:03:18.450885 1210 277646 284238 32 ToyHardwareInterface d16 . resuming after pause of 33 seconds 620 04-28 06:02:45.451686 32999199 277645 284240 42 ToyHardwareInterface d16 . pausing 33 seconds 621 04-28 06:02:45.450629 1057 277646 284238 32 ToyHardwareInterface d16 . pausing 33 seconds 1203 04-28 06:02:43.447400 2003229 277645 284240 42 ToyHardwareInterface d16 . resuming after pause of 33 seconds 1205 04-28 06:02:43.446985 415 277646 284238 32 ToyHardwareInterface d16 . resuming after pause of 33 seconds 1210 04-28 06:02:10.447023 32999962 277645 284240 42 ToyHardwareInterface d16 . pausing 33 seconds 1211 04-28 06:02:10.446730 293 277646 284238 32 ToyHardwareInterface d16 . pausing 33 seconds 1793 04-28 06:02:08.443313 2003417 277646 284238 32 ToyHardwareInterface d16 . resuming after pause of 33 seconds 1795 04-28 06:02:08.442881 432 277645 284240 42 ToyHardwareInterface d16 . resuming after pause of 33 seconds 1800 04-28 06:01:35.443080 32999801 277646 284238 32 ToyHardwareInterface d16 . pausing 33 seconds 1801 04-28 06:01:35.442507 573 277645 284240 42 ToyHardwareInterface d16 . pausing 33 seconds 2384 04-28 06:01:33.439609 2002898 277646 284238 32 ToyHardwareInterface d16 . resuming after pause of 33 seconds 2385 04-28 06:01:33.439549 60 277645 284240 42 ToyHardwareInterface d16 . resuming after pause of 33 seconds 2390 04-28 06:01:00.439339 33000210 277646 284238 32 ToyHardwareInterface d16 . pausing 33 seconds 2391 04-28 06:01:00.439201 138 277645 284240 42 ToyHardwareInterface d16 . pausing 33 seconds 2973 04-28 06:00:58.435643 2003558 277646 284238 32 ToyHardwareInterface d16 . resuming after pause of 33 seconds 2975 04-28 06:00:58.435200 443 277645 284240 42 ToyHardwareInterface d16 . resuming after pause of 33 seconds 2980 04-28 06:00:25.435251 32999949 277646 284238 32 ToyHardwareInterface d16 . pausing 33 seconds 2981 04-28 06:00:25.434714 537 277645 284240 42 ToyHardwareInterface d16 . pausing 33 seconds --2020-04-28_08:33:42-- /home/ron/work/artdaqPrj/demo1-develop/daqlogs/metrics/eventbuilder ron@mu2edaq13 :^) grep 'Average Event Size' evb_27764[78]_metrics.log evb_277647_metrics.log:Tue Apr 28 05:59:50 2020: FileMetric: EventBuilder1.Average Event Size: 288.000000 Bytes. evb_277647_metrics.log:Tue Apr 28 06:00:05 2020: FileMetric: EventBuilder1.Average Event Size: 288.000000 Bytes. evb_277647_metrics.log:Tue Apr 28 06:00:20 2020: FileMetric: EventBuilder1.Average Event Size: 288.000000 Bytes. evb_277647_metrics.log:Tue Apr 28 06:00:35 2020: FileMetric: EventBuilder1.Average Event Size: 288.000000 Bytes. evb_277647_metrics.log:Tue Apr 28 06:00:50 2020: FileMetric: EventBuilder1.Average Event Size: 0.000000 Bytes. evb_277647_metrics.log:Tue Apr 28 06:01:05 2020: FileMetric: EventBuilder1.Average Event Size: 288.000000 Bytes. evb_277647_metrics.log:Tue Apr 28 06:01:20 2020: FileMetric: EventBuilder1.Average Event Size: 0.000000 Bytes. evb_277647_metrics.log:Tue Apr 28 06:01:35 2020: FileMetric: EventBuilder1.Average Event Size: 288.000000 Bytes. evb_277647_metrics.log:Tue Apr 28 06:01:50 2020: FileMetric: EventBuilder1.Average Event Size: 0.000000 Bytes. evb_277647_metrics.log:Tue Apr 28 06:02:05 2020: FileMetric: EventBuilder1.Average Event Size: 0.000000 Bytes. evb_277647_metrics.log:Tue Apr 28 06:02:20 2020: FileMetric: EventBuilder1.Average Event Size: 288.000000 Bytes. evb_277647_metrics.log:Tue Apr 28 06:02:36 2020: FileMetric: EventBuilder1.Average Event Size: 0.000000 Bytes. evb_277647_metrics.log:Tue Apr 28 06:02:51 2020: FileMetric: EventBuilder1.Average Event Size: 288.000000 Bytes. evb_277647_metrics.log:Tue Apr 28 06:03:06 2020: FileMetric: EventBuilder1.Average Event Size: 0.000000 Bytes. evb_277647_metrics.log:Tue Apr 28 06:03:27 2020: FileMetric: EventBuilder1.Average Event Size: 0.000000 Bytes. evb_277648_metrics.log:Tue Apr 28 05:59:50 2020: FileMetric: EventBuilder2.Average Event Size: 288.000000 Bytes. evb_277648_metrics.log:Tue Apr 28 06:00:05 2020: FileMetric: EventBuilder2.Average Event Size: 288.000000 Bytes. evb_277648_metrics.log:Tue Apr 28 06:00:20 2020: FileMetric: EventBuilder2.Average Event Size: 288.000000 Bytes. evb_277648_metrics.log:Tue Apr 28 06:00:35 2020: FileMetric: EventBuilder2.Average Event Size: 288.000000 Bytes. evb_277648_metrics.log:Tue Apr 28 06:00:50 2020: FileMetric: EventBuilder2.Average Event Size: 0.000000 Bytes. evb_277648_metrics.log:Tue Apr 28 06:01:05 2020: FileMetric: EventBuilder2.Average Event Size: 288.000000 Bytes. evb_277648_metrics.log:Tue Apr 28 06:01:20 2020: FileMetric: EventBuilder2.Average Event Size: 0.000000 Bytes. evb_277648_metrics.log:Tue Apr 28 06:01:35 2020: FileMetric: EventBuilder2.Average Event Size: 288.000000 Bytes. evb_277648_metrics.log:Tue Apr 28 06:01:50 2020: FileMetric: EventBuilder2.Average Event Size: 0.000000 Bytes. evb_277648_metrics.log:Tue Apr 28 06:02:05 2020: FileMetric: EventBuilder2.Average Event Size: 0.000000 Bytes. evb_277648_metrics.log:Tue Apr 28 06:02:20 2020: FileMetric: EventBuilder2.Average Event Size: 288.000000 Bytes. evb_277648_metrics.log:Tue Apr 28 06:02:35 2020: FileMetric: EventBuilder2.Average Event Size: 0.000000 Bytes. evb_277648_metrics.log:Tue Apr 28 06:02:51 2020: FileMetric: EventBuilder2.Average Event Size: 288.000000 Bytes. evb_277648_metrics.log:Tue Apr 28 06:03:06 2020: FileMetric: EventBuilder2.Average Event Size: 0.000000 Bytes. evb_277648_metrics.log:Tue Apr 28 06:03:27 2020: FileMetric: EventBuilder2.Average Event Size: 288.000000 Bytes. --2020-04-28_06:10:41--
This all seems good.
#4 Updated by Ron Rechenmacher 9 months ago
- Status changed from Resolved to Reviewed
merged branches into develop; change status to reviewed
#5 Updated by Eric Flumerfelt 6 months ago
- Target version set to artdaq v3_09_00
- Status changed from Reviewed to Closed
- Project changed from artdaq Utilities to artdaq