Project

General

Profile

Bug #20387

Something set all of my requested trace levels to zero

Added by Kurt Biery about 1 year ago. Updated about 1 year ago.

Status:
New
Priority:
Low
Assignee:
-
Category:
-
Target version:
-
Start date:
07/21/2018
Due date:
% Done:

0%

Estimated time:
Experiment:
-
Co-Assignees:
Duration:

Description

I'm not sure if I'm imagining this, but I think that Ron has seen it too, but something set all of my requested trace levels to zero:

[biery@mu2edaq01 artdaqDemo]$ tlvls
mode: M=0 S=0
TID NAME maskM maskS maskT
---- --------------------------------------- ------------------ ------------------ ------------------
0 SharedMemoryManager 0x0000000000000000 0x0000000000000000 0x0000000000000000
1 component01_CommandableInterface 0x0000000000000000 0x0000000000000000 0x0000000000000000
2 component02_CommandableInterface 0x0000000000000000 0x0000000000000000 0x0000000000000000
3 component03_CommandableInterface 0x0000000000000000 0x0000000000000000 0x0000000000000000
4 component04_CommandableInterface 0x0000000000000000 0x0000000000000000 0x0000000000000000
5 component05_CommandableInterface 0x0000000000000000 0x0000000000000000 0x0000000000000000
6 component06_CommandableInterface 0x0000000000000000 0x0000000000000000 0x0000000000000000
7 component07_CommandableInterface 0x0000000000000000 0x0000000000000000 0x0000000000000000
8 component08_CommandableInterface 0x0000000000000000 0x0000000000000000 0x0000000000000000
9 component09_CommandableInterface 0x0000000000000000 0x0000000000000000 0x0000000000000000
10 component10_CommandableInterface 0x0000000000000000 0x0000000000000000 0x0000000000000000
11 EventBuilder1_CommandableInterface 0x0000000000000000 0x0000000000000000 0x0000000000000000
12 EventBuilder2_CommandableInterface 0x0000000000000000 0x0000000000000000 0x0000000000000000
13 EventBuilder3_CommandableInterface 0x0000000000000000 0x0000000000000000 0x0000000000000000
14 DataLogger1_CommandableInterface 0x0000000000000000 0x0000000000000000 0x0000000000000000
15 Dispatcher1_CommandableInterface 0x0000000000000000 0x0000000000000000 0x0000000000000000
16 MPISentry 0x0000000000000000 0x0000000000000000 0x0000000000000000
17 TRACE 0x0000000000000000 0x0000000000000000 0x0000000000000000
18 LoadParameterSet 0x0000000000000000 0x0000000000000000 0x0000000000000000
19 configureMessageFacility 0x0000000000000000 0x0000000000000000 0x0000000000000000
20 GenFileOutput 0x0000000000000000 0x0000000000000000 0x0000000000000000
21 component09_TCPConnect 0x0000000000000000 0x0000000000000000 0x0000000000000000
22 component03_TCPConnect 0x0000000000000000 0x0000000000000000 0x0000000000000000
23 component04_TCPConnect 0x0000000000000000 0x0000000000000000 0x0000000000000000
24 component10_TCPConnect 0x0000000000000000 0x0000000000000000 0x0000000000000000
25 DataLogger1_TCPConnect 0x0000000000000000 0x0000000000000000 0x0000000000000000
26 EventBuilder1_TCPConnect 0x0000000000000000 0x0000000000000000 0x0000000000000000
27 Dispatcher1_TCPConnect 0x0000000000000000 0x0000000000000000 0x0000000000000000
28 component03Main 0x0000000000000000 0x0000000000000000 0x0000000000000000
29 component09Main 0x0000000000000000 0x0000000000000000 0x0000000000000000
30 component04Main 0x0000000000000000 0x0000000000000000 0x0000000000000000
31 component10Main 0x0000000000000000 0x0000000000000000 0x0000000000000000
32 DataLogger1Main 0x0000000000000000 0x0000000000000000 0x0000000000000000
33 EventBuilder2_TCPConnect 0x0000000000000000 0x0000000000000000 0x0000000000000000
34 Dispatcher1Main 0x0000000000000000 0x0000000000000000 0x0000000000000000
35 component01_TCPConnect 0x0000000000000000 0x0000000000000000 0x0000000000000000
36 component07_TCPConnect 0x0000000000000000 0x0000000000000000 0x0000000000000000
37 EventBuilder1Main 0x0000000000000000 0x0000000000000000 0x0000000000000000
38 EventBuilder2Main 0x0000000000000000 0x0000000000000000 0x0000000000000000
39 component01Main 0x0000000000000000 0x0000000000000000 0x0000000000000000
40 component08_TCPConnect 0x0000000000000000 0x0000000000000000 0x0000000000000000
41 component07Main 0x0000000000000000 0x0000000000000000 0x0000000000000000
42 component02_TCPConnect 0x0000000000000000 0x0000000000000000 0x0000000000000000
43 component08Main 0x0000000000000000 0x0000000000000000 0x0000000000000000
44 EventBuilder3_TCPConnect 0x0000000000000000 0x0000000000000000 0x0000000000000000
45 EventBuilder3Main 0x0000000000000000 0x0000000000000000 0x0000000000000000
46 component02Main 0x0000000000000000 0x0000000000000000 0x0000000000000000
47 component05_TCPConnect 0x0000000000000000 0x0000000000000000 0x0000000000000000
48 component06_TCPConnect 0x0000000000000000 0x0000000000000000 0x0000000000000000
49 component05Main 0x0000000000000000 0x0000000000000000 0x0000000000000000
50 component06Main 0x0000000000000000 0x0000000000000000 0x0000000000000000
51 component02_xmlrpc_commander 0x0000000000000000 0x0000000000000000 0x0000000000000000
52 component08_xmlrpc_commander 0x0000000000000000 0x0000000000000000 0x0000000000000000
53 component06_xmlrpc_commander 0x0000000000000000 0x0000000000000000 0x0000000000000000
54 EventBuilder1_xmlrpc_commander 0x0000000000000000 0x0000000000000000 0x0000000000000000
55 component07_xmlrpc_commander 0x0000000000000000 0x0000000000000000 0x0000000000000000
56 DataLogger1_xmlrpc_commander 0x0000000000000000 0x0000000000000000 0x0000000000000000
57 component05_xmlrpc_commander 0x0000000000000000 0x0000000000000000 0x0000000000000000
58 EventBuilder3_xmlrpc_commander 0x0000000000000000 0x0000000000000000 0x0000000000000000
59 component01_xmlrpc_commander 0x0000000000000000 0x0000000000000000 0x0000000000000000
60 Dispatcher1_xmlrpc_commander 0x0000000000000000 0x0000000000000000 0x0000000000000000
61 component04_xmlrpc_commander 0x0000000000000000 0x0000000000000000 0x0000000000000000
62 component03_xmlrpc_commander 0x0000000000000000 0x0000000000000000 0x0000000000000000
63 EventBuilder2_xmlrpc_commander 0x0000000000000000 0x0000000000000000 0x0000000000000000
64 component10_xmlrpc_commander 0x0000000000000000 0x0000000000000000 0x0000000000000000
65 component09_xmlrpc_commander 0x0000000000000000 0x0000000000000000 0x0000000000000000
66 UDP_Receiver 0x0000000000000000 0x0000000000000000 0x0000000000000000
67 Dispatcher1_DataReceiverCore 0x0000000000000000 0x0000000000000000 0x0000000000000000
68 Dispatcher1_DispatcherCore 0x0000000000000000 0x0000000000000000 0x0000000000000000
69 MetricManager 0x0000000000000000 0x0000000000000000 0x0000000000000000
70 Dispatcher1_SharedMemoryEventManager 0x0000000000000000 0x0000000000000000 0x0000000000000000
71 Dispatcher1_DataReceiverManager 0x0000000000000000 0x0000000000000000 0x0000000000000000
72 Dispatcher1_TransferInterface 0x0000000000000000 0x0000000000000000 0x0000000000000000
73 Dispatcher1_TCPSocketTransfer 0x0000000000000000 0x0000000000000000 0x0000000000000000
74 Dispatcher1_TCP_listen_fd 0x0000000000000000 0x0000000000000000 0x0000000000000000
75 DataLogger1_DataReceiverCore 0x0000000000000000 0x0000000000000000 0x0000000000000000
76 DataLogger1_DataLoggerCore 0x0000000000000000 0x0000000000000000 0x0000000000000000
77 DataLogger1_SharedMemoryEventManager 0x0000000000000000 0x0000000000000000 0x0000000000000000
78 DataLogger1_DataReceiverManager 0x0000000000000000 0x0000000000000000 0x0000000000000000
79 DataLogger1_TransferInterface 0x0000000000000000 0x0000000000000000 0x0000000000000000
80 DataLogger1_TCPSocketTransfer 0x0000000000000000 0x0000000000000000 0x0000000000000000
81 DataLogger1_TCP_listen_fd 0x0000000000000000 0x0000000000000000 0x0000000000000000
82 EventBuilder1_DataReceiverCore 0x0000000000000000 0x0000000000000000 0x0000000000000000
83 EventBuilder2_DataReceiverCore 0x0000000000000000 0x0000000000000000 0x0000000000000000
84 EventBuilder1_EventBuilderCore 0x0000000000000000 0x0000000000000000 0x0000000000000000
85 EventBuilder2_EventBuilderCore 0x0000000000000000 0x0000000000000000 0x0000000000000000
86 EventBuilder3_DataReceiverCore 0x0000000000000000 0x0000000000000000 0x0000000000000000
87 EventBuilder3_EventBuilderCore 0x0000000000000000 0x0000000000000000 0x0000000000000000
88 EventBuilder1_SharedMemoryEventManager 0x0000000000000000 0x0000000000000000 0x0000000000000000
89 EventBuilder2_SharedMemoryEventManager 0x0000000000000000 0x0000000000000000 0x0000000000000000
90 EventBuilder3_SharedMemoryEventManager 0x0000000000000000 0x0000000000000000 0x0000000000000000
91 EventBuilder2_DataReceiverManager 0x0000000000000000 0x0000000000000000 0x0000000000000000
92 EventBuilder1_DataReceiverManager 0x0000000000000000 0x0000000000000000 0x0000000000000000
93 EventBuilder3_DataReceiverManager 0x0000000000000000 0x0000000000000000 0x0000000000000000
94 EventBuilder2_TransferInterface 0x0000000000000000 0x0000000000000000 0x0000000000000000
95 EventBuilder1_TransferInterface 0x0000000000000000 0x0000000000000000 0x0000000000000000
96 EventBuilder1_TCPSocketTransfer 0x0000000000000000 0x0000000000000000 0x0000000000000000
97 EventBuilder2_TCPSocketTransfer 0x0000000000000000 0x0000000000000000 0x0000000000000000
98 EventBuilder3_TransferInterface 0x0000000000000000 0x0000000000000000 0x0000000000000000
99 EventBuilder3_TCPSocketTransfer 0x0000000000000000 0x0000000000000000 0x0000000000000000
100 EventBuilder2_TCP_listen_fd 0x0000000000000000 0x0000000000000000 0x0000000000000000
101 EventBuilder1_TCP_listen_fd 0x0000000000000000 0x0000000000000000 0x0000000000000000
102 EventBuilder3_TCP_listen_fd 0x0000000000000000 0x0000000000000000 0x0000000000000000
103 component01_BoardReaderApp 0x0000000000000000 0x0000000000000000 0x0000000000000000
104 component01_BoardReaderCore 0x0000000000000000 0x0000000000000000 0x0000000000000000
105 component04_BoardReaderApp 0x0000000000000000 0x0000000000000000 0x0000000000000000
106 component04_BoardReaderCore 0x0000000000000000 0x0000000000000000 0x0000000000000000
107 component02_BoardReaderApp 0x0000000000000000 0x0000000000000000 0x0000000000000000
108 component02_BoardReaderCore 0x0000000000000000 0x0000000000000000 0x0000000000000000
109 component03_BoardReaderApp 0x0000000000000000 0x0000000000000000 0x0000000000000000
110 component03_BoardReaderCore 0x0000000000000000 0x0000000000000000 0x0000000000000000
111 component09_BoardReaderApp 0x0000000000000000 0x0000000000000000 0x0000000000000000
112 component09_BoardReaderCore 0x0000000000000000 0x0000000000000000 0x0000000000000000
113 component08_BoardReaderApp 0x0000000000000000 0x0000000000000000 0x0000000000000000
114 component08_BoardReaderCore 0x0000000000000000 0x0000000000000000 0x0000000000000000
115 component06_BoardReaderApp 0x0000000000000000 0x0000000000000000 0x0000000000000000
116 component06_BoardReaderCore 0x0000000000000000 0x0000000000000000 0x0000000000000000
117 component05_BoardReaderApp 0x0000000000000000 0x0000000000000000 0x0000000000000000
118 component05_BoardReaderCore 0x0000000000000000 0x0000000000000000 0x0000000000000000
119 component07_BoardReaderApp 0x0000000000000000 0x0000000000000000 0x0000000000000000
120 component07_BoardReaderCore 0x0000000000000000 0x0000000000000000 0x0000000000000000
121 component10_BoardReaderApp 0x0000000000000000 0x0000000000000000 0x0000000000000000
122 component10_BoardReaderCore 0x0000000000000000 0x0000000000000000 0x0000000000000000
123 component01_CommandableFragmentGenerato 0x0000000000000000 0x0000000000000000 0x0000000000000000
124 Fragment 0x0000000000000000 0x0000000000000000 0x0000000000000000
125 ToySimulator 0x0000000000000000 0x0000000000000000 0x0000000000000000
126 component04_CommandableFragmentGenerato 0x0000000000000000 0x0000000000000000 0x0000000000000000
127 component04_RequestReceiver 0x0000000000000000 0x0000000000000000 0x0000000000000000
128 component02_CommandableFragmentGenerato 0x0000000000000000 0x0000000000000000 0x0000000000000000
129 component09_CommandableFragmentGenerato 0x0000000000000000 0x0000000000000000 0x0000000000000000
130 component02_RequestReceiver 0x0000000000000000 0x0000000000000000 0x0000000000000000
131 component09_RequestReceiver 0x0000000000000000 0x0000000000000000 0x0000000000000000
132 component03_CommandableFragmentGenerato 0x0000000000000000 0x0000000000000000 0x0000000000000000
133 component03_RequestReceiver 0x0000000000000000 0x0000000000000000 0x0000000000000000
134 component06_CommandableFragmentGenerato 0x0000000000000000 0x0000000000000000 0x0000000000000000
135 component06_RequestReceiver 0x0000000000000000 0x0000000000000000 0x0000000000000000
136 component08_CommandableFragmentGenerato 0x0000000000000000 0x0000000000000000 0x0000000000000000
137 component05_CommandableFragmentGenerato 0x0000000000000000 0x0000000000000000 0x0000000000000000
138 component05_RequestReceiver 0x0000000000000000 0x0000000000000000 0x0000000000000000
139 component08_RequestReceiver 0x0000000000000000 0x0000000000000000 0x0000000000000000
140 component07_CommandableFragmentGenerato 0x0000000000000000 0x0000000000000000 0x0000000000000000
141 component07_RequestReceiver 0x0000000000000000 0x0000000000000000 0x0000000000000000
142 component10_CommandableFragmentGenerato 0x0000000000000000 0x0000000000000000 0x0000000000000000
143 component10_RequestReceiver 0x0000000000000000 0x0000000000000000 0x0000000000000000
144 Dispatcher1_RequestSender 0x0000000000000000 0x0000000000000000 0x0000000000000000
145 SharedMemoryEventReceiver 0x0000000000000000 0x0000000000000000 0x0000000000000000
146 NetMonTransportService 0x0000000000000000 0x0000000000000000 0x0000000000000000
147 DataLogger1_RequestSender 0x0000000000000000 0x0000000000000000 0x0000000000000000
148 ArtdaqInput 0x0000000000000000 0x0000000000000000 0x0000000000000000
149 NetMonWrapper 0x0000000000000000 0x0000000000000000 0x0000000000000000
150 EventBuilder1_RequestSender 0x0000000000000000 0x0000000000000000 0x0000000000000000
151 EventBuilder2_RequestSender 0x0000000000000000 0x0000000000000000 0x0000000000000000
152 EventBuilder3_RequestSender 0x0000000000000000 0x0000000000000000 0x0000000000000000
153 QuickVec 0x0000000000000000 0x0000000000000000 0x0000000000000000
154 ToyHardwareInterface 0x0000000000000000 0x0000000000000000 0x0000000000000000
155 component01_DataSenderManager 0x0000000000000000 0x0000000000000000 0x0000000000000000
156 component02_DataSenderManager 0x0000000000000000 0x0000000000000000 0x0000000000000000
157 SharedMemoryReader 0x0000000000000000 0x0000000000000000 0x0000000000000000
158 component03_DataSenderManager 0x0000000000000000 0x0000000000000000 0x0000000000000000
159 component04_DataSenderManager 0x0000000000000000 0x0000000000000000 0x0000000000000000
160 component06_DataSenderManager 0x0000000000000000 0x0000000000000000 0x0000000000000000
161 component07_DataSenderManager 0x0000000000000000 0x0000000000000000 0x0000000000000000
162 component09_DataSenderManager 0x0000000000000000 0x0000000000000000 0x0000000000000000
163 component08_DataSenderManager 0x0000000000000000 0x0000000000000000 0x0000000000000000
164 component05_DataSenderManager 0x0000000000000000 0x0000000000000000 0x0000000000000000
165 component10_DataSenderManager 0x0000000000000000 0x0000000000000000 0x0000000000000000
166 RootNetOutput 0x0000000000000000 0x0000000000000000 0x0000000000000000
167 _DataSenderManager 0x0000000000000000 0x0000000000000000 0x0000000000000000
168 component01_TransferInterface 0x0000000000000000 0x0000000000000000 0x0000000000000000
169 component01_TCPSocketTransfer 0x0000000000000000 0x0000000000000000 0x0000000000000000
170 component03_TransferInterface 0x0000000000000000 0x0000000000000000 0x0000000000000000
171 component03_TCPSocketTransfer 0x0000000000000000 0x0000000000000000 0x0000000000000000
172 component02_TransferInterface 0x0000000000000000 0x0000000000000000 0x0000000000000000
173 component02_TCPSocketTransfer 0x0000000000000000 0x0000000000000000 0x0000000000000000
174 component07_TransferInterface 0x0000000000000000 0x0000000000000000 0x0000000000000000
175 component07_TCPSocketTransfer 0x0000000000000000 0x0000000000000000 0x0000000000000000
176 component06_TransferInterface 0x0000000000000000 0x0000000000000000 0x0000000000000000
177 component06_TCPSocketTransfer 0x0000000000000000 0x0000000000000000 0x0000000000000000
178 component04_TransferInterface 0x0000000000000000 0x0000000000000000 0x0000000000000000
179 component04_TCPSocketTransfer 0x0000000000000000 0x0000000000000000 0x0000000000000000
180 component09_TransferInterface 0x0000000000000000 0x0000000000000000 0x0000000000000000
181 component09_TCPSocketTransfer 0x0000000000000000 0x0000000000000000 0x0000000000000000
182 component08_TransferInterface 0x0000000000000000 0x0000000000000000 0x0000000000000000
183 component08_TCPSocketTransfer 0x0000000000000000 0x0000000000000000 0x0000000000000000
184 component05_TransferInterface 0x0000000000000000 0x0000000000000000 0x0000000000000000
185 component05_TCPSocketTransfer 0x0000000000000000 0x0000000000000000 0x0000000000000000
186 component10_TransferInterface 0x0000000000000000 0x0000000000000000 0x0000000000000000
187 component10_TCPSocketTransfer 0x0000000000000000 0x0000000000000000 0x0000000000000000
188 _TransferInterface 0x0000000000000000 0x0000000000000000 0x0000000000000000
189 _TCPSocketTransfer 0x0000000000000000 0x0000000000000000 0x0000000000000000
190 ContainerFragmentLoader 0x0000000000000000 0x0000000000000000 0x0000000000000000
191 DataLogger1_CommandableFragmentGenerato 0x0000000000000000 0x0000000000000000 0x0000000000000000
192 InputUtilities 0x0000000000000000 0x0000000000000000 0x0000000000000000
193 EventBuilder3_CommandableFragmentGenera 0x0000000000000000 0x0000000000000000 0x0000000000000000
194 EventBuilder2_CommandableFragmentGenera 0x0000000000000000 0x0000000000000000 0x0000000000000000
195 EventBuilder1_CommandableFragmentGenera 0x0000000000000000 0x0000000000000000 0x0000000000000000
196 Dispatcher1_CommandableFragmentGenerato 0x0000000000000000 0x0000000000000000 0x0000000000000000
197 UDP_mfPlugin 0x0000000000000000 0x0000000000000000 0x0000000000000000
198 EventBuilder2_DataLoggerApp 0x0000000000000000 0x0000000000000000 0x0000000000000000
199 EventBuilder2App 0x0000000000000000 0x0000000000000000 0x0000000000000000
200 EventBuilder1_DataLoggerApp 0x0000000000000000 0x0000000000000000 0x0000000000000000
201 EventBuilder1App 0x0000000000000000 0x0000000000000000 0x0000000000000000
202 EventBuilder3_DataLoggerApp 0x0000000000000000 0x0000000000000000 0x0000000000000000
203 EventBuilder3App 0x0000000000000000 0x0000000000000000 0x0000000000000000
204 RandomDelayFilter 0x0000000000000000 0x0000000000000000 0x0000000000000000

History

#1 Updated by Ron Rechenmacher about 1 year ago

kurt thinks he has a simple test config that will help reproduce this.
Pester him to commit it.

#2 Updated by Kurt Biery about 1 year ago

Ron,
I noticed this when I was running the mediumsystem_mediumdata simple_test_config, and I believe that the change that triggered this behavior was the addition of a delay of one second per event to the DataLoggers. An example diff of the DataLogger1.fcl file is included below. I was thinking about copying the mediumsystem_mediumdata to a new config called extreme_backpressure, or something like that, and adding the DataLogger delays, but I haven't done that yet.
Kurt

@@ -57,19 +57,23 @@ outputs: {

 }
 physics: {
-  analyzers: {
-
+  filters: {
+   randomDelay: {
+     module_type: RandomDelayFilter
+     delay_distribution_type: "Fixed" 
+     minimum_delay_ms: 110
+   }
+  }

+  analyzers: {
    checkintegrity: {
      module_type: CheckIntegrity
      raw_data_label: daq
      frag_type: TOY1
    }
-
   }

   producers: {
-
      BuildInfo:
      {
        module_type: ArtdaqDemoBuildInfo
@@ -77,7 +81,7 @@ physics: {
      }
   }

-  p2: [ BuildInfo ]
+  p2: [ BuildInfo, randomDelay ]
   my_output_modules: [ normalOutput ]
   my_mpi_output_modules: [ rootNetOutput ]
 }


Also available in: Atom PDF