Project

General

Profile

Bug #19619

Slowness in reconstruction since v06_73_00

Added by Tingjun Yang over 1 year ago. Updated about 1 year ago.

Status:
Closed
Priority:
Normal
Assignee:
-
Start date:
04/10/2018
Due date:
% Done:

0%

Estimated time:
Duration:

Description

Vito reported reconstruction is very slow since v06_73_00.

Hi Tingjun,

I had no luck to get the reco stage processed using dunetpc v06_73_00
The test jobs get stuck.
I also tried interactively with dunetpc v06_73_00 built by the CI and also using dunetpc v06_73_00
from CVMFS.

In all cases the process get stuck.

Once create a valid proxy and I get dunetpc v06_73_00 set up, the command line I'm using is:

lar --rethrow-all -c standard_reco_dune10kt_nu_1x2x6.fcl -n 5 -o
prodgenie_nue_dune10kt_1x2x6_ana_pass_0.root
xroot://fndca1.fnal.gov:1094//pnfs/fnal.gov/usr/dune/scratch/users/vito/ci_validation/shower_reco_efficiency_validation/v06_73_00/CI_build_lar_ci_2058/sim/prodgenie_nue_dune10kt_1x2x6_detsim_0.root

The same commands works with dunetpc v06_72_00.

The last few lines from the log I got are:

[...]
Begin processing the 1st record. run: 20000002 subRun: 0 event: 1 at 09-Apr-2018 17:18:49 CDT
FS.M2DS.FMS1 2D showers in this 3D shower have parent Tjs from different PFParticles
 shower set
%MSG-e pma::Track3D:  PMAlgTrackMaker:pmtrack@ 09-Apr-2018 17:19:27 CDT  run: 20000002 subRun: 0
event: 1
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrackMaker:pmtrack@ 09-Apr-2018 17:19:27 CDT  run: 20000002 subRun: 0
event: 1
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrackMaker:pmtrack@ 09-Apr-2018 17:19:27 CDT  run: 20000002 subRun: 0
event: 1
TuneFullTree failed.
%MSG

those are the same I see for dunetpc v06_72_00 when it start to process the first event.

The full log of my interactive test is available at:
/dune/app/users/vito/CI/standard_reco_dune10kt_nu_1x2x6.log

It is now already 20 minutes that it is stuck.

I also tried to process with dunetpc v06_73_00 a detsim output created with dunetpc v06_72_00, but
no luck.

In the DUNEFD CI test we are using the same FHiCL file, so I was wondering why the CI test is not
affected by this issue.
The log of the most recent DUNEFD reco CI test is available at:

http://dbweb5.fnal.gov:8080/LarCI/app/ns:dune/storage/docs/2018/04/09/stdout_qARb8SW.log

I see at line 118 a different output from the one I pointed out above.

Trying to process with dunetpc v06_73_00 the input file used bu the CI tests it works.

So I'm not sure what is going on with dunetpc v06_73_00.

If you think there is some other test I can do, please, let me know.

Thanks,
    Vito

History

#1 Updated by Tingjun Yang over 1 year ago

Code changes since v06_72_00 (pointed out by Leigh)

diff --git a/larreco/RecoAlg/PMAlg/SortedObjects.h b/larreco/RecoAlg/PMAlg/SortedObjects.h
index 20e2255c..e7c0a7ec 100644
--- a/larreco/RecoAlg/PMAlg/SortedObjects.h
+++ b/larreco/RecoAlg/PMAlg/SortedObjects.h
@@ -62,7 +62,7 @@ class pma::SortedBranchBase : public pma::SortedObjectBase
 public:
        SortedBranchBase(void) : pma::SortedObjectBase() {}
        SortedBranchBase(pma::SortedObjectBase* prevElement, pma::SortedObjectBase* nextElement = 0) :
-               pma::SortedObjectBase(prev, next)
+               pma::SortedObjectBase(prevElement, nextElement)
        {
                if (nextElement) next_vector.push_back(next);
        }

and
index e699a3d5..edb591bc 100644
--- a/larreco/RecoAlg/ProjectionMatchingAlg.cxx
+++ b/larreco/RecoAlg/ProjectionMatchingAlg.cxx
@@ -10,6 +10,7 @@

 #include "larevt/CalibrationDBI/Interface/ChannelStatusService.h" 
 #include "larevt/CalibrationDBI/Interface/ChannelStatusProvider.h" 
+#include "larcorealg/CoreUtils/NumericUtils.h" // util::absDiff()

 #include "messagefacility/MessageLogger/MessageLogger.h" 

@@ -781,7 +782,8 @@ void pma::ProjectionMatchingAlg::ShortenSeg(pma::Track3D& trk, const geo::TPCGeo

                RemoveNotEnabledHits(trk);

-               trk.Optimize(0.0001, false);
+               // trk.Optimize(0.0001, false); // BUG: first argument missing; tentatively:
+               trk.Optimize(-1, 0.0001, false);
                trk.SortHits();

                mf::LogWarning("ProjectionMatchingAlg") << " mse: " << mse;
@@ -1364,7 +1366,7 @@ double pma::ProjectionMatchingAlg::selectInitialHits(pma::Track3D& trk, unsigned
                {
                        hit = trk[ih];

-                       if (abs(hit->Wire() - lastHit->Wire()) > 2)
+                       if (util::absDiff(hit->Wire(), lastHit->Wire()) > 2)
                                break; // break on gap in wire direction

                        last_x = trk.HitDxByView(ih, view);

#2 Updated by Tingjun Yang about 1 year ago

  • Status changed from New to Resolved

#3 Updated by Tingjun Yang about 1 year ago

  • Status changed from Resolved to Closed


Also available in: Atom PDF