DUNE » dunetpc

Vito reported reconstruction is very slow since v06_73_00.

Hi Tingjun,

I had no luck to get the reco stage processed using dunetpc v06_73_00
The test jobs get stuck.
I also tried interactively with dunetpc v06_73_00 built by the CI and also using dunetpc v06_73_00
from CVMFS.

In all cases the process get stuck.

Once create a valid proxy and I get dunetpc v06_73_00 set up, the command line I'm using is:

lar --rethrow-all -c standard_reco_dune10kt_nu_1x2x6.fcl -n 5 -o
prodgenie_nue_dune10kt_1x2x6_ana_pass_0.root
xroot://fndca1.fnal.gov:1094//pnfs/fnal.gov/usr/dune/scratch/users/vito/ci_validation/shower_reco_efficiency_validation/v06_73_00/CI_build_lar_ci_2058/sim/prodgenie_nue_dune10kt_1x2x6_detsim_0.root

The same commands works with dunetpc v06_72_00.

The last few lines from the log I got are:

[...]
Begin processing the 1st record. run: 20000002 subRun: 0 event: 1 at 09-Apr-2018 17:18:49 CDT
FS.M2DS.FMS1 2D showers in this 3D shower have parent Tjs from different PFParticles
 shower set
%MSG-e pma::Track3D:  PMAlgTrackMaker:pmtrack@ 09-Apr-2018 17:19:27 CDT  run: 20000002 subRun: 0
event: 1
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrackMaker:pmtrack@ 09-Apr-2018 17:19:27 CDT  run: 20000002 subRun: 0
event: 1
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrackMaker:pmtrack@ 09-Apr-2018 17:19:27 CDT  run: 20000002 subRun: 0
event: 1
TuneFullTree failed.
%MSG

those are the same I see for dunetpc v06_72_00 when it start to process the first event.

The full log of my interactive test is available at:
/dune/app/users/vito/CI/standard_reco_dune10kt_nu_1x2x6.log

It is now already 20 minutes that it is stuck.

I also tried to process with dunetpc v06_73_00 a detsim output created with dunetpc v06_72_00, but
no luck.

In the DUNEFD CI test we are using the same FHiCL file, so I was wondering why the CI test is not
affected by this issue.
The log of the most recent DUNEFD reco CI test is available at:

http://dbweb5.fnal.gov:8080/LarCI/app/ns:dune/storage/docs/2018/04/09/stdout_qARb8SW.log

I see at line 118 a different output from the one I pointed out above.

Trying to process with dunetpc v06_73_00 the input file used bu the CI tests it works.

So I'm not sure what is going on with dunetpc v06_73_00.

If you think there is some other test I can do, please, let me know.

Thanks,
    Vito