Task #17952

Milestone #17921: Profiling LArSoft Code

do preliminary profiling

Added by Katherine Lato over 3 years ago. Updated about 3 years ago.

Target version:
Start date:
Due date:
% Done:


Estimated time:
Spent time:


Run profiler across all DUNE/LArSoft code.

1) Identify where computing resources are used the most
2) Determine if this is an area worth optimizing or not
3) Suggest improvements and give guidance and assistance.
4) Occasionally, may implement a solution


#1 Updated by Katherine Lato over 3 years ago

  • Category set to Architecture
  • Assignee set to Soon Jun

#2 Updated by Soon Jun over 3 years ago

  • % Done changed from 0 to 10

Installation of Open|Speedshop (under LArSoft/Dune environment on the tev cluster for preliminary tests)

1) Open|Speedshop version 2.2

# setup protodune env through /cvmfs
source /cvmfs/
setup dunetpc v06_54_00 -q e14:prof
export PATH=/home/g4p/products/cmake-3.4.0/bin:${PATH}
# setup gcc that protodune uses
export PATH=/cvmfs/${PATH}
# download openspeedshop from
# and build the offline version from the dune directory (
tar xzf openspeedshop-release-2.2.tar.gz
cd openspeedshop-release-2.2

CC=/cvmfs/ \
CXX=/cvmfs/ \
./install-tool --build-krell-root \
--krell-root-prefix /g4/g4p/dune/krellroot_v2.2 \
--with-openmpi /usr/local/openmpi-1.8.1 \
--with-boost /usr/local/boost_1_60_0 >& build-krell-root.log & 

CC=/cvmfs/ \
CXX=/cvmfs/ \
./install-tool --build-offline \
--krell-root-prefix /g4/g4p/dune/krellroot_v2.2 \
--openss-prefix  /g4/g4p/dune/openspeedshop_v2.2 \
--with-openmpi /usr/local/openmpi-1.8.1 \
--with-boost /usr/local/boost_1_60_0 >& build-offline.log & 

2) Open|Speedshop (version 2.3 CBTF based)

tar xzf openspeedshop-release-2.3
cd openspeedshop-release-2.3

CC=/cvmfs/ \
CXX=/cvmfs/ \
./install-tool --build-krell-root \
--krell-root-prefix /g4/g4p/dune/krellroot_v2.3 \
--with-openmpi /usr/local/openmpi-1.8.1 \
--with-boost /usr/local/boost_1_60_0 >& build-krell-root.log & 

CC=/cvmfs/ \
CXX=/cvmfs/ \
./install-tool --build-cbtf-all \
--krell-root-prefix /g4/g4p/dune/krellroot_v2.3 \
--cbtf-prefix  /g4/g4p/dune/osscbtf_v2.3 \
--with-openmpi /usr/local/openmpi-1.8.1 \
--with-boost /usr/local/boost_1_60_0 >& build-osscbtf.log & 

#3 Updated by Soon Jun over 3 years ago

Initial Profiling tests with LArSoft/Dune (dunetpc version v06_54_00)

  • Check the profilier: open|speedshop (version 2.2) tests on standalone Geant4 applications (SimplifiedCalo, cmsExp) with geant4 version used in dunetpc
    (Done, Successful)
  • Installed art from source (with a temporary fix for the deadlock issue) (Instruction from Lynn/Chris)
    source /cvmfs/
    mkdir /g4/g4p/dune/products
    cp -pr /cvmfs/ /g4/g4p/dune/products/.
    export PRODUCTS=/g4/g4p/dune/products:$PRODUCTS
    mkdir local_art
    cd local_art
    git clone 
    cd art
    git checkout v2_08_03
    #modify art/Utilities/ 
    cd -
    mkdir build
    cd build
    source ../art/ups/setup_for_development -p e14:nu
    buildtool -I /g4/g4p/dune/products -bti -j6
  • fail to run osspcsamp on detsim and reco due to an error from Dyninst::SymtabAPI::Object::parseDwarfFileLineInfo (Investigating)

#4 Updated by Soon Jun over 3 years ago

  • % Done changed from 10 to 20

1) Preliminary profiling test results with IgProf (total memory and CPU performance)
6GeV proton:

2) Consulting an issue of dunetpc profiling with Open|Speedshop with OSS developers (Jim Galarowicz@Krell)!topic/oss-questions/g_wNYLmiKbY
(ossutil error: Assertion `px != 0' failed. from boost smart_ptr)

#5 Updated by Soon Jun over 3 years ago

  • % Done changed from 20 to 50

Have not resolved issues with Open|Speedshop and LArSoft/Dune work-flows.
As an alternative option, tried HPCToolkit ( and successfully profiled typical
simulation and reconstruction chains of LArSoft/protonDune for single particle (6GeV proton) and cosmic (1GeV).
Preliminary profiling results with dunetpc v06_57_00 for those two cases using HPCToolkit and IgProf are available

#6 Updated by Soon Jun over 3 years ago

  • % Done changed from 50 to 80

1) presented ``Preliminary Profiling Results of LArSoft/ProtoDune''at the LArSoft coordination meeting on Dec. 5, 2017 (
2) Add profiling measurement for a DUNE-FD simulation and reconstruction work-flow (prodgenie_nue_dune10kt_1x2x6) of which results are available at (Memory profiling by IgProf) (CPU and other hardware counter profiling by HPCToolkit).

#7 Updated by Soon Jun about 3 years ago

  • % Done changed from 80 to 90

Preliminary rofiling procedure with IgProf and HPCToolkit for typical workflows of LArSoft/protoDune and Dune is summarized at

#8 Updated by Soon Jun about 3 years ago

  • % Done changed from 90 to 100

Add a summary of instruction how to profile typical workflows of LArSoft/(proto)DUNE applications at
which can be linked from the "LArSoft internals" page,

Contact for further support or assistance.

#9 Updated by Soon Jun about 3 years ago

  • Status changed from Assigned to Closed

Also available in: Atom PDF