Project

General

Profile

Task #17952

Milestone #17921: Profiling LArSoft Code

do preliminary profiling

Added by Katherine Lato over 2 years ago. Updated over 2 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
Architecture
Target version:
-
Start date:
10/18/2017
Due date:
% Done:

100%

Estimated time:
Spent time:
Duration:

Description

Run profiler across all DUNE/LArSoft code.

1) Identify where computing resources are used the most
2) Determine if this is an area worth optimizing or not
3) Suggest improvements and give guidance and assistance.
4) Occasionally, may implement a solution

History

#1 Updated by Katherine Lato over 2 years ago

  • Category set to Architecture
  • Assignee set to Soon Jun

#2 Updated by Soon Jun over 2 years ago

  • % Done changed from 0 to 10

Installation of Open|Speedshop (under LArSoft/Dune environment on the tev cluster for preliminary tests)

1) Open|Speedshop version 2.2

#
# setup protodune env through /cvmfs
#
source /cvmfs/dune.opensciencegrid.org/products/dune/setup_dune.sh
setup dunetpc v06_54_00 -q e14:prof
export PATH=/home/g4p/products/cmake-3.4.0/bin:${PATH}
#
# setup gcc that protodune uses
#
export PATH=/cvmfs/fermilab.opensciencegrid.org/products/larsoft/gcc/v6_3_0/Linux64bit+2.6-2.12/bin:${PATH}
export LD_LIBRARY_PATH=/cvmfs/fermilab.opensciencegrid.org/products/larsoft/gcc/v6_3_0/Linux64bit+2.6-2.12/lib:${LD_LIBRARY_PATH}
export LD_LIBRARY_PATH=/cvmfs/fermilab.opensciencegrid.org/products/larsoft/gcc/v6_3_0/Linux64bit+2.6-2.12/lib64:${LD_LIBRARY_PATH}
#
# download openspeedshop from https://sourceforge.net/projects/openss/files/openss/
# and build the offline version from the dune directory (tev.fnal.gov:/g4/g4p/dune)
#
tar xzf openspeedshop-release-2.2.tar.gz
cd openspeedshop-release-2.2

CC=/cvmfs/fermilab.opensciencegrid.org/products/larsoft/gcc/v6_3_0/Linux64bit+2.6-2.12/bin/gcc \
CXX=/cvmfs/fermilab.opensciencegrid.org/products/larsoft/gcc/v6_3_0/Linux64bit+2.6-2.12/bin/g++ \
./install-tool --build-krell-root \
--krell-root-prefix /g4/g4p/dune/krellroot_v2.2 \
--with-openmpi /usr/local/openmpi-1.8.1 \
--with-boost /usr/local/boost_1_60_0 >& build-krell-root.log & 

CC=/cvmfs/fermilab.opensciencegrid.org/products/larsoft/gcc/v6_3_0/Linux64bit+2.6-2.12/bin/gcc \
CXX=/cvmfs/fermilab.opensciencegrid.org/products/larsoft/gcc/v6_3_0/Linux64bit+2.6-2.12/bin/g++ \
./install-tool --build-offline \
--krell-root-prefix /g4/g4p/dune/krellroot_v2.2 \
--openss-prefix  /g4/g4p/dune/openspeedshop_v2.2 \
--with-openmpi /usr/local/openmpi-1.8.1 \
--with-boost /usr/local/boost_1_60_0 >& build-offline.log & 

2) Open|Speedshop (version 2.3 CBTF based)

tar xzf openspeedshop-release-2.3
cd openspeedshop-release-2.3

CC=/cvmfs/fermilab.opensciencegrid.org/products/larsoft/gcc/v6_3_0/Linux64bit+2.6-2.12/bin/gcc \
CXX=/cvmfs/fermilab.opensciencegrid.org/products/larsoft/gcc/v6_3_0/Linux64bit+2.6-2.12/bin/g++ \
./install-tool --build-krell-root \
--krell-root-prefix /g4/g4p/dune/krellroot_v2.3 \
--with-openmpi /usr/local/openmpi-1.8.1 \
--with-boost /usr/local/boost_1_60_0 >& build-krell-root.log & 

CC=/cvmfs/fermilab.opensciencegrid.org/products/larsoft/gcc/v6_3_0/Linux64bit+2.6-2.12/bin/gcc \
CXX=/cvmfs/fermilab.opensciencegrid.org/products/larsoft/gcc/v6_3_0/Linux64bit+2.6-2.12/bin/g++ \
./install-tool --build-cbtf-all \
--krell-root-prefix /g4/g4p/dune/krellroot_v2.3 \
--cbtf-prefix  /g4/g4p/dune/osscbtf_v2.3 \
--with-openmpi /usr/local/openmpi-1.8.1 \
--with-boost /usr/local/boost_1_60_0 >& build-osscbtf.log & 

#3 Updated by Soon Jun over 2 years ago

Initial Profiling tests with LArSoft/Dune (dunetpc version v06_54_00)

  • Check the profilier: open|speedshop (version 2.2) tests on standalone Geant4 applications (SimplifiedCalo, cmsExp) with geant4 version used in dunetpc
    /cvmfs/fermilab.opensciencegrid.org/products/larsoft/geant4/v4_10_3_p01a
    (Done, Successful)
  • Installed art from source (with a temporary fix for the deadlock issue) (Instruction from Lynn/Chris)
    on tev.fnal.gov
    
    source /cvmfs/fermilab.opensciencegrid.org/products/larsoft/setup
    mkdir /g4/g4p/dune/products
    cp -pr /cvmfs/fermilab.opensciencegrid.org/products/larsoft/.upsfiles /g4/g4p/dune/products/.
    export PRODUCTS=/g4/g4p/dune/products:$PRODUCTS
    
    mkdir local_art
    cd local_art
    git clone https://cdcvs.fnal.gov/projects/art 
    cd art
    git checkout v2_08_03
    #modify art/Utilities/LinuxProcMgr.cc 
    cd -
    mkdir build
    cd build
    source ../art/ups/setup_for_development -p e14:nu
    buildtool -I /g4/g4p/dune/products -bti -j6
    
  • fail to run osspcsamp on detsim and reco due to an error from Dyninst::SymtabAPI::Object::parseDwarfFileLineInfo (Investigating)

#4 Updated by Soon Jun over 2 years ago

  • % Done changed from 10 to 20

1) Preliminary profiling test results with IgProf (total memory and CPU performance)
6GeV proton: https://g4cpt.fnal.gov/larsoft/dunetpc_v06_54_00/igprof_pp.html

2) Consulting an issue of dunetpc profiling with Open|Speedshop with OSS developers (Jim Galarowicz@Krell)
https://groups.google.com/a/krellinst.org/forum/?fromgroups#!topic/oss-questions/g_wNYLmiKbY
(ossutil error: Assertion `px != 0' failed. from boost smart_ptr)

#5 Updated by Soon Jun over 2 years ago

  • % Done changed from 20 to 50

Have not resolved issues with Open|Speedshop and LArSoft/Dune work-flows.
As an alternative option, tried HPCToolkit (http://hpctoolkit.org/index.html) and successfully profiled typical
simulation and reconstruction chains of LArSoft/protonDune for single particle (6GeV proton) and cosmic (1GeV).
Preliminary profiling results with dunetpc v06_57_00 for those two cases using HPCToolkit and IgProf are available
at http://g4cpt.fnal.gov/LArSoft.html

#6 Updated by Soon Jun over 2 years ago

  • % Done changed from 50 to 80

1) presented ``Preliminary Profiling Results of LArSoft/ProtoDune''at the LArSoft coordination meeting on Dec. 5, 2017 (https://indico.fnal.gov/event/15868/).
2) Add profiling measurement for a DUNE-FD simulation and reconstruction work-flow (prodgenie_nue_dune10kt_1x2x6) of which results are available at
https://g4cpt.fnal.gov/larsoft/dunetpc_v06_57_00/igprof.html (Memory profiling by IgProf)
https://g4cpt.fnal.gov/larsoft/dunetpc_v06_57_00/hpctoolkit.html (CPU and other hardware counter profiling by HPCToolkit).

#7 Updated by Soon Jun over 2 years ago

  • % Done changed from 80 to 90

Preliminary rofiling procedure with IgProf and HPCToolkit for typical workflows of LArSoft/protoDune and Dune is summarized at
https://cdcvs.fnal.gov/redmine/projects/g4p/wiki/Application_with_a_large_fiducial_volume_of_LAr

#8 Updated by Soon Jun over 2 years ago

  • % Done changed from 90 to 100

Add a summary of instruction how to profile typical workflows of LArSoft/(proto)DUNE applications at
https://cdcvs.fnal.gov/redmine/projects/larsoft/wiki/ProtoDUNE_and_DUNE_Far_Detector_simulation_and_reconstruction_workflows_(v06_57_00)
which can be linked from the "LArSoft internals" page,
https://cdcvs.fnal.gov/redmine/projects/larsoft/wiki/LArSoft_Internals

Contact for further support or assistance.

#9 Updated by Soon Jun over 2 years ago

  • Status changed from Assigned to Closed


Also available in: Atom PDF