Project

General

Profile

Bug #23338

Exporting of archived run configurations became very slow after the upgrade to artdaq_database 1.4.75 and mongodb 4.0.2.

Added by Gennadiy Lukhanin about 2 months ago. Updated 13 days ago.

Status:
Closed
Priority:
Normal
Category:
artdaq-database
Target version:
-
Start date:
09/26/2019
Due date:
% Done:

100%

Estimated time:
Experiment:
-
Co-Assignees:
Duration:

Description

The latest version of artdaq_database 1.4.75 exports archived run configurations much slower than the previous version 1.4.65.

History

#1 Updated by Gennadiy Lukhanin about 2 months ago

artdaq_database uses regular expressions for finding documents that have been assigned to a run configuration or an archived run configuration. The version 4.x of MongoDB requires the “$options” field associated with such regular expressions, e.g. { ‘configuration.name’: { $regex: ‘^23234/TestConfig’}, $options:”m”}. If the $options value is set to “m” as suggested in the MongoDB docs, then database indexes are not used, and a full collection scan is performed. The $options field is mandatory. After experimenting with various approaches I realized, that if the $options filed is set to an empty string, then database indexes are being used for searching, and searching time drops to an order of a few milliseconds.

#2 Updated by Gennadiy Lukhanin about 2 months ago

  • % Done changed from 0 to 90
  • Status changed from New to Work in progress

#3 Updated by Gennadiy Lukhanin about 2 months ago

Please follow the steps below to test this bugfix.

  1. Install the e17:s82:prof variant of artdaq_demo v3_06_01.
  2. Download and install artdaq_database v1_4_77 with the same qualifiers from scisoft.
    1. https://scisoft.fnal.gov/scisoft/packages/artdaq_database/v1_04_77/
  3. Open two shells and setup artdaq_database in both of them.
    1. source [your-artdaq-products-area]/setup
    2. setup artdaq_database v1_04_77 -q e17:prof:s82
    3. export ARTDAQ_DATABASE_URI=mongodb://127.0.0.1/artdaqtest_db
  4. In the first shell start mongodb.
    1. mkdir -p $HOME/database/mongodb
    2. mongod --dbpath=$HOME/database/mongodb --bind_ip=127.0.0.1
    3. leave mongod running in the foreground
  5. Switch to the second shell, which will be used for running conftool.py and confirm that the test database is accessible.
    1. conftool.py readDatabaseInfo
  6. Restore the run history database from a recent backup.
    1. mkdir -p $HOME/database/backup/
      cd $HOME/database/backup/
      tfn=RunHistory.tar.gz.a && for f in  https://cdcvs.fnal.gov/redmine/attachments/download/5529{1/${tfn}e,2/${tfn}d,3/${tfn}c,4/${tfn}b,5/${tfn}a} ; do wget $f; done
      cat ./RunHistory.tar.gz.a* | tar xzvf -
      mongorestore --gzip  --host=127.0.0.1 --port=27017 --db=artdaqtest_db_archive ./cern_pddaq_db_archive
      

      Note: Depending on your server disk performance this step may take from 7 to 15 minutes to complete.
  7. Create a working directory where an archived run configuration will be exported.
    1. mkdir -p $HOME/database/work-dir/ee/
    2. cd $HOME/database/work-dir/ee/
  8. Export several archived configurations from the database.
    1. start=$(date +%s) && \
      config_list=(\
       8613/PhilTest_felix_hit_find_pinning_00005\
       8376/np04_CRT_noprescale_testdir_00004\
       1001739/np04_PM_noprescale_00001\
       8067/KurtTest_felix_hit_find_00012\
       7302/np04_CRT_noprescale_00004\
       6112/np04_WibsReal_Ssps_CRT_prescale3_00004 \
       459/Coldbox_SiPM25V_00009 ) && \
      for c in ${config_list[@]} ; do d=$(echo $c |tr '/' '.'); echo Exporting $c into $d; rm -rf $d; mkdir -p $d; cd $d; \
      conftool.py exportArchivedRunConfiguration $c ;\
      cd ..; done; \
      runtime_seconds=$(($(date +%s) - $start)) && \
      echo Downloaded ${#config_list[@]} configurations in $runtime_seconds seconds. && \
      echo Test summary: $( [[ $runtime_seconds -le 15  ]] && echo Good || echo Poor ) performance.
      
    2. Output example
      Exporting 8613/PhilTest_felix_hit_find_pinning_00005 into 8613.PhilTest_felix_hit_find_pinning_00005
      BULKDOWNLOADER is running on mu2edaq11 and ARTDAQ_DATABASE_URI=mongodb://127.0.0.1/artdaqtest_db_archive
      Downloaded 22 files with 28 threads in 65 msecs.
      Avarage file load time is 82 msecs.
      True
      Exporting 8376/np04_CRT_noprescale_testdir_00004 into 8376.np04_CRT_noprescale_testdir_00004
      BULKDOWNLOADER is running on mu2edaq11 and ARTDAQ_DATABASE_URI=mongodb://127.0.0.1/artdaqtest_db_archive
      Downloaded 42 files with 28 threads in 146 msecs.
      Avarage file load time is 97 msecs.
      True
      Exporting 1001739/np04_PM_noprescale_00001 into 1001739.np04_PM_noprescale_00001
      BULKDOWNLOADER is running on mu2edaq11 and ARTDAQ_DATABASE_URI=mongodb://127.0.0.1/artdaqtest_db_archive
      Downloaded 134 files with 28 threads in 2001 msecs.
      Avarage file load time is 418 msecs.
      True
      Exporting 8067/KurtTest_felix_hit_find_00012 into 8067.KurtTest_felix_hit_find_00012
      BULKDOWNLOADER is running on mu2edaq11 and ARTDAQ_DATABASE_URI=mongodb://127.0.0.1/artdaqtest_db_archive
      Downloaded 23 files with 28 threads in 76 msecs.
      Avarage file load time is 92 msecs.
      True
      Exporting 7302/np04_CRT_noprescale_00004 into 7302.np04_CRT_noprescale_00004
      BULKDOWNLOADER is running on mu2edaq11 and ARTDAQ_DATABASE_URI=mongodb://127.0.0.1/artdaqtest_db_archive
      Downloaded 133 files with 28 threads in 2769 msecs.
      Avarage file load time is 582 msecs.
      True
      Exporting 6112/np04_WibsReal_Ssps_CRT_prescale3_00004 into 6112.np04_WibsReal_Ssps_CRT_prescale3_00004
      BULKDOWNLOADER is running on mu2edaq11 and ARTDAQ_DATABASE_URI=mongodb://127.0.0.1/artdaqtest_db_archive
      Downloaded 139 files with 28 threads in 2996 msecs.
      Avarage file load time is 603 msecs.
      True
      Exporting 459/Coldbox_SiPM25V_00009 into 459.Coldbox_SiPM25V_00009
      BULKDOWNLOADER is running on mu2edaq11 and ARTDAQ_DATABASE_URI=mongodb://127.0.0.1/artdaqtest_db_archive
      Downloaded 6 files with 28 threads in 31 msecs.
      Avarage file load time is 144 msecs.
      True
      Downloaded 7 configurations in 9 seconds.
      Test summary: Good performance.
      
    3. Confirm that the last line reads "Good performance".

#4 Updated by Gennadiy Lukhanin about 2 months ago

  • % Done changed from 90 to 100
  • Status changed from Work in progress to Resolved

#5 Updated by John Freeman about 1 month ago

  • Status changed from Resolved to Reviewed

I ran through Gennadiy's thorough instructions on mu2edaq01 and found everything worked perfectly. Output (also saved in mu2edaq01:/home/jcfree/Documents/issue23338_test.script) was as follows:

Exporting 8613/PhilTest_felix_hit_find_pinning_00005 into 8613.PhilTest_felix_hit_find_pinning_00005
BULKDOWNLOADER is running on mu2edaq01 and ARTDAQ_DATABASE_URI=mongodb://127.0.0.1/artdaqtest_db_archive
Downloaded 22 files with 28 threads in 342 msecs.
Avarage file load time is 435 msecs.
True
Exporting 8376/np04_CRT_noprescale_testdir_00004 into 8376.np04_CRT_noprescale_testdir_00004
BULKDOWNLOADER is running on mu2edaq01 and ARTDAQ_DATABASE_URI=mongodb://127.0.0.1/artdaqtest_db_archive
Downloaded 42 files with 28 threads in 641 msecs.
Avarage file load time is 427 msecs.
True
Exporting 1001739/np04_PM_noprescale_00001 into 1001739.np04_PM_noprescale_00001
BULKDOWNLOADER is running on mu2edaq01 and ARTDAQ_DATABASE_URI=mongodb://127.0.0.1/artdaqtest_db_archive
Downloaded 134 files with 28 threads in 2492 msecs.
Avarage file load time is 520 msecs.
True
Exporting 8067/KurtTest_felix_hit_find_00012 into 8067.KurtTest_felix_hit_find_00012
BULKDOWNLOADER is running on mu2edaq01 and ARTDAQ_DATABASE_URI=mongodb://127.0.0.1/artdaqtest_db_archive
Downloaded 23 files with 28 threads in 261 msecs.
Avarage file load time is 317 msecs.
True
Exporting 7302/np04_CRT_noprescale_00004 into 7302.np04_CRT_noprescale_00004
BULKDOWNLOADER is running on mu2edaq01 and ARTDAQ_DATABASE_URI=mongodb://127.0.0.1/artdaqtest_db_archive
Downloaded 133 files with 28 threads in 3448 msecs.
Avarage file load time is 725 msecs.
True
Exporting 6112/np04_WibsReal_Ssps_CRT_prescale3_00004 into 6112.np04_WibsReal_Ssps_CRT_prescale3_00004
BULKDOWNLOADER is running on mu2edaq01 and ARTDAQ_DATABASE_URI=mongodb://127.0.0.1/artdaqtest_db_archive
Downloaded 139 files with 28 threads in 3649 msecs.
Avarage file load time is 735 msecs.
True
Exporting 459/Coldbox_SiPM25V_00009 into 459.Coldbox_SiPM25V_00009
BULKDOWNLOADER is running on mu2edaq01 and ARTDAQ_DATABASE_URI=mongodb://127.0.0.1/artdaqtest_db_archive
Downloaded 6 files with 28 threads in 181 msecs.
Avarage file load time is 844 msecs.
True
Downloaded 7 configurations in 13 seconds.
Test summary: Good performance.

#6 Updated by Gennadiy Lukhanin 13 days ago

  • Status changed from Reviewed to Closed


Also available in: Atom PDF