Instructions for Developers » History » Version 111
« Previous -
Version 111/141
(diff) -
Next » -
Current version
Fabio Ravera, 11/15/2018 12:46 PM
Instructions for Developers of OTSDAQ¶
Note: we use MRB.
MRB is very useful as it allows users to update the entire dependency tree and only rebuild as needed (instead of having to rebuild the changes and then all dependent products). However, MRB does not support installing to a UPS product area, so it is most useful for development and testing, not for creating release packages and tarballs.
The quick start install script is here: https://cdcvs.fnal.gov/redmine/projects/otsdaq/repository/demo/revisions/develop/raw/tools/quick-mrb-start.sh (https://goo.gl/rm1Ir1)
MRB: mrb
Troubleshooting your otsdaq Installation
Migrating to artdaq database utilities
Set up your products area:¶
Note: The products area has already been setup for you if you are on rulinux03!
To speed things up, you may try to install manually the products:
mkdir mkdir /path/to/products/area # Should be DIFFERENT than where you intend to install otsdaq (for example: otsdaqPrj/otsdaq-mrb-base for otsdaq and otsdaqPrj/products for the products) cd /path/to/products/area # e.g. /data/ups export PRODUCTS=$PWD wget https://cdcvs.fnal.gov/redmine/projects/otsdaq/repository/utilities/revisions/develop/raw/tools/ProductsUpdate.py python ProductsUpdate.py
Alternatively you can use the products saved in the fermilab cvmfs:
yum install https://ecsft.cern.ch/dist/cvmfs/cvmfs-release/cvmfs-release-latest.noarch.rpm yum clean all yum install cvmfs cvmfs-config-default
create the file /etc/cvmfs/default.d/70-artdaq.conf and add the lines
CVMFS_REPOSITORIES=fermilab.opensciencegrid.org
CVMFS_HTTP_PROXY=DIRECT
Then you can refresh the cvmfs configuration:
cvmfs_config setup
Check if CernVM-FS mounts the specified repositories by cvmfs_config probe. If the probe fails, try to restart autofs with sudo service autofs restart.
Otherwise... (here is the link to artdaq bundles)¶
mkdir /path/to/products/area # Should be DIFFERENT than where you intend to install otsdaq (for example: otsdaqPrj/otsdaq-mrb-base for otsdaq and otsdaqPrj/products for the products) cd /path/to/products/area # e.g. /data/ups Products=$PWD # Match the UPS version to your machine's architecture, SLF6 or SLF7 should work. See http://scisoft.fnal.gov/scisoft/packages/ups for latest versions and architectures in the manifest subfolder. wget http://scisoft.fnal.gov/scisoft/bundles/tools/pullProducts chmod +x pullProducts #go here to determine latest artdaq_demo version v#_##_##: scisoft.fnal.gov/scisoft/bundles/artdaq_demo/ #check your local products folder name for the qualifiers (e.g. ots/localProducts_otsdaq_demo_v2_01_00_e15_s64_prof ==> implies s64-e15) ./pullProducts . slf6 artdaq_demo-v#_##_## s41-e10 prof #for example, ./pullProducts . slf7 artdaq_demo-v3_03_00 s64-e15 prof curl http://scisoft.fnal.gov/scisoft/packages/mrb/v1_09_00/mrb-1.09.00-noarch.tar.bz2|tar -jx #curl http://scisoft.fnal.gov/scisoft/packages/xerces_c/v3_1_4a/xerces_c-3.1.4a-sl7-x86_64-e10-prof.tar.bz2|tar -jx curl http://scisoft.fnal.gov/scisoft/packages/xerces_c/v3_1_4a/xerces_c-3.1.4a-slf6-x86_64-e10-prof.tar.bz2|tar -jx #curl http://scisoft.fnal.gov/scisoft/packages/cetbuildtools/v5_06_03/cetbuildtools-5.06.03-noarch.tar.bz2|tar -jx rm -f *.bz2 *.txt
Install system dependencies¶
sudo yum install -y libuuid-devel openssl-devel
Fetch code from the repository¶
cd my/mrb/working/directory (or mkdir my/mrb/working/directory if it doesn't exist)# change to the path to your specific mrb project dir bash source $Products/setup # e.g. /data/ups/setup, Products set in previous step Base=$PWD setup git setup gitflow setup mrb # # NOTE: You must have a UPS area that you have run ./pullProducts . slf6 artdaq-v1_13_00 s35-e10 prof (slf6 artdaq-v1_13_00 s35-e10 prof) in! # You must also have extracted the XDAQ UPS product found in the otsdaq-demo repo to this UPS area! # This has already been done if you are developing on rulinux03! # export MRB_PROJECT=otsdaq mrb newDev -v v2_02_00 -q e15:s64:prof . $Base/local*/setup cd $MRB_SOURCE
If you have write access to the repositories:
mrb gitCheckout ssh://p-otsdaq@cdcvs.fnal.gov/cvs/projects/otsdaq mrb gitCheckout -d otsdaq_demo ssh://p-otsdaq@cdcvs.fnal.gov/cvs/projects/otsdaq-demo mrb gitCheckout -d otsdaq_utilities ssh://p-otsdaq@cdcvs.fnal.gov/cvs/projects/otsdaq-utilities mrb gitCheckout -d otsdaq_components ssh://p-components@cdcvs.fnal.gov/cvs/projects/components #this last package has been removed #mrb gitCheckout -b lukhanin-wip2 -d artdaq_database http://cdcvs.fnal.gov/projects/artdaq-utilities-database # Needed for now, should be removed soon
Otherwise for read-only access to the repositories:
mrb gitCheckout -b develop http://cdcvs.fnal.gov/projects/otsdaq mrb gitCheckout -b develop -d otsdaq_demo http://cdcvs.fnal.gov/projects/otsdaq-demo mrb gitCheckout -b develop -d otsdaq_utilities http://cdcvs.fnal.gov/projects/otsdaq-utilities mrb gitCheckout -b develop -d otsdaq_components http://cdcvs.fnal.gov/projects/components
Finally, build the products¶
# For building TRACE alongside... #mrb svnCheckout svn+ssh://p-trace@cdcvs.fnal.gov/cvs/projects/trace-svn/trunk #mv trunk TRACE #sed -i 's/trunk/TRACE/g' CMakeLists.txt #echo 'include_directories( ${CMAKE_CURRENT_SOURCE_DIR}/TRACE/include )' >>CMakeLists.txt
If you're setting up your own products area:
# Extract the XDAQ product shipped with OTSDAQ, as well as the special version of ARTDAQ cd $Products for file in $MRB_SOURCE/otsdaq_demo/tarballs/*.bz2;do tar -xf $file;done
Then:
cd $MRB_BUILDDIR # A template setup script can be copied from the repo; # it should work, untouched, if you installed to ~/ots/ # cp srcs/otsdaq/tools/setup_mrb.sh . # . setup_mrb.sh # # If you installed somewhere different from ~/ots/ you can edit the file # or proceed with commands manually... source mrbSetEnv setup nodejs v4_5_0 export CETPKG_J=24 mrb build # VERBOSE=1 <hit enter>
To rebuild after making changes to the source:¶
(Note: you can cd to any source directory)
. $Base/local*/setup source mrbSetEnv setup nodejs v4_5_0 mrb b
To do a clean build:¶
. $Base/local*/setup source mrbSetEnv setup nodejs v4_5_0 mrb z;source mrbSetEnv;mrb b
To update/pull the source:¶
. $Base/local*/setup mrb updateSource setup nodejs v4_5_0 mrb z;source mrbSetEnv;mrb b
Before Running:¶
If you are using your own Data directory, and not the one provided through MRB to avoid git conflicts, you must set the USER_DATA environment variable before running.
For example, first copy the configuration provided through MRB:
cp -r /my/mrb/working/directory/srcs/otsdaq_demo/Data /my/new/config/directory/NoGitData export USER_DATA=/full/path/to/new/config/directory/NoGitData
To run:¶
cd my/mrb/working/directory # change to the path to your specific mrb project dir Base=$PWD setup mrb # $MRB_DIR used in next setup script . $Base/local*/setup # "base" variable used by this script and is unset; MRB_BUILDDIR, etc set. source mrbSetEnv # add (most) BUILDDIR dirs to PATHs # Repeat the above steps after opening a new terminal to resume development setup nodejs v4_5_0 StartOTS.sh
Setting up MRB for each new terminal session:¶
source /path/to/products/area/setup # e.g. /data/ups/setup cd my/mrb/working/directory # this is where localProducts_otsdaq_* and srcs resides setup mrb setup git setup nodejs v4_5_0 source local*/setup mrb updateSource source mrbSetEnv export USER_DATA=/full/path/to/new/config/directory/NoGitData
To check-in/push your changes (into all ots respositories at once):¶
#from any path . my/mrb/working/directory/srcs/otsdaq_utilities/otsdaq-utilities/checkinAll.sh "<your comment for check-in>" #verify push more checkinAll.log
For making builds (i.e. new localProducts* folder) of other qualifier sets:¶
# Supported qualifiers: e10:s44:prof, e10:s44:debug, e10:s43:prof, e10:s43:debug, e10:s41:prof, e10:s41:debug #setup your current/previous local products mrb newDev -v v1_00_05 -q e10:s41:prof -f unsetup_all setup mrb source localProducts_<New qualifier set>/setup mrb z;mrbsetenv;mrb b
OTSDAQ Repositories¶
Repository Descriptions¶
Respository Tag | Description |
otsdaq | Main repository containing the code OTSDAQ uses to manage the flow of data |
otsdaq-demo | Contains the "user-servicable" parts, specifically the classes used to communicate with hardware. Also contains useful scripts for controlling the software. |
otsdaq-web | The web interface used for generating project files |
otsdaq-firmware | Pre-built firmware blocks for use on FPGA-based experiment hardware |
otsdaq-utilities | Software components not directly related to data-taking, such as the Logbook, Chat, and Configuration GUI |
Read-Only Mode Checkout Instructions¶
git clone http://cdcvs.fnal.gov/projects/otsdaq(,-firmware,-utilities,-web,-demo)
Read-Write Mode Checkout Instructions¶
git clone ssh://p-otsdaq@cdcvs.fnal.gov/cvs/projects/otsdaq(,-firmware,-utilities,-web,-demo)
To check out an individual repository (for example "firmware"):
git clone ssh://p-otsdaq@cdcvs.fnal.gov/cvs/projects/otsdaq-firmware
Using and developing otsdaq-web¶
Running otsdaq-web the ots_developer account on rulinux03:
A cron job for keeping the server up runs every 15 minutes on rulinux03 under the ots_developer account. Currently, the master repository of otsdaq_web is checked out. If you wish to run the server manually:
ksu ots_developer cd ~/mrbots/srcs/otsdaq_web git checkout sandbox-stable ./watch_server/killServer.sh ./watch_server/StartServer.sh #output is saved to ./watch_server/OtsServer.log
This script creates a PID file for the node process which will run the server. If you try to run while the server is already running, an error will be displayed.
Manually kill the process with
kill $(cat ./watch_server/OtsServer.pid)
or by simply running the script as shown above.
killServer.sh
If you want to edit the cron job use this command:
crontab -e
If you want to edit the Device Picker (choices for the step-by-step features) menu modify this file:
Server/device_picker/device_library.json
If you want to edit the Cactus Picker (choices for the firmware library features) menu modify the cactus folders of the otsdaq-firmware repository (currently the develop branch is being used).