Project

General

Profile

Updating the UPS Product » History » Version 26

Serdar Kurbanov, 08/31/2016 03:56 PM

1 1 Martin Frank
h1. Updating the UPS Product
2 1 Martin Frank
3 1 Martin Frank
This page describes how to build a new version of the novaddt UPS product together with the packages that it depends on and how to roll these new products out on the detectors.
4 1 Martin Frank
5 1 Martin Frank
h2. Building the Products
6 1 Martin Frank
7 2 Serdar Kurbanov
Here's how to build NOvA DDT products
8 2 Serdar Kurbanov
9 7 Serdar Kurbanov
0. Go to /nova/app/users/<user> without executing setup_nova
10 7 Serdar Kurbanov
> cd /nova/app/users/<user>
11 7 Serdar Kurbanov
12 3 Serdar Kurbanov
1. Make changes and commit it to trunk
13 9 Serdar Kurbanov
<pre>
14 9 Serdar Kurbanov
<changes changes changes...>
15 9 Serdar Kurbanov
svn commit <some_file.cc>
16 9 Serdar Kurbanov
</pre>
17 1 Martin Frank
18 1 Martin Frank
2. Checkout NOvA DDT release:
19 9 Serdar Kurbanov
<pre>
20 9 Serdar Kurbanov
svn co svn+ssh://p-novaddt@cdcvs.fnal.gov/cvs/projects/novaddt/novaddt.svn/trunk novaddt
21 9 Serdar Kurbanov
</pre>
22 1 Martin Frank
23 1 Martin Frank
3. Renaming version in trunk:
24 8 Serdar Kurbanov
<pre>
25 8 Serdar Kurbanov
cd <your build dir>/novaddt/ups
26 8 Serdar Kurbanov
chmod +x ups_product_tag.sh 
27 8 Serdar Kurbanov
./ups_product_tag.sh <new version> (example: ./ups_product_tag.sh v03_04_06)
28 8 Serdar Kurbanov
</pre>
29 1 Martin Frank
30 15 Serdar Kurbanov
4. Build novaddt with mrb
31 13 Serdar Kurbanov
<pre>
32 13 Serdar Kurbanov
mkdir <builddir>
33 13 Serdar Kurbanov
cd <builddir>
34 13 Serdar Kurbanov
. /grid/fermiapp/products/nova/externals/setup
35 13 Serdar Kurbanov
setup mrb
36 13 Serdar Kurbanov
export MRB_PROJECT=nova
37 13 Serdar Kurbanov
mrb newDev -v <new version> -q e6:on:prof (example: mrb newDev -v v03_04_06 -q e6:on:prof)
38 13 Serdar Kurbanov
source localProducts_nova_<new version>_e6_on_prof/setup (example: source localProducts_nova_v03_04_06_e6_on_prof/setup)
39 13 Serdar Kurbanov
cd srcs
40 13 Serdar Kurbanov
svn co svn+ssh://p-novaddt@cdcvs.fnal.gov/cvs/projects/novaddt/novaddt.svn/tags/<new version>  novaddt (example: svn co svn+ssh://p-novaddt@cdcvs.fnal.gov/cvs/projects/novaddt/novaddt.svn/tags/v03_04_06  novaddt)
41 1 Martin Frank
>> [
42 6 Serdar Kurbanov
>> if there's a problem with some dependencies, say *novaddtdeps*, you'll need to do ... and then go back to *mrbsetenv* step
43 6 Serdar Kurbanov
>> useful commands:
44 6 Serdar Kurbanov
>> ups list -aK+ <package name> (example: ups list -aK+ <novaddtdeps>) - to list versions of that package available
45 6 Serdar Kurbanov
>> 
46 6 Serdar Kurbanov
>> ]
47 21 Serdar Kurbanov
mrb uc
48 15 Serdar Kurbanov
cd $MRB_BUILDDIR
49 13 Serdar Kurbanov
mrbsetenv
50 13 Serdar Kurbanov
mrb i -j8 --tee
51 13 Serdar Kurbanov
</pre>
52 1 Martin Frank
53 9 Serdar Kurbanov
5. Test if new version works as intended
54 12 Serdar Kurbanov
<pre>
55 12 Serdar Kurbanov
cd <your build dir>
56 12 Serdar Kurbanov
. /grid/fermiapp/products/nova/externals/setup
57 12 Serdar Kurbanov
export PRODUCTS=<your build dir>/localProducts_nova_<new version>_e6_on_prof/:$PRODUCTS
58 12 Serdar Kurbanov
cd ddt_release_02.09.2015/product/
59 12 Serdar Kurbanov
setup novaddt <new version> -q e6:on:prof (example: setup novaddt v03_04_06 -q e6:on:prof)
60 1 Martin Frank
>> uncomment DDTGlobalConfiguration-FD.fcl after "# if using nova binary:"
61 12 Serdar Kurbanov
nova -c srcs/novaddt/DDTGlobalConfigurations/DDTGlobalConfiguration-FD.fcl
62 12 Serdar Kurbanov
</pre> 
63 1 Martin Frank
64 1 Martin Frank
65 1 Martin Frank
66 1 Martin Frank
h2. Deploying the Products
67 1 Martin Frank
68 1 Martin Frank
Install build to FD.
69 1 Martin Frank
70 15 Serdar Kurbanov
1. Ssh to FD server
71 15 Serdar Kurbanov
<pre>
72 1 Martin Frank
ssh novadaq@novadaq-far-master
73 15 Serdar Kurbanov
</pre>
74 5 Serdar Kurbanov
75 1 Martin Frank
2. Copy new build ot FD server (from gpvm node, from your build directory)
76 15 Serdar Kurbanov
<pre>
77 12 Serdar Kurbanov
rsync -avP localProducts_nova_v03_04_06_e6_on_prof/novaddt novadaq@novadaq-far-master:/ddt/externals/.
78 15 Serdar Kurbanov
</pre>
79 12 Serdar Kurbanov
80 1 Martin Frank
3. Check that code works
81 15 Serdar Kurbanov
<pre>
82 13 Serdar Kurbanov
. /ddt/externals/setup
83 13 Serdar Kurbanov
setup novaddt v03_04_06 -q e6:on:prof
84 15 Serdar Kurbanov
>> (a bit questionable) nova -c srcs/novaddt/DDTGlobalConfigurations/DDTGlobalConfiguration-FD.fcl
85 15 Serdar Kurbanov
</pre>
86 11 Serdar Kurbanov
87 18 Serdar Kurbanov
h2. Start changes in control room
88 11 Serdar Kurbanov
> (Make track of what you'r doing n ECL)
89 11 Serdar Kurbanov
> Stop run
90 16 Serdar Kurbanov
> Start DAQ config editor on Monitor1
91 11 Serdar Kurbanov
> Config is FarDetGlobalConfigP1
92 11 Serdar Kurbanov
> Change build version manually
93 11 Serdar Kurbanov
> Enter
94 1 Martin Frank
> Start new run
95 18 Serdar Kurbanov
96 18 Serdar Kurbanov
97 19 Serdar Kurbanov
h2. Building with Jenkins
98 18 Serdar Kurbanov
99 18 Serdar Kurbanov
Using Jenkins simplifies build process.
100 18 Serdar Kurbanov
101 18 Serdar Kurbanov
1. Get certificate to enter Jenkins if you don't have one
102 20 Serdar Kurbanov
> General instructions: https://fermi.service-now.com/kb_view_customer.do?sysparm_article=KB0010657
103 20 Serdar Kurbanov
> Get KCA server CA (https://security.fnal.gov/pki/fermi-kca-ca.pemcertificate)
104 20 Serdar Kurbanov
> Get KCA certificate. For SL6 x86 this link helped me, since yum install krb5-fermi-getcert didn't work. Install packages krb5-fermi-*. http://ftp.scientificlinux.org/linux/fermi/slf6x/i386/os/FermiPackages/
105 18 Serdar Kurbanov
106 18 Serdar Kurbanov
2. Go to Jenkins
107 18 Serdar Kurbanov
> https://buildmaster.fnal.gov/view/Nova/
108 18 Serdar Kurbanov
109 18 Serdar Kurbanov
3. Go to build configuration
110 18 Serdar Kurbanov
> NOvA -> novaddt_build -> Configure
111 18 Serdar Kurbanov
112 18 Serdar Kurbanov
4. Make changes in version. For unclear dependensies go to novaddt/novaddtdeps repo -> tags -> tag you search -> ups -> product_deps
113 18 Serdar Kurbanov
114 18 Serdar Kurbanov
5. Make build (build with parameters)
115 18 Serdar Kurbanov
116 18 Serdar Kurbanov
6. Debug if any problem appears
117 18 Serdar Kurbanov
118 18 Serdar Kurbanov
119 25 Serdar Kurbanov
h2. Deploying products after Jenkins at FD
120 18 Serdar Kurbanov
121 18 Serdar Kurbanov
Do same steps as deploying after mrb build, but downloading to FD server looks differently:
122 18 Serdar Kurbanov
1. Ssh to FD server
123 18 Serdar Kurbanov
<pre>
124 18 Serdar Kurbanov
ssh novadaq@novadaq-far-master
125 18 Serdar Kurbanov
</pre>
126 18 Serdar Kurbanov
127 18 Serdar Kurbanov
2. Get link from Jenkins 
128 18 Serdar Kurbanov
> Jenkins->novaddt_build_output->Last successful atrifacts->link for zip)
129 18 Serdar Kurbanov
130 18 Serdar Kurbanov
3. Download build to download area
131 18 Serdar Kurbanov
<pre>
132 25 Serdar Kurbanov
cd <download folder> (/ddt/externals/DOWNLOAD for FD)
133 18 Serdar Kurbanov
wget <link from Jenkins>
134 18 Serdar Kurbanov
</pre>
135 18 Serdar Kurbanov
136 18 Serdar Kurbanov
4. Unzip and untar link from Jenkins to /ddt/externals
137 18 Serdar Kurbanov
138 18 Serdar Kurbanov
5. Check that code works
139 18 Serdar Kurbanov
140 1 Martin Frank
6. Go to control room to make changes
141 1 Martin Frank
142 25 Serdar Kurbanov
143 25 Serdar Kurbanov
h2. Deploying products after Jenkins at ND
144 25 Serdar Kurbanov
145 25 Serdar Kurbanov
1. Ssh to ND server
146 25 Serdar Kurbanov
<pre>
147 25 Serdar Kurbanov
ssh novadaq@novadaq-near-master
148 25 Serdar Kurbanov
</pre>
149 25 Serdar Kurbanov
150 25 Serdar Kurbanov
2. Get link from Jenkins 
151 25 Serdar Kurbanov
> Jenkins->novaddt_build_output->Last successful atrifacts->link for zip)
152 25 Serdar Kurbanov
153 25 Serdar Kurbanov
3. Download build to download area
154 25 Serdar Kurbanov
<pre>
155 26 Serdar Kurbanov
cd <download folder> (/nova/novaddt/ups/DOWNLOAD)
156 25 Serdar Kurbanov
wget <link from Jenkins>
157 25 Serdar Kurbanov
</pre>
158 25 Serdar Kurbanov
159 26 Serdar Kurbanov
4. Unzip and untar link from Jenkins to /nova/novaddt/ups
160 25 Serdar Kurbanov
161 25 Serdar Kurbanov
5. Check that code works
162 25 Serdar Kurbanov
163 25 Serdar Kurbanov
6. Go to control room to make changes
164 23 Serdar Kurbanov
165 23 Serdar Kurbanov
166 22 Serdar Kurbanov
h2. Extended version of how to deploy products to FD
167 22 Serdar Kurbanov
168 22 Serdar Kurbanov
1. Upload new build to novadaq@novadaq-far-master
169 22 Serdar Kurbanov
170 22 Serdar Kurbanov
2. svn is not installed on novadaq@novadaq-far-master. You need to use novadaq@novadaq-far-master-02 to use svn to update DDTGlobalConfiguration-FD.fcl
171 22 Serdar Kurbanov
Note: you may need to log in to novadaq@novadaq-far-master-02 through novadaq@novadaq-far-master, since direct access is not allowed for some reason.
172 22 Serdar Kurbanov
173 22 Serdar Kurbanov
3. Update fcl using instructions from instruction on updating GlobalConfiguration
174 22 Serdar Kurbanov
175 22 Serdar Kurbanov
4. Pick a buffer node which is in the run (see DAQ application manager or ask RunCoordinator if needed) and log in there. By default, node 70 is ok.
176 22 Serdar Kurbanov
<pre>
177 22 Serdar Kurbanov
ssh novadaq@novadaq-far-farm-70
178 22 Serdar Kurbanov
</pre>
179 1 Martin Frank
180 23 Serdar Kurbanov
5. Copy fcl file (it gives you name of the file after step 3) to your directory on ssh novadaq@novadaq-far-farm-70 and change it so it doens't run in verbose and not producing trigger messages.
181 22 Serdar Kurbanov
182 22 Serdar Kurbanov
6. Make setup
183 22 Serdar Kurbanov
<pre>
184 22 Serdar Kurbanov
. /ddt/externals/setup
185 22 Serdar Kurbanov
setup novaddt <version> -q <qualifier> (for example: setup novaddt v03_09_02 -q e6:on:prof)
186 22 Serdar Kurbanov
</pre>
187 22 Serdar Kurbanov
188 22 Serdar Kurbanov
7. Interprocess communication shell setup
189 22 Serdar Kurbanov
<pre>
190 22 Serdar Kurbanov
ipcs (it shows you shared memory segments, you need to pick up most used one)
191 22 Serdar Kurbanov
export SHMRW_KEY=0x0001700F
192 1 Martin Frank
</pre>
193 23 Serdar Kurbanov
194 23 Serdar Kurbanov
8. Run your fcl using command allowing to connect to shared memory segment
195 23 Serdar Kurbanov
<pre>
196 23 Serdar Kurbanov
ddt-filter -c <DDTGlobalConfiguration.fcl in your folder>
197 23 Serdar Kurbanov
</pre>
198 23 Serdar Kurbanov
199 23 Serdar Kurbanov
9. If it runs that means it works. You want to make sure you see event-by-enent messages that says you it's looking at events
200 23 Serdar Kurbanov
201 23 Serdar Kurbanov
10. Stop the run,  go to DAQConfigurationManager -> DDTConfiguration -> change fcl and ddt version to latest one -> restart the run
202 24 Serdar Kurbanov
203 24 Serdar Kurbanov
204 24 Serdar Kurbanov
h2. Accesing DB to see versions of DDTGlobalConfiguration.fcl versions used for run
205 24 Serdar Kurbanov
206 24 Serdar Kurbanov
(See https://cdcvs.fnal.gov/redmine/projects/novadaq-config-mgmt/wiki/Useful_Commands_and_Files_for_DB_Configuration for more info)
207 24 Serdar Kurbanov
208 24 Serdar Kurbanov
<pre>
209 24 Serdar Kurbanov
setup_online
210 24 Serdar Kurbanov
source `which setupNovaDBEnvVars.sh`
211 24 Serdar Kurbanov
runDBQuery "select * from fardet.ddt_manager_connect_parameters order by id desc" 
212 24 Serdar Kurbanov
</pre>