Project

General

Profile

Task #13885

Move raw and processed data to per-run directories

Added by Arthur Kreymer about 3 years ago. Updated almost 2 years ago.

Status:
Work in progress
Priority:
High
Start date:
09/19/2016
Due date:
10/01/2016
% Done:

100%

Estimated time:
10.00 h
Spent time:
Duration: 13

Description

We are approaching 32K files in one directory in PNFS, too many.

Per discussions are weekly ops and Annie meetins,
we will move to one directory per Run.

reorganize.sh (3.18 KB) reorganize.sh Steven Gardiner, 09/19/2016 06:03 PM

History

#1 Updated by Arthur Kreymer about 3 years ago

  • % Done changed from 0 to 10

Counted daq files and raw files, evaluated runs.

RAWS=`/bin/ls /pnfs/annie/persistent/raw`
echo ${RAWS} | wc -w
31198

RFS=`printf "${RAWS}\n" | grep ^RAW`
echo ${RFS} | wc -w
11627

RUNS=`for RAW in ${RFS} ; do
echo ${RAW} | cut -f 3 -d R | cut -f 1 -d S
done | sort -nu `

echo ${RUNS} | wc -w

-bash-4.1$ echo ${RUNS} | wc -w
135

-bash-4.1$ echo ${RUNS}
58 62 68 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 116 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 172 173 174 175 176 177 178 179 180 181 182 183 184 186 188 189 190 191 192 194 195 196 197 198 199 200 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 234 235 236 237 238 243 244 245 246 247

#2 Updated by Steven Gardiner about 3 years ago

Wrote script (/annie/app/users/gardiner/reorganize.sh) that simulates the reorganization without actually moving anything around yet. The script handles the 3 data file types (^RAWData, ^Data, and ^V2Data) separately.

We'll need to finalize how we want to organize the files before running the script "for real."

#3 Updated by Matthew Wetstein almost 2 years ago

  • % Done changed from 50 to 100


Also available in: Atom PDF