NOvA - raw2root - memory issue when large file size are processed
When NOvA software try to raw2root files larger that about 2.5 Gb the memory usage exceed the limit allowed on worker nodes.
Looking for all FD and ND files, including all streams, the larger file processed through raw2root have e size of about 2.6 Gb, they are
fardet_r00018614_s35_t02.raw - File Size: 2823953284
fardet_r00018232_s01_t02.raw - File Size: 2810664080
Looking for all FD and ND files newer than Oct 30 2014 (post shutdown(?)), including all streams and that don't have child in artdaq tier (it means these files haven't been successfully processed through raw2root), we have the files:
fardet_r00018315_s00_t04.raw - File Size: 2823796320
fardet_r00018315_s02_t04.raw - File Size: 2828028240
fardet_r00018315_s04_t04.raw - File Size: 2830475184
fardet_r00019390_s03_t04.raw - File Size: 2851532576
fardet_r00018200_s04_DDslowmono.raw - File Size: 2869736884
fardet_r00018314_s01_t04.raw - File Size: 2881578160
fardet_r00018314_s06_t04.raw - File Size: 2956135416
fardet_r00018992_s10_t02.raw - File Size: 2964850948
fardet_r00018367_s01_t02.raw - File Size: 2971664640
and more with larger size.
The limit of the file size for a raw file to be raw2root is about 2.63Gb (~2823940997b).
On the end of April 2015 we agreed with NOvA people to exclude from raw2root process the files in Stream 4 (i.e. with t04 in their name).
However some files produced on May 7th by stream "all" had a large size that prevented them to be successfully processed by raw2root. The list of those file is:
fardet_r00019523_s21_DDenergy.raw - File Size: 4963184128
fardet_r00019523_s22_DDenergy.raw - File Size: 3610772648
fardet_r00019523_s23_DDenergy.raw - File Size: 3427006428
fardet_r00019523_s24_DDenergy.raw - File Size: 3309056156
NOvA people are aware of that.