Matthew, Chris, Kanika, Susan, Nick, Dominick, Bruno, Nate, Satish, Ken Herner, Craig, Ryan, Gareth, Michael G., Michael Kirby, Tanya, Paola.
- Matthew will be away next week, Dominick will chair the Thursday meeting in his stead (13/11/14), there will be no Monday meeting.
- Follow up on the output verbosity discussion - we want to remove the traces and increase the message service verbosity. Chris will figure this out and he’ll communicate this to Nate.
- Paola has implemented the log file management policy we discussed the other week on novagpvn01. Files older than 14 days are zipped in place and files older than 3 months are deleted.
- Nothing has happened with the meta-data format feature request - Matthew will follow up on that.
Jobsub migration (Ken Herner)¶
CD would like us to update to the new jobsub client. The differences between this client and our existing one are detailed in the slide. The slides also contain links to overall FIFE documentation. The main difference is that the new job sub client supports the use of multiple servers (not like now where we’re pinned to gpsn01 and bandwidth limited as a result). Another major difference is in the handling of output log files. These are now not put on bluearc by default, instead the user must request them. CD would like us to transition before the end of the year.
There are a couple of missing features which mean that production is not able to fully transition at this time, such as missing support group accounts (novapro) and kcron certificates (which is apparently now working). However it was decided that we could begin a staged transition now. This transition will begin with Dominick and Satish changing their submission modes to use this. We will also update Satish's common submission scripts to default to jobsub client then point users at these. In the longer term, Jeny and Paola can transition when the group account features are in place.
Assorted other items that were discussed as part of this include:
- Chris asked about the status of dags? Nate says we don’t use these anymore. Apparently these do work with the new job sub anyway.
- Nate pointed out that the lack of backwards compatibility is a big hurdle for the NOvA transition and there will be lots of man hours needed to check and debug this change. Now is not an ideal time for this due to the impending first analysis.
- If we wanted to automatically fetch our logs back we'd need to write something to do this.
- A number of people pointed out that it would be nice to have a webservice for accessing logs.
- Dominick asked about a Python api? Apparently this doesn’t exist.
- It was noted that all exporting and nova specific options have made the migration and should work as before.
PID progress (Gavin)¶
FD MC cry and genie have all been LEM sum and pid-parted. Will be PID'ed now. He'll run genie first then the CRY.
DB issues? (Jon)¶
The reported DB issues look to be resolved with a change to the back-end cache system. We will see how this change stands up during the next push. Jon noted that he's in contact with the DB admins - who actually don’t administer them, they just monitor. Jon also noted that there are resources allocated for interactive use which could be used for debugging/quick jobs. These are based on the DBI (database interface) detecting that you aren’t running on the grid. Chris is worried that this might not pick up all offsite jobs, as it’s based on detecting condor systems For example Caltech doesn’t use these.
Geojittered v2 samples (Nate)¶
Done and sent to Satish.
ND bad channels tag (Jon P. et al)¶
v4 of bad channels for ND and FD should be available now. This means we can now request a specific fixed tag of bad channels for use in reconstruction. To do this in an existing release we'd need a back port to the fcl file only. It was decided that the use of a tagged version of bad channels should go in with the next pass of reco. To do this completely we'd need to close off a dataset, then make bad channels then add in fcl hooks, or alternatively when we tag release a release we could tag validity tables and use those. The conclusion of the group was that this will need care to do properly when we do the final pass.
- ND calibration is finished, so we now need to tag then run ND reco.
- Nate ND MC: he has been having SAMweb problems where the fcl files fail to be retrieved by SAM - do we want to make more