Anna M, Steve W, Marc M, Robert I, Brandon W, Vladimir P. Yuyi G, Margherita VW.
- News (Anna)
- v2_3_0 update (All)
We are having a "Data Challenge 1 (DC1) for ProtoDUNE SP: processing files coming from CERN and testing the whole data flow CERN -> FNAL -> process to FERMI Grid -> ENSTORE.
For now we are not copying back to CERN but we are ready for that as it was successfully tested with MCC9.
For DC1 we have configured POMS in a keepup mode where all new files, coming from CERN (input files), are processed every 4 hours.
As input files we are using the MCC9 files already processed within POMS and copied to CERN.
POMS has been configured for recovery purposes.
Agreed with Andrew three phases:
1) NULL: just copy, sleep x min and copy out, to test the infrustructure. Job are submitted using jobsub
2) NULL_RECO: same as NULL but introducing Lar executable which takes a fcl file with nothing to do.
3) MINI-RECO that mimic the processing (reconstruction step) of files.
Has anybody seen all this besides Anna?
We are testing, but it seems we have problems in dCache.
Is anyone approaching DCache issue?
For now we want to things to run. Since last DCache upgrade it looks like experiments are using it much more.
What we are doing now is to have the whole workflow between CERN and FNAL to work.
Then we will discuss inside CD about the DCache problem. The production group (Anna + Marc) is making sure that the flow is working. Ruth was a little worried, she wanted us to jump to the entire payload provided by ProtoDUNE people in
order we can move on MCC10. Doing the test step by step is worth to.
Checking plots on DCache history viewing plots from FIFEMON :Microboone was throwing 20-30 TBytes/hour in DCache..
2) v2_3_0 update
Work on HOLD/RELEASE button; discussion about ROLES. As “Coordinator’ role I can hold campaigns created by other people.
About ROOT: this was a ROLE, in the new implementation will be an “attribute”, a “flag” for the user.
OLD implementation: 4 layers of ROLES:
NEW implementation discussion:
ROOT is an attribute of the user. Any user with a ROOT flag can change status for other users.
(Example active/inactive campaign).
There will be 3 ROLES:
It will not be a big change in the database.
Will be FERRY project able to handle those 3 roles? Right now there are only two roles: production and analysis. They will need to add the Coordinator role.
We can suggest they do that. They have a sort of superuser production flag. On some experiments the person with PRODUCTION role can kill analysis jobs of a user with ANALYSIS role. They should also have the COORDINATOR role. We can
suggest an enhancement for them.
I asked to change parameters on jobs submission and this should be done by the COORDINATOR.
Can a production user kill others production users’ jobs?
WE should be able to kill each others jobs as production role. This is especially important if you are on shift. But NOT analysis jobs.
What if I am on shift as production role and need to kill an analysis job?
Need to contact coordinator.
Issue with HOLD/RELEASE hierarchy :
The old picture was:
^ | PRD
^ | ANALYSIS
Now we are saying different: still vertical towards coordinator BUT horizontal for the lower roles:
PRD1 < ----- > PRD2 YES kill each other jobs
ANA1 < --/-- > ANA2 NO kill each other jobs.
For Analysis, if needed to kill the jobs, need to contact Coordinator.
Everybody needs to have a coordinator for each experiment.
Once role structure is in place, need to decide who has permission to hold.
My understanding is that, once the structure is implemented, then we can assign the permissions.
About who should be the coordinator, probably the exp liaison..
Need to talk about email idea: we need to define what we want and who and when to notify people. Need to have a meeting about all this.
We will have a separate discussion about this. But maybe we would like to have notification when campaign stage changes. We should have a list of possible situations and for each one the user can decide to subscribe.
- Have a separate meeting with Anna, Marc and Steve and discuss about e-mails report