Current & future development¶
Not in any sort of order of importance or priority.
- Convert to Python 2.7 (long term supported Python 2 version)
- Update to Oracle 11.2 client (preferably before Oracle drops support for the 10.2 client)
- Use Kerberos authentication for DB connections (needs some solution to admin command authorization)
- Convert to Python 2.7 - mostly done, but linked to station v9 changes below.
- Split station/fss/stager components into separate products since you often don't need all of them, and they tend to change at different rates (should be part of v9 release).
- Release v9 with assorted improvements to project/get next file CORBA interfaces - mostly done, but needs testing.
- Better support for NAS - allow stagers on multiple hosts to manage the same storage for load balancing and redundancy
- Support dCache over parallel NFS - similar to NAS, but needs to trigger dCache file staging from tape instead of normal file transfers
- SSL support for at least some operations - in these paranoid times we may not want stager file handling commands passing around in clear text (omniORB supports SSL, so this may be as simple as turning it on).
- Maybe split the station specific data from the main database (the station can already store some information about cached files locally to improve retrieval performance; maybe we don't need to put this stuff in the central DB at all).
- Get experiments using it for real.
- Minerva production
- ART integration
- Add more functionality as needed.
- Move dataset definition editor from prototype to production; add experiment specific features.
- Move project web API into station/project master processes - more scalable than sending it all through the base web server.
- Implement new metadata schemas for the new experiments. Probably the most complicated thing on this list. Needs plenty of thought and design - query tools, integration with existing SAM components, etc.
File Transfer Service¶
- Extend use beyond NOvA raw data. Next version will support plugin metadata extraction tools, which will make this easier.
- Improve behaviour and monitoring when relaying files (for example the Ash River FTS should show the progress of its files all the way to enstore, even though it doesn't physically handle them once they've been forwarded to FNAL).
We probably want the FTS to directly call some replacement of sam_cp, which should:
- have a simpler, more useful configuration system
- still deal appropriately with re-run, interrupted copies
- probably use ifdh for most transfers
- optionally declare files upon completed
- Long term support for D0: will the existing dedicated CAB with local SAM cache remain, or will we need to support jobs running on non-dedicated worker nodes?
- Replace the Orbacus CORBA nameservice with the omniORB one (Orbacus won't even compile on modern OSs). It should be transparent to clients, but this needs to be tested.