Project

General

Profile

Feature #4319

Scale SFA

Added by Alexander Moibenko about 6 years ago. Updated about 6 years ago.

Status:
New
Priority:
Normal
Start date:
07/11/2013
Due date:
08/01/2013
% Done:

100%

Estimated time:
30.00 h
Spent time:
Duration: 22

Description

To scale SFA for increasing number of users we need to be able to add more cache servers.
Cache servers provide cache as NAS and for some configuration packaging and unpackaging files locally on cache servers.
This is how SFA currently operates in production.
The current approach is described in SFA HLD document (https://cdcvs.fnal.gov/redmine/attachments/download/6908/small_files_hld_v21.pdf). In current implementation a new cache server can also be added with its cache mounted on all disk and migrator nodes, but then packaging and unpackaging must be done over NAS (and not locally on cache server), which has proven to be an issue for NFS currently used as NAS provider.
The new approach is to group disk movers and migrators around dedicated disk library (cluster) to access a dedicated cache provided by certain cache server. In such an approach only members of the cluster can access data.

Scaling SFA through clusters.docx (141 KB) Scaling SFA through clusters.docx Alexander Moibenko, 07/15/2013 09:32 AM
Scaling SFA through clusters.pdf (613 KB) Scaling SFA through clusters.pdf Alexander Moibenko, 07/17/2013 04:21 PM
Scaling SFA through clusters.pdf (614 KB) Scaling SFA through clusters.pdf Alexander Moibenko, 07/26/2013 10:18 AM

History

#1 Updated by Alexander Moibenko about 6 years ago

  • Assignee set to Alexander Moibenko

#2 Updated by Alexander Moibenko about 6 years ago

  • Due date set to 08/01/2013
  • % Done changed from 0 to 50
  • Estimated time set to 30.00 h

#3 Updated by Alexander Moibenko about 6 years ago

Policy applies only for writes.
If we want reads come through new cache we need to assign a new library to corresponding disk volumes just as we do for tape volumes when we move then from one library to another.

Question. If we install a new system around a new disk library manager cluster, and move an existing policy to this new cluster, will they be able to access files they wrote prior to the move?

Gene

#5 Updated by Alexander Moibenko about 6 years ago

Added description of multi-processed migrator and its communication with migration dispatcher in case when it ran out of allowed number of processes.

#6 Updated by Alexander Moibenko about 6 years ago

  • % Done changed from 50 to 70

#8 Updated by Alexander Moibenko about 6 years ago

  • % Done changed from 70 to 100


Also available in: Atom PDF