Project

General

Profile

Bluearc Summary

Bluearc is a proprietary, high performance file server, used by several critical CS file sysetems.
  • Supports NFS, CIFS and FTP access
  • up to 100K I/O per second
  • dual 10GBit network links
    • Nominal 0.5 GByte/second
    • Has delivered up to 1 GByte/second
  • Low latency directory access
    • ls at about 50000/second
    • ls -l at 500 to 5000/second
  • We have deployed a single head to serve all the data areas
    • A separate head serves the application areas
    • Separate heads serve the other critical CS infrastructure
Overloads of the head, which primarily motivate this dismount plan,
can produce a couple of effects.
  • Overall slowdowns of data transfer to new clients
  • Head crashes, causing failover to an alternate head.

Overall slowdowns seem be fair, every client gets an equal share of the capacity.
But this is not helpful when one user has created 5000+ clients on Fermigrid.

The primary symptom of an overloaded head is something called Running Bossock Fibers (RBF).
This is the number of software threads handling network requests.
The RBF reported by the present heads saturates at 384.
RBF and open file counts are tracked at http://dbweb0.fnal.gov/BAMon/app/active_files
  • RBF over about 100 produces general slowdowns
  • RBF over 380 sustained can crash the head.
  • Bluearc overloads have been caused by
    • Fermigrid jobs directly reading or writing unique files, causing head contention
    • Fermigrid jobs doing chmod -R to a high level directory.
    • Writing to a file system that is over 98% full.
    • Fermigrid or interactive jobs opening and closing files rapidly (once per read).