Rex Data Handling Shifts¶

We do one week Data Handling shifts.

General shift activities and checklists are recorded in the rexops ECL
This ECL is used for both DH and GRID operations.

DH shifts are scheduled by the REX-DH-Support Rota in Service Now.

Shift duties¶

• DAILY
  • Watch for DH issues in SNOW at https://fermi.service-now.com/
    • See that all items are assigned to someone.
  • Fill out a 'DH check' checklist
• WEEKLY
  • Prepare a weekly shift summary, with an executive summary posted to the ECL

Monitoring web pages¶

DH shift checklist links are underlined

IFRONT

• SAMWEB MEMORY CPU
• GRIDFTP SERVERS - project
• GRIDFTP SERVERS - Bestman
• FIFEMON OLD
• BLUEARC OPEN FILES
• BLUEARC PERFORMANCE
• BLUEARC ACCESS LOGS
• PNFS LISTING PERFORMANCE
• DCACHE FTP PERFORMANCE
• Enstore TAPES
• DCACHE SPACE TRANSFERS CHIMERA PERSISTENT
• FTS
  • NOVADAQ01 PENDING STATUS
  • NOVADAQ02 PENDING STATUS
  • NOVADAQ03 PENDING STATUS
  • NOVA GPCF028 STATUS
  • LARIAT PROD STATUS
  • MINOS PROD STATUS
  • NOVA SAM01 STATUS
  • NOVA SAM02 STATUS
  • NOVA SAM03 STATUS
  • IF04 STATUS
  • SAMD STATUS
• KDC TIMINGS* BLUEARC CAPACITY
• ENSTORE DCACHE
• MINOS Data
• MINOS Overview
• Network
  • ifora2

CDF

• SAM AT A GLANCE
• DH AT A GLANCE
• SAM ENSTORE DCACHE
• Database Monitoring
• old DH page
• Network
  • fcdfora3
  • fcdfora3-nfs

D0

• SAM AT A GLANCE
• Watch for swap usage, see INC000000355620
  • d0dbsrv20 memory
  • d0dbsrv21 memory
• SAM ENSTORE SAM PLOTS DCACHE
• DH AT A GLANCE
• Network
  • d0ora3

REX MONITORING

• SAM STATIONS
• MAAWS PLOTS

FACILITY

• PLANNED COMPUTING OUTAGES
• GCC Temperature plots
• LOAD SHED PLAN 2014
  • 2013 obsolete
  • 2012 obsolete

( updated {{last_modified}} kreymer)