Nightly build triage guide


  • At approximately 9am each weekday morning, the designated triage person (DTP) (or a substitute if the original is unavailable) shall review the emails regarding nightly builds.
  • The art CI dashboard page will always be available for a top-down view.
  • DTP shall make a determination of the general class of any failure according to the disposition guide below, and send a summary email to the artists list specifying any failures and the person (or persons) who are best equipped to investigate further.

Disposition guide

  • Any problems which have been mentioned in the immediately-previous summary should be noted only as ongoing issues with enough detail to distinguish them without ambiguity.
  • Any problems which appear to be related to the CI or build service infrastructure should be directed to Marc.
  • Problems at the pullProducts or checkout stages should be directed to Lynn or Chris.
  • Product setup or dependency problems at the beginning of the build stage should be directed to Lynn or Chris.
  • Build problems which can reasonably be attributed to the build system should be directed to Lynn or Chris.
  • Compilation problems at the build stage or test failures should be directed to:
    • The person responsible for any recent work in that particular area, as discernible from commit messages or personal knowledge; or
    • The person with most recent working knowledge of that area according to personal knowledge; or
    • The appropriate person in the following list:
      • hep_concurrency: Paul or Kyle.
      • cetlib_except: Marc or Chris.
      • cetlib: Marc or Chris.
      • fhiclcpp: Chris or Kyle.
      • messagefacility: Paul or Chris.
      • canvas: Kyle or Marc.
      • canvas_root_io: Kyle or Marc.
      • gallery: Marc or Chris.
      • art: Kyle or Marc.
      • critic: Kyle or Marc.
  • If there is any doubt as to the determination of the most suitable party this should be mentioned in the summary email. In this case, if anyone feels confident to take on the issue they should reply to all; otherwise the project lead takes responsibility for ensuring that the issue will be discussed no later than the next triage meeting.