Project

General

Profile

Feature #23381

Harmonize CI test scripts

Added by Marco Mambelli about 1 year ago. Updated about 1 month ago.

Status:
Closed
Priority:
Normal
Category:
-
Target version:
Start date:
04/23/2020
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Stakeholders:
Duration:

Description

Provide some utility function in utils.sh to harmonize the various CI script (shellcheck, run_pylint, unittest, ...).
  • invocation options should be the same
  • ability to run on current or multiple branches
  • similar return codes (script failed, warnings found, errors found, ok, ...)
  • similar output summary (table, email)

This will need to be in a separate branch and coordinated w/ Fermilab's CI

Some links about formats:

List report example

filename info warning errors
name1 1 0 2

XML report example

Pass (one test, that linting was run)
<?xml version="1.0" encoding="UTF-8"?>
<testsuites name="jest tests" tests="1" failures="0" time="0.346">
  <testsuite name="undefined" errors="0" failures="0" skipped="0" timestamp="2019-06-07T15:44:08" time="0.092" tests="1">
    <testcase classname=" OAS Linter" name=" OAS Linter" time="0.092">
    </testcase>
  </testsuite>
</testsuites>

Fail (multiple tests, one for each warning from the linter)
<?xml version="1.0" encoding="UTF-8"?>
<testsuites name="jest tests" tests="4" failures="4" time="0.39">
  <testsuite name="undefined" errors="0" failures="4" skipped="0" timestamp="2019-06-07T15:50:20" time="0.148" tests="4">
    <testcase classname=" schema-properties-type if.then test failed:undefined != object - #/components/schemas/AddressRegion/oneOf/0 (schema-properties-type)" name=" schema-properties-type if.then test failed:undefined != object - #/components/schemas/AddressRegion/oneOf/0 (schema-properties-type)" time="0.148">
      <failure>schema-properties-type if.then test failed:undefined != object</failure>
    </testcase>
    <testcase classname=" schema-properties-type if.then test failed:undefined != object - #/components/schemas/AddressRegion/oneOf/1 (schema-properties-type)" name=" schema-properties-type if.then test failed:undefined != object - #/components/schemas/AddressRegion/oneOf/1 (schema-properties-type)" time="0.148">
      <failure>schema-properties-type if.then test failed:undefined != object</failure>
    </testcase>
    <testcase classname=" schema-properties-type if.then test failed:undefined != object - #/components/schemas/AuctionListingAddress/oneOf/0 (schema-properties-type)" name=" schema-properties-type if.then test failed:undefined != object - #/components/schemas/AuctionListingAddress/oneOf/0 (schema-properties-type)" time="0.148">
      <failure>schema-properties-type if.then test failed:undefined != object</failure>
    </testcase>
    <testcase classname=" schema-properties-type if.then test failed:undefined != object - #/components/schemas/AuctionListingAddress/oneOf/1 (schema-properties-type)" name=" schema-properties-type if.then test failed:undefined != object - #/components/schemas/AuctionListingAddress/oneOf/1 (schema-properties-type)" time="0.148">
      <failure>schema-properties-type if.then test failed:undefined != object</failure>
    </testcase>
  </testsuite>
</testsuites>


Subtasks

Feature #24345: Provide a runner to run the different tests with consistent CLIClosedMarco Mambelli


Related issues

Blocked by GlideinWMS - Feature #18042: Change exit code of pylint and unit test scriptsClosed10/26/2017

History

#1 Updated by Marco Mambelli 10 months ago

  • Target version changed from v3_6_2 to v3_6_3

#2 Updated by Marco Mambelli 6 months ago

  • Subject changed from Harmonize CI scripts to Harmonize CI test scripts

#3 Updated by Marco Mambelli 6 months ago

  • Start date changed from 10/04/2019 to 04/23/2020
  • Due date set to 04/23/2020

due to changes in a related task: #24345

#4 Updated by Marco Mambelli 6 months ago

  • Related to Feature #18042: Change exit code of pylint and unit test scripts added

#5 Updated by Marco Mambelli 6 months ago

  • Related to deleted (Feature #18042: Change exit code of pylint and unit test scripts)

#6 Updated by Marco Mambelli 6 months ago

  • Blocked by Feature #18042: Change exit code of pylint and unit test scripts added

#7 Updated by Marco Mambelli 6 months ago

  • Target version changed from v3_6_3 to v3_6_4

#8 Updated by Marco Mambelli about 1 month ago

  • Status changed from New to Resolved

runtest.sh solves most of the items
Opened a separate ticket about a better and different output format [#24794]

#9 Updated by Marco Mambelli about 1 month ago

  • Status changed from Resolved to Closed


Also available in: Atom PDF