Feature #23381
Harmonize CI test scripts
Start date:
04/23/2020
Due date:
% Done:
100%
Estimated time:
(Total: 0.00 h)
Stakeholders:
Description
Provide some utility function in utils.sh to harmonize the various CI script (shellcheck, run_pylint, unittest, ...).
- invocation options should be the same
- ability to run on current or multiple branches
- similar return codes (script failed, warnings found, errors found, ok, ...)
- similar output summary (table, email)
This will need to be in a separate branch and coordinated w/ Fermilab's CI
Some links about formats:- https://testanything.org/
- https://help.catchsoftware.com/display/ET/JUnit+Format
- https://circleci.com/docs/2.0/collect-test-data/
- https://jestjs.io/docs/en/cli (--reporters jest-junit)
- https://www.baeldung.com/checkstyle-java
- https://checkstyle.org/checks.html
List report example
filename info warning errors name1 1 0 2
XML report example
Pass (one test, that linting was run) <?xml version="1.0" encoding="UTF-8"?> <testsuites name="jest tests" tests="1" failures="0" time="0.346"> <testsuite name="undefined" errors="0" failures="0" skipped="0" timestamp="2019-06-07T15:44:08" time="0.092" tests="1"> <testcase classname=" OAS Linter" name=" OAS Linter" time="0.092"> </testcase> </testsuite> </testsuites> Fail (multiple tests, one for each warning from the linter) <?xml version="1.0" encoding="UTF-8"?> <testsuites name="jest tests" tests="4" failures="4" time="0.39"> <testsuite name="undefined" errors="0" failures="4" skipped="0" timestamp="2019-06-07T15:50:20" time="0.148" tests="4"> <testcase classname=" schema-properties-type if.then test failed:undefined != object - #/components/schemas/AddressRegion/oneOf/0 (schema-properties-type)" name=" schema-properties-type if.then test failed:undefined != object - #/components/schemas/AddressRegion/oneOf/0 (schema-properties-type)" time="0.148"> <failure>schema-properties-type if.then test failed:undefined != object</failure> </testcase> <testcase classname=" schema-properties-type if.then test failed:undefined != object - #/components/schemas/AddressRegion/oneOf/1 (schema-properties-type)" name=" schema-properties-type if.then test failed:undefined != object - #/components/schemas/AddressRegion/oneOf/1 (schema-properties-type)" time="0.148"> <failure>schema-properties-type if.then test failed:undefined != object</failure> </testcase> <testcase classname=" schema-properties-type if.then test failed:undefined != object - #/components/schemas/AuctionListingAddress/oneOf/0 (schema-properties-type)" name=" schema-properties-type if.then test failed:undefined != object - #/components/schemas/AuctionListingAddress/oneOf/0 (schema-properties-type)" time="0.148"> <failure>schema-properties-type if.then test failed:undefined != object</failure> </testcase> <testcase classname=" schema-properties-type if.then test failed:undefined != object - #/components/schemas/AuctionListingAddress/oneOf/1 (schema-properties-type)" name=" schema-properties-type if.then test failed:undefined != object - #/components/schemas/AuctionListingAddress/oneOf/1 (schema-properties-type)" time="0.148"> <failure>schema-properties-type if.then test failed:undefined != object</failure> </testcase> </testsuite> </testsuites>
Subtasks
Related issues
History
#1 Updated by Marco Mambelli about 1 year ago
- Target version changed from v3_6_2 to v3_6_3
#2 Updated by Marco Mambelli 11 months ago
- Subject changed from Harmonize CI scripts to Harmonize CI test scripts
#3 Updated by Marco Mambelli 11 months ago
- Start date changed from 10/04/2019 to 04/23/2020
- Due date set to 04/23/2020
due to changes in a related task: #24345
#4 Updated by Marco Mambelli 11 months ago
- Related to Feature #18042: Change exit code of pylint and unit test scripts added
#5 Updated by Marco Mambelli 11 months ago
- Related to deleted (Feature #18042: Change exit code of pylint and unit test scripts)
#6 Updated by Marco Mambelli 11 months ago
- Blocked by Feature #18042: Change exit code of pylint and unit test scripts added
#7 Updated by Marco Mambelli 10 months ago
- Target version changed from v3_6_3 to v3_6_4
#8 Updated by Marco Mambelli 6 months ago
- Status changed from New to Resolved
runtest.sh solves most of the items
Opened a separate ticket about a better and different output format [#24794]
#9 Updated by Marco Mambelli 6 months ago
- Status changed from Resolved to Closed