Project

General

Profile

Feature #23645

test_runner pick CI_EXP_CODE from the configuration file

Added by Gianluca Petrillo 2 months ago. Updated about 1 month ago.

Status:
Work in progress
Priority:
Normal
Target version:
-
Start date:
11/21/2019
Due date:
% Done:

90%

Estimated time:
Duration:

Description

In order to utilize the --update option, test_runner needs the environment variable CI_EXP_CODE to be defined.

If it is acceptable to have a parameter CI_EXP_CODE defined in the [DEFAULT] section of the C.I. test configuration file (e.g. ci_tests.cfg), is it possible to have test_runner pick the value of CI_EXP_CODE from that configuration file instead of the environment?
If instead the protocol imposes that CI_EXP_CODE must not be present in the configuration file, is it possible to allow to pass it as a program argument (e.g. test_runner --expcode=LARSOFTCODE)?
The details of which takes precedence should be known, but this request does not recommend any specific approach.

History

#1 Updated by Vito Di Benedetto about 2 months ago

Gianluca, are you requesting this feature with the purpose to be able to generate a set of reference files manually?

In the current configuration by design users are able to provide multiple cfg file to test_runner, in this case all [DEFAULT] sections are merged together. Because of this the last cfg wins and it will set CI_EXP_CODE. Although this is predictable behavior, it could lead to confusion, so I would avoid this.

Currently the CI_EXP_CODE environment variable is set through the CI workflow configuration, this ensure that it is set properly.

Having support for a new option --expcode to set CI_EXP_CODE can be done. Sill I'll prefer to have only one way at a time enabled to set CI_EXP_CODE.
I mean, if both --expcode and the environment variable CI_EXP_CODE are set, this will cause the code to fail.

Do you think this will work for your use case of this feature?

#2 Updated by Gianluca Petrillo about 2 months ago

Yes, that would work.
Also, it would be good that if the necessary variable/configuration is not set, the script would abort immediately instead of producing an error after all the tests are done. I don't know if this implies too much complication though.

And: yep, I was trying to create reference files locally.

#3 Updated by Vito Di Benedetto about 2 months ago

  • Assignee set to Vito Di Benedetto
  • Status changed from New to Accepted

Gianluca Petrillo wrote:

Yes, that would work.

OK, I'll include the --expcode option for test_runner

Also, it would be good that if the necessary variable/configuration is not set, the script would abort immediately instead of producing an error after all the tests are done. I don't know if this implies too much complication though.

That's reasonable, I'll try to get this check in.

And: yep, I was trying to create reference files locally.

Just to make sure we are on the same page,
the location of reference files is defined in the C.I. test configuration file (i.e. ci_tests.cfg), usually the path is a dCache path.
Though newly generated reference files are not replacing the previous set of reference files.
The CI workflow to update reference files will first check that the new set of reference files passes all regression test, then it run a step to update current reference files with new ones.

#4 Updated by Gianluca Petrillo about 2 months ago

Vito Di Benedetto wrote:

Just to make sure we are on the same page,
the location of reference files is defined in the C.I. test configuration file (i.e. ci_tests.cfg), usually the path is a dCache path.
Though newly generated reference files are not replacing the previous set of reference files.
The CI workflow to update reference files will first check that the new set of reference files passes all regression test, then it run a step to update current reference files with new ones.

The test_runner script also seems to do something similar, attempting to copy the new reference files only after the successful completion of the requested tests.
But with the exception that it is unable to either create missing subdirectories nor to overwrite existing reference files, and will abort at the first failure.

#5 Updated by Vito Di Benedetto about 1 month ago

  • % Done changed from 0 to 50
  • Status changed from Accepted to Work in progress

#6 Updated by Vito Di Benedetto about 1 month ago

  • % Done changed from 50 to 90

This feature is ready.
I also included code to create reference files destination folders, in case they don't exist yet.

In the default configuration, the reference file destination is expected to be set in ci_tests.cfg,
If the test_runner option -u is used, it is also required to set up an identifier value, usually it is a timestamp,
the identifier environment variable is defined in the ci_tests.cfg.
In case the identifier is not set, test_runner will provide the command to set it.

Reference files will be created in the reference destination folder, their file name includes the identifier. Usually this step uses a sequential CI regression test suite.
To verify that the code results are reproducible, it is needed to run the parallel CI regression test suite equivalent to the sequential one. The previous identifier here is still needed to point to the new reference files.

If a new set of reference files needs to be created in the same session, and the reference file destination is on dCache,
it is needed to update the identifier, otherwise the copy will fail because by default ifdh will not override existing files.
I prefer to keep this behavior to avoid that reference files used in production would be accidentally overridden.

If you like to test this you could checkout and setup the branch vito_23645 for generic_ci and lar_ci repositories.
There will be a test release by mid January, but, if you want to test this earlier, I can cut a test release of the CI with this feature and few more updates in coming days.



Also available in: Atom PDF