Project

General

Profile

Support #22928

Fermilab Frontend not communicating w/ upgraded Factory

Added by Marco Mambelli 4 days ago.

Status:
New
Priority:
Normal
Category:
-
Target version:
Start date:
07/12/2019
Due date:
% Done:

0%

Estimated time:
Stakeholders:
Duration:

Description

After the Factory upgraded to 3.4.5 the Fermilab Frontend gpfrontend02, 3.4.2, stopped requesting jobs: was seeing the schedd jobs, was seeing the Factory entries (up), was not requesting glideins to the Factory, almost all stats were 0 (except few glideins running in an entry). Don't know if there were actually no glideins or if it was seeing none, anyway it should have been requesting some.
After upgrading the Frontend to 3.4.5 the requests restarted but glideins in one group were not reporting back.
Later we found out that there was a broken script, Exp_CVMFS_check.sh (missing line continuations). Probably the other groups had glideins queued submitted when a previous version of the script was the current one (glideins download the script version that was current at submission time, not run time).

Problem seems solved now for Factory 3.4.5 and Frontend 3.4.5
Remains to test and reproduce the possible problem w/ Factory 3.4.5 and Frontend 3.4.2. Is it there an incompatibility that was missed at testing?



Also available in: Atom PDF