Project

General

Profile

Bug #14164

Entries in downtime slow down the ramp-up for the frontend

Added by Marco Mambelli over 2 years ago. Updated about 1 year ago.

Status:
Closed
Priority:
Normal
Category:
-
Target version:
Start date:
10/14/2016
Due date:
% Done:

0%

Estimated time:
First Occurred:
Occurs In:
Stakeholders:
Duration:

Description

When calculating the glidein to assign to each entry the frontend is dividing for all the matching entries (including the ones in downtime), causing a slower ramp-up.
This behavior has been there for many version.

10 requests, 2 entries, one in downtime. Note only 6 MaxRun in the first iteration:

[2016-10-14 14:15:04,438] INFO: glideinFrontendElement:1808:     5(   10     5     0     0)     0(    0 10000) |     0     0     0     0 |     0     0     0 |     0     0 | Down ITB_FC_CE2@gfactory_instance@gfactory_service@fermicloud384.fnal.gov
[2016-10-14 14:15:04,439] INFO: glideinFrontendElement:1808:     5(   10     5     0     0)     0(    0 10000) |     0     0     0     0 |     0     0     0 |     1     6 | Up   ITB_FC_CE3@gfactory_instance@gfactory_service@fermicloud384.fnal.gov

To correct the behavior and improve the ramp-up, the function splitting the glideins proportionally should not consider the matching ones in downtime.
Options:
- add an attribute that will not match in the group the downtime entries
- fix countMatch (in glideinFrontendLib) not to count entries in downtime

NOTE: pay attention not to cause a division by 0

10 requests, 2 entries both up, same behavior:

[2016-10-14 14:30:05,437] INFO: glideinFrontendElement:1808:     5(   10     5     0     0)     0(    0 10000) |     0     0     0     0 |     0     0     0 |     1     6 | Up   ITB_FC_CE2@gfactory_instance@gfactory_service@fermicloud384.fnal.gov
[2016-10-14 14:30:05,438] INFO: glideinFrontendElement:1808:     5(   10     5     0     0)     0(    0 10000) |     0     0     0     0 |     0     0     0 |     1     6 | Up   ITB_FC_CE3@gfactory_instance@gfactory_service@fermicloud384.fnal.gov

History

#1 Updated by Parag Mhashilkar over 1 year ago

  • Assignee set to Lorena Lobato Pardavila
  • Target version set to v3_2_22

Several code changes have been made since the ticket was created. Please investigate if the problem still exists before fixing.

#2 Updated by Marco Mambelli over 1 year ago

  • Target version changed from v3_2_22 to v3_2_23

#3 Updated by Lorena Lobato Pardavila about 1 year ago

  • Status changed from New to Assigned

#4 Updated by Lorena Lobato Pardavila about 1 year ago

Have done several test in my environment where FrontEnd (fermicloud364) is connected to 2 Factories (fermicloud137) and . Not observed any slow ramp-up between interactions when calculating the glidein to assign to each entry. Always 0 when downtime entries.

[2018-04-16 15:33:19,785] INFO: glideinFrontendElement:1724: All children terminated - took 0.19551897049 seconds
[2018-04-16 15:33:19,785] INFO: glideinFrontendElement:521: Total matching idle 993 (old 10min 0 60min 0) running 1 limit 10000
[2018-04-16 15:33:19,786] INFO: glideinFrontendElement:1886:             Jobs in schedd queues                 |           Slots         |       Cores       | Glidein Req | Factory/Entry Information
[2018-04-16 15:33:19,786] INFO: glideinFrontendElement:1887: Idle (match  eff   old  uniq )  Run ( here  max ) | Total  Idle   Run  Fail | Total  Idle   Run | Idle MaxRun | State Factory
[2018-04-16 15:33:19,787] INFO: glideinFrontendElement:1875:    17(  993    17     0     0)     1(    1 10000) |     1     0     1     0 |     1     0     1 |     5    21 | Up   ITB_FC_CE2@gfactory_instance@gfactory_service@fermicloud137.fnal.gov
[2018-04-16 15:33:19,789] INFO: glideinFrontendElement:1875:    17(  993    17     0     0)     1(    0 10000) |     0     0     0     0 |     0     0     0 |     1     4 | Up   CMSHTPC_T2_US_Caltech_cit2_op@gfactory_instance@ITBGOC@glidein-itb.grid.iu.edu
[2018-04-16 15:33:19,790] INFO: glideinFrontendElement:1875:    17(  993    17     0     0)     1(    0 10000) |     0     0     0     0 |     0     0     0 |     1     4 | Up   CMSHTPC_T2_US_Caltech_cit_op@gfactory_instance@ITBGOC@glidein-itb.grid.iu.edu
[2018-04-16 15:33:19,791] INFO: glideinFrontendElement:1875:    17(  993    17     0     0)     1(    0 10000) |     0     0     0     0 |     0     0     0 |     1     4 | Up   CMSHTPC_T2_US_MIT_ce01@gfactory_instance@ITBGOC@glidein-itb.grid.iu.edu
[2018-04-16 15:33:19,793] INFO: glideinFrontendElement:1875:    17(  993    17     0     0)     1(    0 10000) |     0     0     0     0 |     0     0     0 |     1     4 | Up   CMSHTPC_T2_US_UCSD_gw2@gfactory_instance@ITBGOC@glidein-itb.grid.iu.edu
[2018-04-16 15:33:19,794] INFO: glideinFrontendElement:1875:    17(  993    17     0     0)     1(    0 10000) |     0     0     0     0 |     0     0     0 |     1     4 | Up   CMSHTPC_T2_US_UCSD_gw4@gfactory_instance@ITBGOC@glidein-itb.grid.iu.edu
[2018-04-16 15:33:19,795] INFO: glideinFrontendElement:1875:    17(  993    17     0     0)     1(    0 10000) |     0     0     0     0 |     0     0     0 |     1     4 | Up   CMSHTPC_T2_US_UCSD_gw6@gfactory_instance@ITBGOC@glidein-itb.grid.iu.edu
[2018-04-16 15:33:19,796] INFO: glideinFrontendElement:1875:    17(  993    17     0     0)     1(    0 10000) |     0     0     0     0 |     0     0     0 |     1     4 | Up   CMSHTPC_T2_US_UCSD_gw7@gfactory_instance@ITBGOC@glidein-itb.grid.iu.edu
[2018-04-16 15:33:19,797] INFO: glideinFrontendElement:1875:    17(  993    17     0     0)     1(    0 10000) |     0     0     0     0 |     0     0     0 |     1     4 | Up   CMSHTPC_T3_US_Colorado_heposg01@gfactory_instance@ITBGOC@glidein-itb.grid.iu.edu
[2018-04-16 15:33:19,798] INFO: glideinFrontendElement:1875:    17(  993    17     0     0)     1(    0 10000) |     0     0     0     0 |     0     0     0 |     1     4 | Up   CMSHTPC_T3_US_NotreDame_deepthought@gfactory_instance@ITBGOC@glidein-itb.grid.iu.edu
[2018-04-16 15:33:19,799] INFO: glideinFrontendElement:1875:    17(  993    17     0     0)     1(    0 10000) |     0     0     0     0 |     0     0     0 |     1     4 | Up   CMSHTPC_T3_US_Omaha_crane@gfactory_instance@ITBGOC@glidein-itb.grid.iu.edu
[2018-04-16 15:33:19,800] INFO: glideinFrontendElement:1875:    17(  993    17     0     0)     1(    0 10000) |     0     0     0     0 |     0     0     0 |     1     4 | Up   CMSHTPC_T3_US_Omaha_tusker@gfactory_instance@ITBGOC@glidein-itb.grid.iu.edu
[2018-04-16 15:33:19,801] INFO: glideinFrontendElement:1875:    17(  993    17     0     0)     1(    0 10000) |     0     0     0     0 |     0     0     0 |     1     3 | Up   CMSHTPC_T3_US_Omaha_tusker_2x@gfactory_instance@ITBGOC@glidein-itb.grid.iu.edu
[2018-04-16 15:33:19,802] INFO: glideinFrontendElement:1875:    17(  993    17     0     0)     1(    0 10000) |     0     0     0     0 |     0     0     0 |     0     0 | Down CMS_T1_US_FNAL_condce3_opp@gfactory_instance@ITBGOC@glidein-itb.grid.iu.edu
[2018-04-16 15:33:19,803] INFO: glideinFrontendElement:1875:    17(  993    17     0     0)     1(    0 10000) |     0     0     0     0 |     0     0     0 |     0     0 | Down CMS_T1_US_FNAL_condce_opp1@gfactory_instance@ITBGOC@glidein-itb.grid.iu.edu*
[2018-04-16 15:33:19,804] INFO: glideinFrontendElement:1875:    17(  993    17     0     0)     1(    0 10000) |     0     0     0     0 |     0     0     0 |     5    20 | Up   CMS_T2_US_Nebraska_Red_gw1_whole_op@gfactory_instance@ITBGOC@glidein-itb.grid.iu.edu
[2018-04-16 15:33:19,805] INFO: glideinFrontendElement:1875:    17(  993    17     0     0)     1(    0 10000) |     0     0     0     0 |     0     0     0 |     5    20 | Up   CMS_T2_US_Nebraska_Red_gw2_whole_op@gfactory_instance@ITBGOC@glidein-itb.grid.iu.edu
[2018-04-16 15:33:19,806] INFO: glideinFrontendElement:1875:    17(  993    17     0     0)     1(    0 10000) |     0     0     0     0 |     0     0     0 |     5    20 | Up   CMS_T2_US_Nebraska_Red_whole_op@gfactory_instance@ITBGOC@glidein-itb.grid.iu.edu
[2018-04-16 15:33:19,807] INFO: glideinFrontendElement:1875:    17(  993    17     0     0)     1(    0 10000) |     0     0     0     0 |     0     0     0 |     5    20 | Up   CMS_T2_US_Nebraska_sandhills_ce1_op@gfactory_instance@ITBGOC@glidein-itb.grid.iu.edu
[2018-04-16 15:33:19,808] INFO: glideinFrontendElement:1875:    17(  993    17     0     0)     1(    0 10000) |     0     0     0     0 |     0     0     0 |     5    20 | Up   CMS_T3_US_Omaha_crane@gfactory_instance@ITBGOC@glidein-itb.grid.iu.edu
[2018-04-16 15:33:19,809] INFO: glideinFrontendElement:1875:    17(  993    17     0     0)     1(    0 10000) |     0     0     0     0 |     0     0     0 |     5    20 | Up   CMS_T3_US_Omaha_tusker@gfactory_instance@ITBGOC@glidein-itb.grid.iu.edu
[2018-04-16 15:33:19,810] INFO: glideinFrontendElement:1875:    17(  993    17     0     0)     1(    0 10000) |     0     0     0     0 |     0     0     0 |     5    20 | Up   Engage_US_MWT2_iut2_condce@gfactory_instance@ITBGOC@glidein-itb.grid.iu.edu
[2018-04-16 15:33:19,812] INFO: glideinFrontendElement:1875:    17(  993    17     0     0)     1(    0 10000) |     0     0     0     0 |     0     0     0 |     1     4 | Up   Engage_US_MWT2_iut2_condce_mcore@gfactory_instance@ITBGOC@glidein-itb.grid.iu.edu
[2018-04-16 15:33:19,813] INFO: glideinFrontendElement:1875:    17(  993    17     0     0)     1(    0 10000) |     0     0     0     0 |     0     0     0 |     5    20 | Up   Engage_US_MWT2_osg_condce@gfactory_instance@ITBGOC@glidein-itb.grid.iu.edu
[2018-04-16 15:33:19,814] INFO: glideinFrontendElement:1875:    17(  993    17     0     0)     1(    0 10000) |     0     0     0     0 |     0     0     0 |     1     4 | Up   Engage_US_MWT2_osg_condce_mcore@gfactory_instance@ITBGOC@glidein-itb.grid.iu.edu
[2018-04-16 15:33:19,815] INFO: glideinFrontendElement:1875:    17(  993    17     0     0)     1(    0 10000) |     0     0     0     0 |     0     0     0 |     5    20 | Up   Engage_US_MWT2_uct2_condce@gfactory_instance@ITBGOC@glidein-itb.grid.iu.edu
[2018-04-16 15:33:19,816] INFO: glideinFrontendElement:1875:    17(  993    17     0     0)     1(    0 10000) |     0     0     0     0 |     0     0     0 |     1     4 | Up   Engage_US_MWT2_uct2_condce_mcore@gfactory_instance@ITBGOC@glidein-itb.grid.iu.edu
[2018-04-16 15:33:19,816] INFO: glideinFrontendElement:1875:    17(  993    17     0     0)     1(    0 10000) |     0     0     0     0 |     0     0     0 |     5    20 | Up   Glow_US_Syracuse_condor-ce2@gfactory_instance@ITBGOC@glidein-itb.grid.iu.edu
[2018-04-16 15:33:19,817] INFO: glideinFrontendElement:1875:    17(  993    17     0     0)     1(    0 10000) |     0     0     0     0 |     0     0     0 |     5    20 | Up   Glow_US_Syracuse_condor-ce2_rhel6@gfactory_instance@ITBGOC@glidein-itb.grid.iu.edu
[2018-04-16 15:33:19,818] INFO: glideinFrontendElement:1875:    17(  993    17     0     0)     1(    0 10000) |     0     0     0     0 |     0     0     0 |     5    20 | Up   Glow_US_Syracuse_condor-ce3@gfactory_instance@ITBGOC@glidein-itb.grid.iu.edu
[2018-04-16 15:33:19,819] INFO: glideinFrontendElement:1875:    17(  993    17     0     0)     1(    0 10000) |     0     0     0     0 |     0     0     0 |     5    20 | Up   Glow_US_Syracuse_condor-ce3_rhel6@gfactory_instance@ITBGOC@glidein-itb.grid.iu.edu
[2018-04-16 15:33:19,820] INFO: glideinFrontendElement:1875:    17(  993    17     0     0)     1(    0 10000) |     0     0     0     0 |     0     0     0 |     1     4 | Up   HCCHTPC_US_Wisconsin_osg01@gfactory_instance@ITBGOC@glidein-itb.grid.iu.edu
[2018-04-16 15:33:19,821] INFO: glideinFrontendElement:1875:    17(  993    17     0     0)     1(    0 10000) |     0     0     0     0 |     0     0     0 |     1     4 | Up   HCCHTPC_US_Wisconsin_osg01_rhel7@gfactory_instance@ITBGOC@glidein-itb.grid.iu.edu
[2018-04-16 15:33:19,822] INFO: glideinFrontendElement:1875:    17(  993    17     0     0)     1(    0 10000) |     0     0     0     0 |     0     0     0 |     5    20 | Up   HCC_US_BNL_gk01@gfactory_instance@ITBGOC@glidein-itb.grid.iu.edu
[2018-04-16 15:33:19,829] INFO: glideinFrontendElement:1875:    17(  993    17     0     0)     1(    0 10000) |     0     0     0     0 |     0     0     0 |     5    20 | Up   HCC_US_BNL_gk02@gfactory_instance@ITBGOC@glidein-itb.grid.iu.edu
[2018-04-16 15:33:19,830] INFO: glideinFrontendElement:1875:    17(  993    17     0     0)     1(    0 10000) |     0     0     0     0 |     0     0     0 |     5    20 | Up   HCC_US_BU_atlas-net2@gfactory_instance@ITBGOC@glidein-itb.grid.iu.edu
[2018-04-16 15:33:19,831] INFO: glideinFrontendElement:1875:    17(  993    17     0     0)     1(    0 10000) |     0     0     0     0 |     0     0     0 |     5    20 | Up   HCC_US_BU_atlas-net2_long@gfactory_instance@ITBGOC@glidein-itb.grid.iu.edu
[2018-04-16 15:33:19,832] INFO: glideinFrontendElement:1875:    17(  993    17     0     0)     1(    0 10000) |     0     0     0     0 |     0     0     0 |     5    20 | Up   HCC_US_Beocat_osg-ce@gfactory_instance@ITBGOC@glidein-itb.grid.iu.edu
[2018-04-16 15:33:19,833] INFO: glideinFrontendElement:1875:    17(  993    17     0     0)     1(    0 10000) |     0     0     0     0 |     0     0     0 |     5    20 | Up   HCC_US_Michigan_gate02@gfactory_instance@ITBGOC@glidein-itb.grid.iu.edu
[2018-04-16 15:33:19,834] INFO: glideinFrontendElement:1875:    17(  993    17     0     0)     1(    0 10000) |     0     0     0     0 |     0     0     0 |     5    20 | Up   HCC_US_Wisconsin_osg01@gfactory_instance@ITBGOC@glidein-itb.grid.iu.edu
[2018-04-16 15:33:19,835] INFO: glideinFrontendElement:1875:    17(  993    17     0     0)     1(    0 10000) |     0     0     0     0 |     0     0     0 |     5    20 | Up   HCC_US_Wisconsin_osg01_rhel7@gfactory_instance@ITBGOC@glidein-itb.grid.iu.edu
[2018-04-16 15:33:19,836] INFO: glideinFrontendElement:1875:    17(  993    17     0     0)     1(    0 10000) |     0     0     0     0 |     0     0     0 |     5    20 | Up   OSG_US_ASU_OCTILLO@gfactory_instance@ITBGOC@glidein-itb.grid.iu.edu
[2018-04-16 15:33:19,837] INFO: glideinFrontendElement:1875:    17(  993    17     0     0)     1(    0 10000) |     0     0     0     0 |     0     0     0 |     5    20 | Up   OSG_US_Clemson-Palmetto_condce@gfactory_instance@ITBGOC@glidein-itb.grid.iu.edu
[2018-04-16 15:33:19,838] INFO: glideinFrontendElement:1875:    17(  993    17     0     0)     1(    0 10000) |     0     0     0     0 |     0     0     0 |     1     4 | Up   OSG_US_Clemson-Palmetto_condce_mcore@gfactory_instance@ITBGOC@glidein-itb.grid.iu.edu
[2018-04-16 15:33:19,839] INFO: glideinFrontendElement:1875:    17(  993    17     0     0)     1(    0 10000) |     0     0     0     0 |     0     0     0 |     5    20 | Up   OSG_US_DANFORTH@gfactory_instance@ITBGOC@glidein-itb.grid.iu.edu
[2018-04-16 15:33:19,840] INFO: glideinFrontendElement:1875:    17(  993    17     0     0)     1(    0 10000) |     0     0     0     0 |     0     0     0 |     5    20 | Up   OSG_US_Hyak_osg@gfactory_instance@ITBGOC@glidein-itb.grid.iu.edu
[2018-04-16 15:33:19,841] INFO: glideinFrontendElement:1875:    17(  993    17     0     0)     1(    0 10000) |     0     0     0     0 |     0     0     0 |     5    20 | Up   OSG_US_MWT2_mwt2_condce@gfactory_instance@ITBGOC@glidein-itb.grid.iu.edu
[2018-04-16 15:33:19,843] INFO: glideinFrontendElement:1875:    17(  993    17     0     0)     1(    0 10000) |     0     0     0     0 |     0     0     0 |     1     4 | Up   OSG_US_MWT2_mwt2_condce_mcore@gfactory_instance@ITBGOC@glidein-itb.grid.iu.edu
[2018-04-16 15:33:19,844] INFO: glideinFrontendElement:1875:    17(  993    17     0     0)     1(    0 10000) |     0     0     0     0 |     0     0     0 |     1     4 | Up   OSG_US_Stanford_Proclus@gfactory_instance@ITBGOC@glidein-itb.grid.iu.edu
[2018-04-16 15:33:19,845] INFO: glideinFrontendElement:1875:    17(  993    17     0     0)     1(    0 10000) |     0     0     0     0 |     0     0     0 |     5    20 | Up   OSG_US_USD_LEGACY_ce5@gfactory_instance@ITBGOC@glidein-itb.grid.iu.edu
[2018-04-16 15:33:19,846] INFO: glideinFrontendElement:1875:    17(  993    17     0     0)     1(    0 10000) |     0     0     0     0 |     0     0     0 |     5    20 | Up   OSG_US_WORKSHOP_10@gfactory_instance@ITBGOC@glidein-itb.grid.iu.edu
[2018-04-16 15:33:19,847] INFO: glideinFrontendElement:1875:    17(  993    17     0     0)     1(    0 10000) |     0     0     0     0 |     0     0     0 |     5    20 | Up   OSG_US_WORKSHOP_1@gfactory_instance@ITBGOC@glidein-itb.grid.iu.edu
[2018-04-16 15:33:19,848] INFO: glideinFrontendElement:1875:    17(  993    17     0     0)     1(    0 10000) |     0     0     0     0 |     0     0     0 |     5    20 | Up   OSG_US_WORKSHOP_2@gfactory_instance@ITBGOC@glidein-itb.grid.iu.edu
[2018-04-16 15:33:19,848] INFO: glideinFrontendElement:1875:    17(  993    17     0     0)     1(    0 10000) |     0     0     0     0 |     0     0     0 |     5    20 | Up   OSG_US_WORKSHOP_3@gfactory_instance@ITBGOC@glidein-itb.grid.iu.edu
[2018-04-16 15:33:19,849] INFO: glideinFrontendElement:1875:    17(  993    17     0     0)     1(    0 10000) |     0     0     0     0 |     0     0     0 |     5    20 | Up   OSG_US_WORKSHOP_4@gfactory_instance@ITBGOC@glidein-itb.grid.iu.edu
[2018-04-16 15:33:19,850] INFO: glideinFrontendElement:1875:    17(  993    17     0     0)     1(    0 10000) |     0     0     0     0 |     0     0     0 |     5    20 | Up   OSG_US_WORKSHOP_5@gfactory_instance@ITBGOC@glidein-itb.grid.iu.edu
[2018-04-16 15:33:19,851] INFO: glideinFrontendElement:1875:    17(  993    17     0     0)     1(    0 10000) |     0     0     0     0 |     0     0     0 |     5    20 | Up   OSG_US_WORKSHOP_6@gfactory_instance@ITBGOC@glidein-itb.grid.iu.edu
[2018-04-16 15:33:19,852] INFO: glideinFrontendElement:1875:    17(  993    17     0     0)     1(    0 10000) |     0     0     0     0 |     0     0     0 |     5    20 | Up   OSG_US_WORKSHOP_7@gfactory_instance@ITBGOC@glidein-itb.grid.iu.edu
[2018-04-16 15:33:19,853] INFO: glideinFrontendElement:1875:    17(  993    17     0     0)     1(    0 10000) |     0     0     0     0 |     0     0     0 |     5    20 | Up   OSG_US_WORKSHOP_8@gfactory_instance@ITBGOC@glidein-itb.grid.iu.edu
[2018-04-16 15:33:19,854] INFO: glideinFrontendElement:1875:    17(  993    17     0     0)     1(    0 10000) |     0     0     0     0 |     0     0     0 |     5    20 | Up   OSG_US_WORKSHOP_9@gfactory_instance@ITBGOC@glidein-itb.grid.iu.edu
[2018-04-16 15:33:19,855] INFO: glideinFrontendElement:1875:    17(  993    17     0     0)     1(    0 10000) |     0     0     0     0 |     0     0     0 |     5    20 | Up   OSG_US_WSU_GRID_ce2@gfactory_instance@ITBGOC@glidein-itb.grid.iu.edu
[2018-04-16 15:33:19,856] INFO: glideinFrontendElement:1886:             Jobs in schedd queues                 |           Slots         |       Cores       | Glidein Req | Factory/Entry Information
[2018-04-16 15:33:19,857] INFO: glideinFrontendElement:1887: Idle (match  eff   old  uniq )  Run ( here  max ) | Total  Idle   Run  Fail | Total  Idle   Run | Idle MaxRun | State Factory
[2018-04-16 15:33:19,857] INFO: glideinFrontendElement:1875:  1003(58587  1003     0     0)    59(    1  590k) |     1     0     1     0 |     1     0     1 |   215   860 | Up   Sum of useful factories
[2018-04-16 15:33:19,857] INFO: glideinFrontendElement:1875:    34( 1986    34     0     0)     2(    0 20000) |     0     0     0     0 |     0     0     0 |     0     0 | Down Sum of down factories
[2018-04-16 15:33:19,858] INFO: glideinFrontendElement:1875:     0(    0     0     0     0)     0(    0     0) |     0     0     0     0 |     0     0     0 |     0     0 | Down Unmatched
[2018-04-16 15:33:19,886] INFO: glideinFrontendElement:792: Advertising global and singular requests for factory fermicloud137.fnal.gov
[2018-04-16 15:33:19,889] DEBUG: glideinFrontendInterface:1020: In create Advertize work
[2018-04-16 15:33:19,889] DEBUG: glideinFrontendInterface:1062: Checking Credential file /scratch/llobato/pilot_proxy ...
[2018-04-16 15:33:19,890] DEBUG: glideinFrontendInterface:1105: Advertizing credential /scratch/llobato/pilot_proxy with (5 idle, 21 max run) for request ITB_FC_CE2@gfactory_instanc

Gonna do more tests incrasing number of jobs to make sure there problem doesn't persist.

#5 Updated by Lorena Lobato Pardavila about 1 year ago

  • Status changed from Assigned to Resolved

After done several tests (groups of 10, 20, 50, 100, 150 and 200 jobs), it seems to me that problem doesn't persist. In none of the cases I have observed any slow ramp-up in the down entries between interactions when calculating the glidein to assign to each entry.

Ex:

[2018-04-25 11:39:40,464] INFO: glideinFrontendElement:1875:     0(    0     0     0     0)     0(    0 10000) |     0     0     0     0 |     0     0     0 |     0     0 | Up   CMSHTPC_T3_US_Omaha_tusker_2x@gfactory_instance@ITBGOC@glidein-itb.grid.iu.edu
[2018-04-25 11:39:40,465] INFO: glideinFrontendElement:1875:     0(    0     0     0     0)     0(    0 10000) |     0     0     0     0 |     0     0     0 |     0     0 | Down CMS_T1_US_FNAL_condce3_opp@gfactory_instance@ITBGOC@glidein-itb.grid.iu.edu
[2018-04-25 11:39:40,467] INFO: glideinFrontendElement:1875:     0(    0     0     0     0)     0(    0 10000) |     0     0     0     0 |     0     0     0 |     0     0 | Down CMS_T1_US_FNAL_condce_opp1@gfactory_instance@ITBGOC@glidein-itb.grid.iu.edu
[2018-04-25 11:39:40,472] INFO: glideinFrontendElement:1875:     0(    0     0     0     0)     0(    0 10000) |     0     0     0     0 |     0     0     0 |     0     0 | Up   CMS_T2_US_Nebraska_Red_gw1_whole_op@gfactory_instance@ITBGOC@glidein-itb.grid.iu.edu
[2018-04-25 11:40:40,522] INFO: glideinFrontendElement:1875:     2(  100     2     0     0)     0(    0 10000) |     0     0     0     0 |     0     0     0 |     1     2 | Up   CMSHTPC_T3_US_Omaha_tusker_2x@gfactory_instance@ITBGOC@glidein-itb.grid.iu.edu
[2018-04-25 11:40:40,524] INFO: glideinFrontendElement:1875:     2(  100     2     0     0)     0(    0 10000) |     0     0     0     0 |     0     0     0 |     0     0 | Down CMS_T1_US_FNAL_condce3_opp@gfactory_instance@ITBGOC@glidein-itb.grid.iu.edu
[2018-04-25 11:40:40,526] INFO: glideinFrontendElement:1875:     2(  100     2     0     0)     0(    0 10000) |     0     0     0     0 |     0     0     0 |     0     0 | Down CMS_T1_US_FNAL_condce_opp1@gfactory_instance@ITBGOC@glidein-itb.grid.iu.edu
[2018-04-25 11:40:40,528] INFO: glideinFrontendElement:1875:     2(  100     2     0     0)     0(    0 10000) |     0     0     0     0 |     0     0     0 |     1     3 | Up   CMS_T2_US_Nebraska_Red_gw1_whole_op@gfactory_instance@ITBGOC@glidein-itb.grid.iu.edu
[2018-04-25 11:42:40,724] INFO: glideinFrontendElement:1875:     2(   93     2     0     0)     1(    0 10000) |     0     0     0     0 |     0     0     0 |     1     2 | Up   CMSHTPC_T3_US_Omaha_tusker_2x@gfactory_instance@ITBGOC@glidein-itb.grid.iu.edu
[2018-04-25 11:42:40,726] INFO: glideinFrontendElement:1875:     2(   93     2     0     0)     1(    0 10000) |     0     0     0     0 |     0     0     0 |     0     0 | Down CMS_T1_US_FNAL_condce3_opp@gfactory_instance@ITBGOC@glidein-itb.grid.iu.edu
[2018-04-25 11:42:40,728] INFO: glideinFrontendElement:1875:     2(   93     2     0     0)     1(    0 10000) |     0     0     0     0 |     0     0     0 |     0     0 | Down CMS_T1_US_FNAL_condce_opp1@gfactory_instance@ITBGOC@glidein-itb.grid.iu.edu
[2018-04-25 11:42:40,730] INFO: glideinFrontendElement:1875:     2(   93     2     0     0)     1(    0 10000) |     0     0     0     0 |     0     0     0 |     1     3 | Up   CMS_T2_US_Nebraska_Red_gw1_whole_op@gfactory_instance@ITBGOC@glidein-itb.grid.iu.edu
[2018-04-25 11:43:40,713] INFO: glideinFrontendElement:1875:     2(   85     2     0     0)     2(    0 10000) |     0     0     0     0 |     0     0     0 |     1     2 | Up   CMSHTPC_T3_US_Omaha_tusker_2x@gfactory_instance@ITBGOC@glidein-itb.grid.iu.edu
[2018-04-25 11:43:40,714] INFO: glideinFrontendElement:1875:     2(   85     2     0     0)     2(    0 10000) |     0     0     0     0 |     0     0     0 |     0     0 | Down CMS_T1_US_FNAL_condce3_opp@gfactory_instance@ITBGOC@glidein-itb.grid.iu.edu
[2018-04-25 11:43:40,716] INFO: glideinFrontendElement:1875:     2(   85     2     0     0)     2(    0 10000) |     0     0     0     0 |     0     0     0 |     0     0 | Down CMS_T1_US_FNAL_condce_opp1@gfactory_instance@ITBGOC@glidein-itb.grid.iu.edu
[2018-04-25 11:43:40,718] INFO: glideinFrontendElement:1875:     2(   85     2     0     0)     2(    0 10000) |     0     0     0     0 |     0     0     0 |     1     3 | Up   CMS_T2_US_Nebraska_Red_gw1_whole_op@gfactory_instance@ITBGOC@glidein-itb.grid.iu.edu
[2018-04-25 11:44:41,204] INFO: glideinFrontendElement:1875:     2(   73     2     0     0)     3(    0 10000) |     0     0     0     0 |     0     0     0 |     1     2 | Up   CMSHTPC_T3_US_Omaha_tusker_2x@gfactory_instance@ITBGOC@glidein-itb.grid.iu.edu
[2018-04-25 11:44:41,205] INFO: glideinFrontendElement:1875:     2(   73     2     0     0)     3(    0 10000) |     0     0     0     0 |     0     0     0 |     0     0 | Down CMS_T1_US_FNAL_condce3_opp@gfactory_instance@ITBGOC@glidein-itb.grid.iu.edu
[2018-04-25 11:44:41,207] INFO: glideinFrontendElement:1875:     2(   73     2     0     0)     3(    0 10000) |     0     0     0     0 |     0     0     0 |     0     0 | Down CMS_T1_US_FNAL_condce_opp1@gfactory_instance@ITBGOC@glidein-itb.grid.iu.edu
[2018-04-25 11:44:41,209] INFO: glideinFrontendElement:1875:     2(   73     2     0     0)     3(    0 10000) |     0     0     0     0 |     0     0     0 |     1     3 | Up   CMS_T2_US_Nebraska_Red_gw1_whole_op@gfactory_instance@ITBGOC@glidein-itb.grid.iu.edu
[2018-04-25 11:46:40,871] INFO: glideinFrontendElement:1875:     1(   48     1     0     0)     5(    0 10000) |     0     0     0     0 |     0     0     0 |     1     2 | Up   CMSHTPC_T3_US_Omaha_tusker_2x@gfactory_instance@ITBGOC@glidein-itb.grid.iu.edu
[2018-04-25 11:46:40,874] INFO: glideinFrontendElement:1875:     1(   48     1     0     0)     5(    0 10000) |     0     0     0     0 |     0     0     0 |     0     0 | Down CMS_T1_US_FNAL_condce3_opp@gfactory_instance@ITBGOC@glidein-itb.grid.iu.edu
[2018-04-25 11:46:40,875] INFO: glideinFrontendElement:1875:     1(   48     1     0     0)     5(    0 10000) |     0     0     0     0 |     0     0     0 |     0     0 | Down CMS_T1_US_FNAL_condce_opp1@gfactory_instance@ITBGOC@glidein-itb.grid.iu.edu
[2018-04-25 11:46:40,877] INFO: glideinFrontendElement:1875:     1(   48     1     0     0)     5(    0 10000) |     0     0     0     0 |     0     0     0 |     1     2 | Up   CMS_T2_US_Nebraska_Red_gw1_whole_op@gfactory_instance@ITBGOC@glidein-itb.grid.iu.edu

......

This is one of the examples where it can be appreciated that jobs are running and the function splitting the glideins proportionally is not considering the matching ones in downtime.

#6 Updated by Marco Mambelli about 1 year ago

  • Target version changed from v3_2_23 to v3_4_0

#7 Updated by Marco Mambelli about 1 year ago

  • Status changed from Resolved to Closed


Also available in: Atom PDF