Project

General

Profile

igor.txt

Igor's description of problem - Burt Holzman, 04/05/2012 03:25 PM

 
1
Subject: More factory bugs
2
From: Igor Sfiligoi <sfiligoi@fnal.gov>
3
Date: 04/05/12 14:43
4
To: glideinwms <glideinwms@fnal.gov>
5
CC: Timothy Mortensen <tmortensen@ucsd.edu>, jeffrey dost <jdost@ucsd.edu>
6

    
7
Hi guys.
8

    
9
I think we just hit a new factory bug!
10
One of the FEs for this entry is requesting glideins, but getting none!
11

    
12
In particular
13
entry: CMS_T3_US_Omaha_tusker
14
fe:    EngageVO-submit3-v1_0.main
15

    
16
Below the details.
17

    
18
This has been going on for hours, so it is a major problem for us.
19

    
20
Any idea welcome... investigating further now.
21

    
22
Igor
23

    
24

    
25
[1232] gfactory@glidein-1 ~$ condor_status -any -const 'ReqGlidein == "CMS_T3_US_Omaha_tusker@Production_v4_3@UCSD"&&ReqIdleGlideins=!=UNDEFINED' -format '%-90s ' Name -format '%5i ' ReqIdleGlideins -format '%5i\n' ReqMaxRunningGlideins
26
CMS_T3_US_Omaha_tusker@Production_v4_3@UCSD@CHTC-glidein-v1_0.main                            41  2625
27
CMS_T3_US_Omaha_tusker@Production_v4_3@UCSD@CMSVO-vocms157-ITB-v2_5_2.main                     0     0
28
CMS_T3_US_Omaha_tusker@Production_v4_3@UCSD@EngageVO-submit3-v1_0.main                        21   487
29
CMS_T3_US_Omaha_tusker@Production_v4_3@UCSD@EngageVO-submit3-v1_0.main_no_voms                 1   172
30
CMS_T3_US_Omaha_tusker@Production_v4_3@UCSD@EngageVO-submit3-v1_0.targeted_sites               0     0
31
CMS_T3_US_Omaha_tusker@Production_v4_3@UCSD@NEBioGrid-v1_2.main                                8    15
32
CMS_T3_US_Omaha_tusker@Production_v4_3@UCSD@UCSD-v5_3.main                                     0     1
33
CMS_T3_US_Omaha_tusker@Production_v4_3@UCSD@UCSDRok-v1_0.offsite                               0     0
34
CMS_T3_US_Omaha_tusker@Production_v4_3@UCSD@glidein-unl-edu_OSG_gWMSFrontend.T2Overflow        0     0
35
CMS_T3_US_Omaha_tusker@Production_v4_3@UCSD@glidein-unl-edu_OSG_gWMSFrontend.main              1   920
36
CMS_T3_US_Omaha_tusker@Production_v4_3@UCSD@nanoHUB-v1_0.main                                  0     1
37
CMS_T3_US_Omaha_tusker@Production_v4_3@UCSD@nanoHUB-v1_0.targeted_sites                        0     1
38
CMS_T3_US_Omaha_tusker@Production_v4_3@UCSD@osg-xsede-grid-iu-edu_OSG_gWMSFrontend.main        0     0
39

    
40
[1234] gfactory@glidein-1 ~/glideinsubmit/glidein_Production_v4_3$ less log/entry_CMS_T3_US_Omaha_tusker/factory.20120405.info.log
41
[2012-04-05T12:33:44-07:00 75710] Iteration at Thu Apr  5 12:33:44 2012
42
[2012-04-05T12:33:47-07:00 75710] Found 13 tasks to work on using existing factory key.
43
[2012-04-05T12:33:47-07:00 75710] Found 13 total tasks to work on
44
[2012-04-05T12:33:48-07:00 75710] Checking downtime for frontend glowVO security class: frontend (entry CMS_T3_US_Omaha_tusker).
45
[2012-04-05T12:33:48-07:00 75710] Client CHTC-glidein-v1_0.main (secid: glowVO_frontend) requesting 41 glideins, max running 2625, remove excess 'NO'
46
[2012-04-05T12:33:48-07:00 75710]   Params: {'GLIDEIN_Glexec_Use': 'NEVER', u'GLIDECLIENT_Start': u'TARGET.WantGlidein', u'USE_MATCH_AUTH': u'True', u'GLIDECLIENT_Group_Start': u'(isUndefined(TARGET.estimated_run_hours) || TARGET.estimated_run_hours*3600 <= GLIDEIN_ToDie-CurrentTime)', u'GLIDEIN_Entry_Start': u'True', u'GLIDEIN_Entry_Rank': u'1', u'GLIDEIN_Collector': u'glidein.chtc.wisc.edu:9620-9669'}
47
[2012-04-05T12:33:48-07:00 75710]   Decrypted Param Names: [u'SecurityName', u'nr_x509_proxies', u'x509_proxy_0', u'x509_proxy_0_identifier', u'x509_proxy_0_security_class']
48
[2012-04-05T12:33:48-07:00 75710] Client CHTC-glidein-v1_0.main (secid: glowVO_frontend) schedd status {1: 0, 2: 471, 5: 2}
49
[2012-04-05T12:33:48-07:00 75710] Info: qc: {2: 471, 5: 2}
50
[2012-04-05T12:33:48-07:00 75710] Submitted 10 glideins to schedd_glideins7@glidein-1.t2.ucsd.edu: [(462565L, 0), (462565L, 1), (462565L, 2), (462565L, 3), (462565L, 4), (462565L, 5), (462565L, 6), (462565L, 7), (462565L, 8), (462565L, 9)]
51
[2012-04-05T12:33:48-07:00 75710] Checking downtime for frontend nanoHUB security class: frontend (entry CMS_T3_US_Omaha_tusker).
52
[2012-04-05T12:33:49-07:00 75710] Client nanoHUB-v1_0.main (secid: nanoHUB_frontend) requesting 0 glideins, max running 1, remove excess 'NO'
53
[2012-04-05T12:33:49-07:00 75710]   Params: {'GLIDEIN_Glexec_Use': 'NEVER', u'GLIDECLIENT_Start': u'(DESIRED_Sites=?=UNDEFINED) || (DESIRED_Sites=?="any") || stringListMember(GLIDEIN_Entry_Name,DESIRED_Sites)', u'USE_MATCH_AUTH': u'True', u'GLIDEIN_Entry_Start': u'True', u'GLIDEIN_Entry_Rank': u'1', u'GLIDEIN_Collector': u'devcondor.nanohub.org:9620-9629'}
54
[2012-04-05T12:33:49-07:00 75710]   Decrypted Param Names: [u'SecurityName', u'nr_x509_proxies', u'x509_proxy_0', u'x509_proxy_0_identifier', u'x509_proxy_0_security_class']
55
[2012-04-05T12:33:49-07:00 75710] Client nanoHUB-v1_0.main (secid: nanoHUB_frontend) schedd status {1: 0}
56
[2012-04-05T12:33:49-07:00 75710] Info: qc: {}
57
[2012-04-05T12:33:49-07:00 75710] Have enough glideins: 0=idle 0=req_idle, not submitting
58
[2012-04-05T12:33:49-07:00 75710] Checking downtime for frontend UCSDCMS security class: cmspilot (entry CMS_T3_US_Omaha_tusker).
59
[2012-04-05T12:33:49-07:00 75710] Checking downtime for frontend UCSDCMS security class: cmspilot (entry CMS_T3_US_Omaha_tusker).
60
[2012-04-05T12:33:49-07:00 75710] Checking downtime for frontend UCSDCMS security class: cmspilot (entry CMS_T3_US_Omaha_tusker).
61
[2012-04-05T12:33:49-07:00 75710] Checking downtime for frontend UCSDCMS security class: cmspilot (entry CMS_T3_US_Omaha_tusker).
62
[2012-04-05T12:33:49-07:00 75710] Checking downtime for frontend UCSDCMS security class: cmspilot (entry CMS_T3_US_Omaha_tusker).
63
[2012-04-05T12:33:49-07:00 75710] Checking downtime for frontend UCSDCMS security class: cmspilot (entry CMS_T3_US_Omaha_tusker).
64
[2012-04-05T12:33:49-07:00 75710] Checking downtime for frontend UCSDCMS security class: cmspilot (entry CMS_T3_US_Omaha_tusker).
65
[2012-04-05T12:33:49-07:00 75710] Checking downtime for frontend UCSDCMS security class: cmspilot (entry CMS_T3_US_Omaha_tusker).
66
[2012-04-05T12:33:50-07:00 75710] Checking downtime for frontend UCSDCMS security class: cmspilot (entry CMS_T3_US_Omaha_tusker).
67
[2012-04-05T12:33:50-07:00 75710] Checking downtime for frontend UCSDCMS security class: cmspilot (entry CMS_T3_US_Omaha_tusker).
68
[2012-04-05T12:33:50-07:00 75710] Client UCSD-v5_3.main (secid: UCSDCMS_cmspilot) requesting 0 glideins, max running 1, remove excess 'NO'
69
[2012-04-05T12:33:50-07:00 75710]   Params: {u'GLIDEIN_Glexec_Use': u'OPTIONAL', u'USE_MATCH_AUTH': u'True', u'GLIDEIN_Collector': u'glidein-collector.t2.ucsd.edu:9620-9919'}
70
[2012-04-05T12:33:50-07:00 75710]   Decrypted Param Names: [u'x509_proxy_7_identifier', u'x509_proxy_1_identifier', u'x509_proxy_4_security_class', u'SecurityName', u'x509_proxy_9', u'x509_proxy_2_security_class', u'nr_x509_proxies', u'x509_proxy_4_identifier', u'x509_proxy_7_security_class', u'x509_proxy_8_security_class', u'x509_proxy_1_security_class', u'x509_proxy_3_security_class', u'x509_proxy_3_identifier', u'x509_proxy_8_identifier', u'x509_proxy_6_security_class', u'x509_proxy_5_security_class', u'x509_proxy_4', u'x509_proxy_5', u'x509_proxy_6', u'x509_proxy_7', u'x509_proxy_0', u'x509_proxy_1', u'x509_proxy_2', u'x509_proxy_0_security_class', u'x509_proxy_3', u'x509_proxy_8', u'x509_proxy_0_identifier', u'x509_proxy_9_security_class', u'x509_proxy_9_identifier', u'x509_proxy_6_identifier', u'x509_proxy_5_identifier', u'x509_proxy_2_identifier']
71
[2012-04-05T12:33:50-07:00 75710] Client UCSD-v5_3.main (secid: UCSDCMS_cmspilot) schedd status {1: 0, 2: 16}
72
[2012-04-05T12:33:50-07:00 75710] Info: qc: {2: 1}
73
[2012-04-05T12:33:50-07:00 75710] Have enough glideins: 0=idle 0=req_idle, not submitting
74
[2012-04-05T12:33:50-07:00 75710] Info: qc: {2: 1}
75
[2012-04-05T12:33:50-07:00 75710] Have enough glideins: 0=idle 0=req_idle, not submitting
76
[2012-04-05T12:33:50-07:00 75710] Info: qc: {2: 2}
77
[2012-04-05T12:33:50-07:00 75710] Have enough glideins: 0=idle 0=req_idle, not submitting
78
[2012-04-05T12:33:50-07:00 75710] Info: qc: {2: 1}
79
[2012-04-05T12:33:50-07:00 75710] Have enough glideins: 0=idle 0=req_idle, not submitting
80
[2012-04-05T12:33:50-07:00 75710] Info: qc: {2: 2}
81
[2012-04-05T12:33:50-07:00 75710] Have enough glideins: 0=idle 0=req_idle, not submitting
82
[2012-04-05T12:33:50-07:00 75710] Info: qc: {2: 1}
83
[2012-04-05T12:33:50-07:00 75710] Have enough glideins: 0=idle 0=req_idle, not submitting
84
[2012-04-05T12:33:50-07:00 75710] Info: qc: {2: 2}
85
[2012-04-05T12:33:50-07:00 75710] Have enough glideins: 0=idle 0=req_idle, not submitting
86
[2012-04-05T12:33:50-07:00 75710] Info: qc: {2: 2}
87
[2012-04-05T12:33:50-07:00 75710] Have enough glideins: 0=idle 0=req_idle, not submitting
88
[2012-04-05T12:33:50-07:00 75710] Info: qc: {2: 2}
89
[2012-04-05T12:33:50-07:00 75710] Have enough glideins: 0=idle 0=req_idle, not submitting
90
[2012-04-05T12:33:50-07:00 75710] Info: qc: {2: 2}
91
[2012-04-05T12:33:50-07:00 75710] Have enough glideins: 0=idle 0=req_idle, not submitting
92
[2012-04-05T12:33:50-07:00 75710] Checking downtime for frontend EngageVO-submit3 security class: frontend (entry CMS_T3_US_Omaha_tusker).
93
[2012-04-05T12:33:50-07:00 75710] Client EngageVO-submit3-v1_0.main_no_voms (secid: EngageVO-submit3_frontend) requesting 1 glideins, max running 172, remove excess 'NO'
94
[2012-04-05T12:33:50-07:00 75710]   Params: {'GLIDEIN_Glexec_Use': 'NEVER', u'GLIDECLIENT_Rank': u'1', u'GLIDECLIENT_Start': u'True', u'GLIDEIN_Monitoring_Enabled': u'False', u'USE_MATCH_AUTH': u'True', u'GLIDEIN_Collector': u'engage-submit3.renci.org:9620,engage-submit3.renci.org:9621,engage-submit3.renci.org:9622,engage-submit3.renci.org:9623,engage-submit3.renci.org:9624,engage-submit3.renci.org:9625,engage-submit3.renci.org:9626,engage-submit3.renci.org:9627,engage-submit3.renci.org:9628,engage-submit3.renci.org:9629,engage-submit3.renci.org:9630,engage-submit3.renci.org:9631,engage-submit3.renci.org:9632,engage-submit3.renci.org:9633,engage-submit3.renci.org:9634,engage-submit3.renci.org:9635,engage-submit3.renci.org:9636,engage-submit3.renci.org:9637,engage-submit3.renci.org:9638,engage-submit3.renci.org:9639,engage-submit3.renci.org:9640,engage-submit3.renci.org:9641,engage-submit3.renci.org:9642,engage-submit3.renci.org:9643,engage-submit3.renci.org:9644,engage-submit3.renci.org:9645,engage-submit3.renci.org:9646,engage-submit3.renci.org:9647,engage-submit3.renci.org:9648,engage-submit3.renci.org:9649,engage-submit3.renci.org:9650,engage-submit3.renci.org:9651,engage-submit3.renci.org:965
95
2,engage-submit3.renci.org:9653,engage-submit3.renci.org:9654,engage-submit3.renci.org:9655,engage-submit3.renci.org:9656,engage-submit3.renci.org:9657,engage-submit3.renci.org:9658,engage-submit3.renci.org:9659,engage-submit3.renci.org:9660,engage-submit3.renci.org:9661,engage-submit3.renci.org:9662,engage-submit3.renci.org:9663,engage-submit3.renci.org:9664,engage-submit3.renci.org:9665,engage-submit3.renci.org:9666,engage-submit3.renci.org:9667,engage-submit3.renci.org:9668,engage-submit3.renci.org:9669'}
96
[2012-04-05T12:33:50-07:00 75710]   Decrypted Param Names: [u'SecurityName', u'nr_x509_proxies', u'x509_proxy_0', u'x509_proxy_0_identifier', u'x509_proxy_0_security_class']
97
[2012-04-05T12:33:50-07:00 75710] Client EngageVO-submit3-v1_0.main_no_voms (secid: EngageVO-submit3_frontend) schedd status {1: 0, 2: 151}
98
[2012-04-05T12:33:50-07:00 75710] Info: qc: {2: 151}
99
[2012-04-05T12:33:51-07:00 75710] Submitted 1 glideins to schedd_glideins7@glidein-1.t2.ucsd.edu: [(462566L, 0)]
100
[2012-04-05T12:33:51-07:00 75710] Checking downtime for frontend CMS-CERN security class: frontend (entry CMS_T3_US_Omaha_tusker).
101
[2012-04-05T12:33:51-07:00 75710] Client CMSVO-vocms157-ITB-v2_5_2.main (secid: CMS-CERN_frontend) requesting 0 glideins, max running 0, remove excess 'ALL'
102
[2012-04-05T12:33:51-07:00 75710]   Params: {'GLIDEIN_Glexec_Use': 'NEVER', u'GLIDECLIENT_Rank': u'1', u'GLIDECLIENT_Start': u'True', u'USE_MATCH_AUTH': u'True', u'GLIDECLIENT_Group_Start': u'(stringListMember(GLIDEIN_CMSSite,DESIRED_Sites,",")=?=True)&&((GLIDEIN_Is_HTPC=?=True)=?=(DESIRES_HTPC=?=True))', u'GLIDEIN_Collector': u'vocms105.cern.ch:9620-10019'}
103
[2012-04-05T12:33:51-07:00 75710]   Decrypted Param Names: [u'SecurityName', u'nr_x509_proxies', u'x509_proxy_0', u'x509_proxy_0_identifier', u'x509_proxy_0_security_class']
104
[2012-04-05T12:33:51-07:00 75710] Client CMSVO-vocms157-ITB-v2_5_2.main (secid: CMS-CERN_frontend) schedd status {1: 0}
105
[2012-04-05T12:33:51-07:00 75710] Info: qc: {}
106
[2012-04-05T12:33:51-07:00 75710] Have enough glideins: 0=idle 0=req_idle, not submitting
107
[2012-04-05T12:33:51-07:00 75710] Checking downtime for frontend EngageVO-submit3 security class: frontend (entry CMS_T3_US_Omaha_tusker).
108
[2012-04-05T12:33:51-07:00 75710] Client EngageVO-submit3-v1_0.targeted_sites (secid: EngageVO-submit3_frontend) requesting 0 glideins, max running 0, remove excess 'ALL'
109
[2012-04-05T12:33:51-07:00 75710]   Params: {'GLIDEIN_Glexec_Use': 'NEVER', u'GLIDECLIENT_Rank': u'1', u'GLIDECLIENT_Start': u'True', u'GLIDEIN_Monitoring_Enabled': u'False', u'USE_MATCH_AUTH': u'True', u'GLIDEIN_Collector': u'engage-submit3.renci.org:9620,engage-submit3.renci.org:9621,engage-submit3.renci.org:9622,engage-submit3.renci.org:9623,engage-submit3.renci.org:9624,engage-submit3.renci.org:9625,engage-submit3.renci.org:9626,engage-submit3.renci.org:9627,engage-submit3.renci.org:9628,engage-submit3.renci.org:9629,engage-submit3.renci.org:9630,engage-submit3.renci.org:9631,engage-submit3.renci.org:9632,engage-submit3.renci.org:9633,engage-submit3.renci.org:9634,engage-submit3.renci.org:9635,engage-submit3.renci.org:9636,engage-submit3.renci.org:9637,engage-submit3.renci.org:9638,engage-submit3.renci.org:9639,engage-submit3.renci.org:9640,engage-submit3.renci.org:9641,engage-submit3.renci.org:9642,engage-submit3.renci.org:9643,engage-submit3.renci.org:9644,engage-submit3.renci.org:9645,engage-submit3.renci.org:9646,engage-submit3.renci.org:9647,engage-submit3.renci.org:9648,engage-submit3.renci.org:9649,engage-submit3.renci.org:9650,engage-submit3.renci.org:9651,engage-submit3.renci.org:965
110
2,engage-submit3.renci.org:9653,engage-submit3.renci.org:9654,engage-submit3.renci.org:9655,engage-submit3.renci.org:9656,engage-submit3.renci.org:9657,engage-submit3.renci.org:9658,engage-submit3.renci.org:9659,engage-submit3.renci.org:9660,engage-submit3.renci.org:9661,engage-submit3.renci.org:9662,engage-submit3.renci.org:9663,engage-submit3.renci.org:9664,engage-submit3.renci.org:9665,engage-submit3.renci.org:9666,engage-submit3.renci.org:9667,engage-submit3.renci.org:9668,engage-submit3.renci.org:9669'}
111
[2012-04-05T12:33:51-07:00 75710]   Decrypted Param Names: [u'SecurityName', u'nr_x509_proxies', u'x509_proxy_0', u'x509_proxy_0_identifier', u'x509_proxy_0_security_class']
112
[2012-04-05T12:33:51-07:00 75710] Client EngageVO-submit3-v1_0.targeted_sites (secid: EngageVO-submit3_frontend) schedd status {1: 0}
113
[2012-04-05T12:33:51-07:00 75710] Info: qc: {}
114
[2012-04-05T12:33:51-07:00 75710] Have enough glideins: 0=idle 0=req_idle, not submitting
115
[2012-04-05T12:33:51-07:00 75710] Too many glideins: idle=1, running=151, max_running=0
116
[2012-04-05T12:33:51-07:00 75710] Removing 0 running glideins
117
[2012-04-05T12:33:51-07:00 75710] No glideins requested.
118
[2012-04-05T12:33:51-07:00 75710] Checking downtime for frontend OSG_XSEDE security class: frontend (entry CMS_T3_US_Omaha_tusker).
119
[2012-04-05T12:33:51-07:00 75710] Client osg-xsede-grid-iu-edu_OSG_gWMSFrontend.main (secid: OSG_XSEDE_frontend) requesting 0 glideins, max running 0, remove excess 'NO'
120
[2012-04-05T12:33:51-07:00 75710]   Params: {'GLIDEIN_Glexec_Use': 'NEVER', u'USE_MATCH_AUTH': u'True', u'GLIDECLIENT_Rank': u'1', u'GLIDEIN_Collector': u'osg-xsede.grid.iu.edu:9701-9750'}
121
[2012-04-05T12:33:51-07:00 75710]   Decrypted Param Names: [u'SecurityName', u'nr_x509_proxies', u'x509_proxy_0', u'x509_proxy_0_identifier', u'x509_proxy_0_security_class']
122
[2012-04-05T12:33:51-07:00 75710] Client osg-xsede-grid-iu-edu_OSG_gWMSFrontend.main (secid: OSG_XSEDE_frontend) schedd status {1: 0, 2: 1}
123
[2012-04-05T12:33:51-07:00 75710] Info: qc: {2: 1}
124
[2012-04-05T12:33:51-07:00 75710] Have enough glideins: 0=idle 0=req_idle, not submitting
125
[2012-04-05T12:33:51-07:00 75710] Checking downtime for frontend NeBioGrid security class: frontend (entry CMS_T3_US_Omaha_tusker).
126
[2012-04-05T12:33:51-07:00 75710] WARNING: Security Class frontend is currently in a downtime window for Entry: CMS_T3_US_Omaha_tusker. Skipping proxy 0.
127
[2012-04-05T12:33:51-07:00 75710] WARNING: Found proxies for client NEBioGrid-v1_2.main but the security class was in downtime, setting entry into downtime for advertising
128
[2012-04-05T12:33:51-07:00 75710] Checking downtime for frontend hcc-glidein security class: frontend (entry CMS_T3_US_Omaha_tusker).
129
[2012-04-05T12:33:51-07:00 75710] Client glidein-unl-edu_OSG_gWMSFrontend.T2Overflow (secid: hcc-glidein_frontend) requesting 0 glideins, max running 0, remove excess 'ALL'
130
[2012-04-05T12:33:51-07:00 75710]   Params: {u'GLIDEIN_Glexec_Use': u'OPTIONAL', u'IsT2Overflow': u'True', u'PREEMPT_GRACE_TIME': 34800, u'USE_MATCH_AUTH': u'True', u'GLIDECLIENT_Group_Start': u'(TARGET.IsT2Overflow =?= TRUE)', u'GLIDEIN_Entry_Rank': u'((Owner=!="boinc")*10000)', u'GLIDEIN_Collector': u'glidein.unl.edu:9620-9819'}
131
[2012-04-05T12:33:51-07:00 75710]   Decrypted Param Names: [u'SecurityName', u'nr_x509_proxies', u'x509_proxy_0', u'x509_proxy_0_identifier', u'x509_proxy_0_security_class']
132
[2012-04-05T12:33:51-07:00 75710] Client glidein-unl-edu_OSG_gWMSFrontend.T2Overflow (secid: hcc-glidein_frontend) schedd status {1: 0}
133
[2012-04-05T12:33:51-07:00 75710] Info: qc: {}
134
[2012-04-05T12:33:51-07:00 75710] Have enough glideins: 0=idle 0=req_idle, not submitting
135
[2012-04-05T12:33:51-07:00 75710] Checking downtime for frontend nanoHUB security class: frontend (entry CMS_T3_US_Omaha_tusker).
136
[2012-04-05T12:33:52-07:00 75710] Client nanoHUB-v1_0.targeted_sites (secid: nanoHUB_frontend) requesting 0 glideins, max running 1, remove excess 'NO'
137
[2012-04-05T12:33:52-07:00 75710]   Params: {'GLIDEIN_Glexec_Use': 'NEVER', u'GLIDECLIENT_Start': u'(DESIRED_Sites=?=UNDEFINED) || (DESIRED_Sites=?="any") || stringListMember(GLIDEIN_Entry_Name,DESIRED_Sites)', u'USE_MATCH_AUTH': u'True', u'GLIDEIN_Entry_Start': u'True', u'GLIDEIN_Entry_Rank': u'1', u'GLIDEIN_Collector': u'devcondor.nanohub.org:9620-9629'}
138
[2012-04-05T12:33:52-07:00 75710]   Decrypted Param Names: [u'SecurityName', u'nr_x509_proxies', u'x509_proxy_0', u'x509_proxy_0_identifier', u'x509_proxy_0_security_class']
139
[2012-04-05T12:33:52-07:00 75710] Client nanoHUB-v1_0.targeted_sites (secid: nanoHUB_frontend) schedd status {1: 0}
140
[2012-04-05T12:33:52-07:00 75710] Info: qc: {}
141
[2012-04-05T12:33:52-07:00 75710] Have enough glideins: 0=idle 0=req_idle, not submitting
142
[2012-04-05T12:33:52-07:00 75710] Checking downtime for frontend UCSDRok security class: frontend (entry CMS_T3_US_Omaha_tusker).
143
[2012-04-05T12:33:52-07:00 75710] Client UCSDRok-v1_0.offsite (secid: UCSDRok_frontend) requesting 0 glideins, max running 0, remove excess 'ALL'
144
[2012-04-05T12:33:52-07:00 75710]   Params: {u'GLIDEIN_Java_Use': u'True', u'GLIDEIN_Job_Max_Time': 43000, 'GLIDEIN_Glexec_Use': 'NEVER', u'USE_MATCH_AUTH': u'True', u'GLIDEIN_Entry_Start': u'(DESIRED_Sites=?="any") || stringListMember(GLIDEIN_Site,DESIRED_Sites)', u'GLIDEIN_Entry_Rank': u'1', u'GLIDEIN_Collector': u'osg.ctbp.ucsd.edu:9620-9719'}
145
[2012-04-05T12:33:52-07:00 75710]   Decrypted Param Names: [u'SecurityName', u'x509_proxy_0', u'x509_proxy_0_identifier', u'nr_x509_proxies', u'x509_proxy_0_security_class']
146
[2012-04-05T12:33:52-07:00 75710] Client UCSDRok-v1_0.offsite (secid: UCSDRok_frontend) schedd status {1: 0}
147
[2012-04-05T12:33:52-07:00 75710] Info: qc: {}
148
[2012-04-05T12:33:52-07:00 75710] Have enough glideins: 0=idle 0=req_idle, not submitting
149
[2012-04-05T12:33:52-07:00 75710] Checking downtime for frontend hcc-glidein security class: frontend (entry CMS_T3_US_Omaha_tusker).
150
[2012-04-05T12:33:52-07:00 75710] Client glidein-unl-edu_OSG_gWMSFrontend.main (secid: hcc-glidein_frontend) requesting 0 glideins, max running 920, remove excess 'NO'
151
[2012-04-05T12:33:52-07:00 75710]   Params: {u'GLIDEIN_Glexec_Use': u'OPTIONAL', u'USE_MATCH_AUTH': u'True', u'PREEMPT_GRACE_TIME': 34800, u'GLIDEIN_Entry_Rank': u'((Owner=!="boinc")*10000)', u'GLIDEIN_Collector': u'glidein.unl.edu:9620-9819'}
152
[2012-04-05T12:33:52-07:00 75710]   Decrypted Param Names: [u'SecurityName', u'nr_x509_proxies', u'x509_proxy_0', u'x509_proxy_0_identifier', u'x509_proxy_0_security_class']
153
[2012-04-05T12:33:52-07:00 75710] Client glidein-unl-edu_OSG_gWMSFrontend.main (secid: hcc-glidein_frontend) schedd status {1: 0}
154
[2012-04-05T12:33:52-07:00 75710] Info: qc: {}
155
[2012-04-05T12:33:52-07:00 75710] Have enough glideins: 0=idle 0=req_idle, not submitting
156
[2012-04-05T12:33:52-07:00 75710] Checking downtime for frontend EngageVO-submit3 security class: frontend (entry CMS_T3_US_Omaha_tusker).
157
[2012-04-05T12:33:52-07:00 75710] Client EngageVO-submit3-v1_0.main (secid: EngageVO-submit3_frontend) requesting 0 glideins, max running 487, remove excess 'NO'
158
[2012-04-05T12:33:52-07:00 75710]   Params: {'GLIDEIN_Glexec_Use': 'NEVER', u'GLIDECLIENT_Rank': u'1', u'GLIDECLIENT_Start': u'True', u'GLIDEIN_Monitoring_Enabled': u'False', u'USE_MATCH_AUTH': u'True', u'GLIDEIN_Collector': u'engage-submit3.renci.org:9620,engage-submit3.renci.org:9621,engage-submit3.renci.org:9622,engage-submit3.renci.org:9623,engage-submit3.renci.org:9624,engage-submit3.renci.org:9625,engage-submit3.renci.org:9626,engage-submit3.renci.org:9627,engage-submit3.renci.org:9628,engage-submit3.renci.org:9629,engage-submit3.renci.org:9630,engage-submit3.renci.org:9631,engage-submit3.renci.org:9632,engage-submit3.renci.org:9633,engage-submit3.renci.org:9634,engage-submit3.renci.org:9635,engage-submit3.renci.org:9636,engage-submit3.renci.org:9637,engage-submit3.renci.org:9638,engage-submit3.renci.org:9639,engage-submit3.renci.org:9640,engage-submit3.renci.org:9641,engage-submit3.renci.org:9642,engage-submit3.renci.org:9643,engage-submit3.renci.org:9644,engage-submit3.renci.org:9645,engage-submit3.renci.org:9646,engage-submit3.renci.org:9647,engage-submit3.renci.org:9648,engage-submit3.renci.org:9649,engage-submit3.renci.org:9650,engage-submit3.renci.org:9651,engage-submit3.renci.org:965
159
2,engage-submit3.renci.org:9653,engage-submit3.renci.org:9654,engage-submit3.renci.org:9655,engage-submit3.renci.org:9656,engage-submit3.renci.org:9657,engage-submit3.renci.org:9658,engage-submit3.renci.org:9659,engage-submit3.renci.org:9660,engage-submit3.renci.org:9661,engage-submit3.renci.org:9662,engage-submit3.renci.org:9663,engage-submit3.renci.org:9664,engage-submit3.renci.org:9665,engage-submit3.renci.org:9666,engage-submit3.renci.org:9667,engage-submit3.renci.org:9668,engage-submit3.renci.org:9669'}
160
[2012-04-05T12:33:52-07:00 75710]   Decrypted Param Names: [u'SecurityName', u'nr_x509_proxies', u'x509_proxy_0', u'x509_proxy_0_identifier', u'x509_proxy_0_security_class']
161
[2012-04-05T12:33:52-07:00 75710] Client EngageVO-submit3-v1_0.main (secid: EngageVO-submit3_frontend) schedd status {1: 0}
162
[2012-04-05T12:33:52-07:00 75710] Info: qc: {}
163
[2012-04-05T12:33:52-07:00 75710] Have enough glideins: 0=idle 0=req_idle, not submitting
164
[2012-04-05T12:33:52-07:00 75710] Advertize
165
[2012-04-05T12:33:53-07:00 75710] Writing stats
166
[2012-04-05T12:33:53-07:00 75710] log_stats written
167
[2012-04-05T12:33:53-07:00 75710] qc_stats written
168
[2012-04-05T12:33:53-07:00 75710] rrd_stats written
169
[2012-04-05T12:33:53-07:00 75710] Sleep 60s
170

    
171

    
172
            <max_jobs comment="raised limit temporarily until untracked GLOW jobs complete" held="2000" idle="4000" running="10000">
173
               <default_max_jobs_frontend held="50" idle="100" running="5000"/>
174
               <max_job_frontends>
175
               </max_job_frontends>
176
            </max_jobs>
177
            <release max_per_cycle="20" sleep="0.2"/>
178
            <remove max_per_cycle="5" sleep="0.2"/>
179
            <restrictions require_voms_proxy="False"/>
180
            <submit cluster_size="10" max_per_cycle="10" sleep="2"/>