Project

General

Profile

Bug #4797

Ingestion Fail (I think)

Added by Chris D'Andrea about 6 years ago. Updated almost 6 years ago.

Status:
Assigned
Priority:
Low
Assignee:
Start date:
10/15/2013
Due date:
% Done:

0%

Estimated time:
Duration:

Description

Unknown error when trying to ingest hosts & apply tags in mass. Below is a copy of some of the output. I also attach the small script that I'm running.

I output the transient name and the host target id; then try uploading the host as a target, followed by posting on the transient a link to the host.

You can see for the top entry everything works; the host had previously been ingested, and the next line is returned by the successful post on the transient with the host name. For the next transient, shit went bad. The transient and host both previously existed, and I looked at the RA and DEC (only other information being used here), and neither is a strange value. It appears that the host ingestion is what was bad. But I don't know why.

DES12X3daw SVA1_COADD-2938744498 {
"detail": "identifier already exists: SVA1_COADD-2938744498",
"error": 400
} {u'post_id': u'525d6312abe8ac56c128eb93'}

DES13X3egg SVA1_COADD-2937822704
Traceback (most recent call last):
File "des-ingest-hosts.py", line 38, in <module>
atc_funcs.incept_target(name,item2,item3)
File "/data3/data2/home/cdandrea/DES/atc/byChris/atc_funcs.py", line 28, in incept_target
pprint.pprint( client.post( "targets", data ) )
File "/home/cdandrea/pythonTools/bin/atc.py", line 45, in post
stream = self.opener.open( request )
File "/data3/DECAM/global/Python-2.7.3/lib/python2.7/urllib2.py", line 400, in open
response = self._open(req, data)
File "/data3/DECAM/global/Python-2.7.3/lib/python2.7/urllib2.py", line 418, in _open
'_open', req)
File "/data3/DECAM/global/Python-2.7.3/lib/python2.7/urllib2.py", line 378, in _call_chain
result = func(*args)
File "/data3/DECAM/global/Python-2.7.3/lib/python2.7/urllib2.py", line 1215, in https_open
return self.do_open(httplib.HTTPSConnection, req)
File "/data3/DECAM/global/Python-2.7.3/lib/python2.7/urllib2.py", line 1177, in do_open
raise URLError(err)
urllib2.URLError: <urlopen error [Errno -2] Name or service not known>

des-ingest-hosts.py (942 Bytes) des-ingest-hosts.py Chris D'Andrea, 10/15/2013 05:12 PM

History

#1 Updated by Rollin Thomas about 6 years ago

I don't think this is an ATC bug. It looks like something like the URL being formed and passed to the open( request ) call is messed up but I don't see exactly how this can be okay for one target but fail for the next. If you try to incept the target X3egg target again, what happens?

Googling the error message I see that there are a bunch of cases that cause it, one of them being connection throttling, another being that someone's Python installation was messed up.

Would be helpful to get a copy of your atc_funcs module to inspect as well.

If it's connection throttling (too many connections may be disallowed by your network) then we may need to come up with a better bulk ingestion solution.

#2 Updated by Rollin Thomas almost 6 years ago

  • Priority changed from Normal to Low

I'm going to push the priority down on this one. I don't think it's a bug with ATC, and you evidently were able to get something going on your end. I think we could see a lot less bugs and a lot faster interaction ith ATC from the REST interface if I added batch POSTs but that may need to wait till post-season.

#3 Updated by Rollin Thomas almost 6 years ago

  • Status changed from New to Assigned


Also available in: Atom PDF