glideinWMS

This would depend a lot on Issue #2180 to reduce the size of the pilot logs that are currently returned from the pilots.

EC2 API defines ec2-get-console-output which allows you to get the last 64kb of the console output of a terminated VM. This data is saved for up to an hour after termination. I have put in a request for the Condor team to implement this call in the ec2_gahp so that we can retrieve logs using condor on the factory.

The log retrieval process would look like:

1) launch vm - save contextualization logs
2) launch pilot - save logs
3) shutdown
3a) aggressively compress logs
3b) base64 encode the compressed logs
3c) write base64 encoded logs to the console
4) the ec2_gahp will retrieve the console output and write it out as a job log.

Now the factory has all the logs from the VM. We can debug as necessary.