Support dynamic connection lifetime between client and the server
Currently client sets the CURL_TIMEOUT to 30 sec. This results in the max connection time to stay active to be 30 sec. This fixed timeout doesn't work when dealing with server side tasks that can take long. For example, when compressing a sandbox, submitting several hundred jobs, queering a busy schedd, etc.
We need a means to support dynamic connection lifetimes that handle timeouts more gracefully.
Possible approach involves some combination of following changes
- Change CURL_TIMEOUT to something large like 10 min or not use this at all. Client times out the connection using CURLOPT_LOW_SPEED_TIME and CURLOPT_LOW_SPEED_LIMIT
- Server periodically sends busy status to the client that client understands and displays the info on the screen appropriately. We may need to use CURLOPT_XFERINFOFUNCTION/CURLOPT_PROGRESSFUNCTION for this.