Project

General

Profile

Support #3625

Survey Strategy database

Added by Ben Hambrecht over 7 years ago. Updated over 7 years ago.

Status:
Closed
Priority:
Urgent
Assignee:
Start date:
03/27/2013
Due date:
04/19/2013
% Done:

0%

Estimated time:
Duration: 24

Description

Hi Santi,

your module looks already very good. I added a bit so it outputs a survey selection flag as well. But there is a problem when running it without recreating the database (after a previous run where it got recreated successfully). Under what circumstances does the database have to be recreated? Would it have to be if the pipe is run with a new set of galaxies? It would be inconvenient to have to recreate it in every production run.

1]: Fiber_Allocation
... Reading Param File for Fiber_Allocation
... running Survey_Strategy
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/mpl_toolkits/__init__.py:2: UserWarning: Module readline was already imported from /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/IPython/utils/rlineimpl.pyc, but /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/readline is being added to sys.path
  __import__('pkg_resources').declare_namespace(__name__)
Loading common configuration...
Loading command line configuration...
Setting file mode creation mask to 0002...
Setting up logging system...
27-03-2013 14:32  INFO     PAUdm_init      Log file in /Users/ben/DESpec.log
27-03-2013 14:32  INFO     PAUdm_init      Setting up local despecdb SQLite database...
The database will be ERASED and recreated. Do you want to proceed? (y/N): n
27-03-2013 14:32  INFO     PAUdm_init      Data Base not restored.
27-03-2013 14:32  INFO     PAUdm_init      PAUdm initialization complete.
27-03-2013 14:32  DEBUG    ObsSim          Observatory information
27-03-2013 14:32  DEBUG    ObsSim           Latitude: -30:10:10.8
27-03-2013 14:32  DEBUG    ObsSim           Longitide: -70:48:23.5
27-03-2013 14:32  DEBUG    ObsSim           Elevation: 2207.0
27-03-2013 14:32  DEBUG    ObsSim           Temperature: 15.0
27-03-2013 14:32  DEBUG    ObsSim           Pressure: 1010.0
Planning survey...
27-03-2013 14:32  INFO     ObsSim          Planning targets for field Test
27-03-2013 14:32  DEBUG    ObsSim          Field area is RA [54.000 to 80.000] - DEC [-40.000 to -27.000]
27-03-2013 14:32  INFO     ObsSim          Filter Tray tray_r
27-03-2013 14:32  DEBUG    ObsSim          Adding new target at RA: 55.663240  DEC: -38.702925
27-03-2013 14:32  DEBUG    ObsSim          Adding new target at RA: 58.104628  DEC: -38.702925
27-03-2013 14:32  DEBUG    ObsSim          Adding new target at RA: 60.546017  DEC: -38.702925
27-03-2013 14:32  DEBUG    ObsSim          Adding new target at RA: 62.987405  DEC: -38.702925
27-03-2013 14:32  DEBUG    ObsSim          Adding new target at RA: 65.428794  DEC: -38.702925
27-03-2013 14:32  DEBUG    ObsSim          Adding new target at RA: 67.870182  DEC: -38.702925
27-03-2013 14:32  DEBUG    ObsSim          Adding new target at RA: 70.311571  DEC: -38.702925
27-03-2013 14:32  DEBUG    ObsSim          Adding new target at RA: 72.752959  DEC: -38.702925
27-03-2013 14:32  DEBUG    ObsSim          Adding new target at RA: 75.194348  DEC: -38.702925
27-03-2013 14:32  DEBUG    ObsSim          Adding new target at RA: 77.635736  DEC: -38.702925
27-03-2013 14:32  DEBUG    ObsSim          Adding new target at RA: 76.442085  DEC: -37.052925
27-03-2013 14:32  DEBUG    ObsSim          Adding new target at RA: 74.054784  DEC: -37.052925
27-03-2013 14:32  DEBUG    ObsSim          Adding new target at RA: 71.667482  DEC: -37.052925
27-03-2013 14:32  DEBUG    ObsSim          Adding new target at RA: 69.280181  DEC: -37.052925
27-03-2013 14:32  DEBUG    ObsSim          Adding new target at RA: 66.892879  DEC: -37.052925
27-03-2013 14:32  DEBUG    ObsSim          Adding new target at RA: 64.505578  DEC: -37.052925
27-03-2013 14:32  DEBUG    ObsSim          Adding new target at RA: 62.118276  DEC: -37.052925
27-03-2013 14:32  DEBUG    ObsSim          Adding new target at RA: 59.730974  DEC: -37.052925
27-03-2013 14:32  DEBUG    ObsSim          Adding new target at RA: 57.343673  DEC: -37.052925
27-03-2013 14:32  DEBUG    ObsSim          Adding new target at RA: 58.512401  DEC: -35.402925
27-03-2013 14:32  DEBUG    ObsSim          Adding new target at RA: 60.849856  DEC: -35.402925
27-03-2013 14:32  DEBUG    ObsSim          Adding new target at RA: 63.187312  DEC: -35.402925
27-03-2013 14:32  DEBUG    ObsSim          Adding new target at RA: 65.524767  DEC: -35.402925
27-03-2013 14:32  DEBUG    ObsSim          Adding new target at RA: 67.862223  DEC: -35.402925
27-03-2013 14:32  DEBUG    ObsSim          Adding new target at RA: 70.199679  DEC: -35.402925
27-03-2013 14:32  DEBUG    ObsSim          Adding new target at RA: 72.537134  DEC: -35.402925
27-03-2013 14:32  DEBUG    ObsSim          Adding new target at RA: 74.874590  DEC: -35.402925
27-03-2013 14:32  DEBUG    ObsSim          Adding new target at RA: 77.212045  DEC: -35.402925
27-03-2013 14:32  DEBUG    ObsSim          Adding new target at RA: 76.066290  DEC: -33.752925
27-03-2013 14:32  DEBUG    ObsSim          Adding new target at RA: 73.774780  DEC: -33.752925
27-03-2013 14:32  DEBUG    ObsSim          Adding new target at RA: 71.483270  DEC: -33.752925
27-03-2013 14:32  DEBUG    ObsSim          Adding new target at RA: 69.191760  DEC: -33.752925
27-03-2013 14:32  DEBUG    ObsSim          Adding new target at RA: 66.900251  DEC: -33.752925
27-03-2013 14:32  DEBUG    ObsSim          Adding new target at RA: 64.608741  DEC: -33.752925
27-03-2013 14:32  DEBUG    ObsSim          Adding new target at RA: 62.317231  DEC: -33.752925
27-03-2013 14:32  DEBUG    ObsSim          Adding new target at RA: 60.025721  DEC: -33.752925
27-03-2013 14:32  DEBUG    ObsSim          Adding new target at RA: 57.734211  DEC: -33.752925
27-03-2013 14:32  DEBUG    ObsSim          Adding new target at RA: 55.442701  DEC: -33.752925
27-03-2013 14:32  DEBUG    ObsSim          Adding new target at RA: 56.567283  DEC: -32.102925
27-03-2013 14:32  DEBUG    ObsSim          Adding new target at RA: 58.816448  DEC: -32.102925
27-03-2013 14:32  DEBUG    ObsSim          Adding new target at RA: 61.065613  DEC: -32.102925
27-03-2013 14:32  DEBUG    ObsSim          Adding new target at RA: 63.314778  DEC: -32.102925
27-03-2013 14:32  DEBUG    ObsSim          Adding new target at RA: 65.563943  DEC: -32.102925
27-03-2013 14:32  DEBUG    ObsSim          Adding new target at RA: 67.813108  DEC: -32.102925
27-03-2013 14:32  DEBUG    ObsSim          Adding new target at RA: 70.062273  DEC: -32.102925
27-03-2013 14:32  DEBUG    ObsSim          Adding new target at RA: 72.311438  DEC: -32.102925
27-03-2013 14:32  DEBUG    ObsSim          Adding new target at RA: 74.560603  DEC: -32.102925
27-03-2013 14:32  DEBUG    ObsSim          Adding new target at RA: 76.809768  DEC: -32.102925
27-03-2013 14:32  DEBUG    ObsSim          Adding new target at RA: 75.704690  DEC: -30.452925
27-03-2013 14:32  DEBUG    ObsSim          Adding new target at RA: 73.494533  DEC: -30.452925
27-03-2013 14:32  DEBUG    ObsSim          Adding new target at RA: 71.284377  DEC: -30.452925
27-03-2013 14:32  DEBUG    ObsSim          Adding new target at RA: 69.074221  DEC: -30.452925
27-03-2013 14:32  DEBUG    ObsSim          Adding new target at RA: 66.864065  DEC: -30.452925
27-03-2013 14:32  DEBUG    ObsSim          Adding new target at RA: 64.653909  DEC: -30.452925
27-03-2013 14:32  DEBUG    ObsSim          Adding new target at RA: 62.443753  DEC: -30.452925
27-03-2013 14:32  DEBUG    ObsSim          Adding new target at RA: 60.233597  DEC: -30.452925
27-03-2013 14:32  DEBUG    ObsSim          Adding new target at RA: 58.023441  DEC: -30.452925
27-03-2013 14:32  DEBUG    ObsSim          Adding new target at RA: 55.813285  DEC: -30.452925
27-03-2013 14:32  DEBUG    ObsSim          Adding new target at RA: 56.900410  DEC: -28.802925
27-03-2013 14:32  DEBUG    ObsSim          Adding new target at RA: 59.074659  DEC: -28.802925
27-03-2013 14:32  DEBUG    ObsSim          Adding new target at RA: 61.248909  DEC: -28.802925
27-03-2013 14:32  DEBUG    ObsSim          Adding new target at RA: 63.423158  DEC: -28.802925
27-03-2013 14:32  DEBUG    ObsSim          Adding new target at RA: 65.597408  DEC: -28.802925
27-03-2013 14:32  DEBUG    ObsSim          Adding new target at RA: 67.771657  DEC: -28.802925
27-03-2013 14:32  DEBUG    ObsSim          Adding new target at RA: 69.945907  DEC: -28.802925
27-03-2013 14:32  DEBUG    ObsSim          Adding new target at RA: 72.120156  DEC: -28.802925
27-03-2013 14:32  DEBUG    ObsSim          Adding new target at RA: 74.294406  DEC: -28.802925
27-03-2013 14:32  DEBUG    ObsSim          Adding new target at RA: 76.468655  DEC: -28.802925
27-03-2013 14:32  DEBUG    ObsSim          Adding new target at RA: 78.642905  DEC: -28.802925
27-03-2013 14:32  INFO     ObsSim          Field area complete for filter tray tray_r
27-03-2013 14:32  INFO     ObsSim          Adding new tiles to the local database...
27-03-2013 14:32  INFO     ObsSim          ..database update complete.
...done.
Scheduling survey...
27-03-2013 14:32  INFO     ObsSim          >>>>>>>>  Night 2013-01-01 12:00:00  <<<<<<<<
27-03-2013 14:32  DEBUG    ObsSim           Horizon for Sunset & Sunrise: 0:00:00.0
27-03-2013 14:32  DEBUG    ObsSim           Horizon for Astronomical Twilight: -18:00:00.0
27-03-2013 14:32  DEBUG    ObsSim          TWILIGHT TIMES
27-03-2013 14:32  DEBUG    ObsSim           Sunset: 2013/1/1 23:48:39
27-03-2013 14:32  DEBUG    ObsSim           Dusk: 2013/1/2 01:26:07
27-03-2013 14:32  DEBUG    ObsSim           Dawn: 2013/1/2 08:08:25
27-03-2013 14:32  DEBUG    ObsSim           Sunrise: 2013/1/2 09:45:53
27-03-2013 14:32  DEBUG    ObsSim           Evening twilight time: 1:37:28.062825
27-03-2013 14:32  DEBUG    ObsSim           Morning twilight time: 1:37:27.813718
27-03-2013 14:32  DEBUG    ObsSim           >>> Total DARK time: 6:42:18.300921
27-03-2013 14:32  DEBUG    ObsSim          No time limit. Observation time from dusk till dawn.
27-03-2013 14:32  DEBUG    ObsSim          Fields VISIBILITY
27-03-2013 14:32  DEBUG    ObsSim          Altitude limit is 34.85 to constraint Airmass < 1.75
27-03-2013 14:32  DEBUG    ObsSim          Field Test: [54.0, 80.0, -40.0, -27.0]
27-03-2013 14:32  DEBUG    ObsSim          Field is visible at start of the night
27-03-2013 14:32  DEBUG    ObsSim          Field set during the observation time
27-03-2013 14:32  DEBUG    ObsSim          Field VISIBLE above airmass limit: 01:26 - 07:34 (6h 7m)
27-03-2013 14:32  DEBUG    ObsSim          Total observation time: 6:42:18.300921
27-03-2013 14:32  DEBUG    ObsSim          Remaining 345 targets of field Test
27-03-2013 14:32  DEBUG    ObsSim          Total visibility time: 6:07:54.777816
27-03-2013 14:32  INFO     ObsSim          Observation efficiency combining field visibility is 91.45%
27-03-2013 14:32  WARNING  ObsSim          Already SCHEDULED targets for current night 2013-01-01 12:00:00! IGNORING this night...
27-03-2013 14:32  INFO     ObsSim          >>>>>>>>  Night 2013-01-02 12:00:00  <<<<<<<<
27-03-2013 14:32  DEBUG    ObsSim           Horizon for Sunset & Sunrise: 0:00:00.0
27-03-2013 14:32  DEBUG    ObsSim           Horizon for Astronomical Twilight: -18:00:00.0
27-03-2013 14:32  DEBUG    ObsSim          TWILIGHT TIMES
27-03-2013 14:32  DEBUG    ObsSim           Sunset: 2013/1/2 23:48:51
27-03-2013 14:32  DEBUG    ObsSim           Dusk: 2013/1/3 01:26:10
27-03-2013 14:32  DEBUG    ObsSim           Dawn: 2013/1/3 08:09:18
27-03-2013 14:32  DEBUG    ObsSim           Sunrise: 2013/1/3 09:46:37
27-03-2013 14:32  DEBUG    ObsSim           Evening twilight time: 1:37:19.236873
27-03-2013 14:32  DEBUG    ObsSim           Morning twilight time: 1:37:18.972225
27-03-2013 14:32  DEBUG    ObsSim           >>> Total DARK time: 6:43:08.572548
27-03-2013 14:32  DEBUG    ObsSim          No time limit. Observation time from dusk till dawn.
27-03-2013 14:32  DEBUG    ObsSim          Fields VISIBILITY
27-03-2013 14:32  DEBUG    ObsSim          Altitude limit is 34.85 to constraint Airmass < 1.75
27-03-2013 14:32  DEBUG    ObsSim          Field Test: [54.0, 80.0, -40.0, -27.0]
27-03-2013 14:32  DEBUG    ObsSim          Field is visible at start of the night
27-03-2013 14:32  DEBUG    ObsSim          Field set during the observation time
27-03-2013 14:32  DEBUG    ObsSim          Field VISIBLE above airmass limit: 01:26 - 07:30 (6h 3m)
27-03-2013 14:32  DEBUG    ObsSim          Total observation time: 6:43:08.572548
27-03-2013 14:32  DEBUG    ObsSim          Remaining 345 targets of field Test
27-03-2013 14:32  DEBUG    ObsSim          Total visibility time: 6:03:56.020968
27-03-2013 14:32  INFO     ObsSim          Observation efficiency combining field visibility is 90.28%
27-03-2013 14:32  WARNING  ObsSim          Already SCHEDULED targets for current night 2013-01-02 12:00:00! IGNORING this night...
27-03-2013 14:32  INFO     ObsSim          Committing changes to db...
27-03-2013 14:32  INFO     ObsSim          Exporting targets file for simulation...
27-03-2013 14:32  WARNING  ObsSim          Summary not ready yet.
...done.
27-03-2013 14:32  INFO     ObsSim          Yaml to data bank process...
27-03-2013 14:32  DEBUG    ObsSim          Scanning yaml file...
27-03-2013 14:32  DEBUG    ObsSim          Updating Data Bank HDF5 file...
Number of tiles: 0
27-03-2013 14:32  INFO     ObsSim          Yaml successfully exported to databank.
Setting Tile ID to each galaxy
27-03-2013 14:32  INFO     ObsSim          Loading tile centers from databank...
27-03-2013 14:32  INFO     ObsSim          Loading galaxies from databank...
min input 55.6234 -37.9592
max input 76.491 -30.1108
tiles min input---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/IPython/utils/py3compat.pyc in execfile(fname, *where)
    176             else:
    177                 filename = fname
--> 178             __builtin__.execfile(filename, *where)

/Users/ben/Documents/DESpec/repo/despec/Survey_Strategy/Survey_Strategy_Serrano/ObsSim.py in <module>()
   1608 # =====================================================
   1609 if __name__ == '__main__':
-> 1610         main()
   1611 
   1612 

/Users/ben/Documents/DESpec/repo/despec/Survey_Strategy/Survey_Strategy_Serrano/ObsSim.py in main()
   1604         if Tile:
   1605                 print "Setting Tile ID to each galaxy" 
-> 1606                 pau_survey_strategy.set_tile_number()
   1607 
   1608 # =====================================================

/Users/ben/Documents/DESpec/repo/despec/Survey_Strategy/Survey_Strategy_Serrano/ObsSim.py in set_tile_number(self)
   1499                 print "min input", numpy.amin(gals_ra_in), numpy.amin(gals_dec_in)
   1500                 print "max input", numpy.amax(gals_ra_in), numpy.amax(gals_dec_in)
-> 1501                 print "tiles min input", numpy.amin(tile_centers_ra), numpy.amin(tile_centers_dec)
   1502                 print "tiles max input", numpy.amax(tile_centers_ra), numpy.amax(tile_centers_dec)
   1503 

/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/core/fromnumeric.pyc in amin(a, axis, out)
   1893     except AttributeError:
   1894         return _wrapit(a, 'min', axis, out)
-> 1895     return amin(axis, out)
   1896 
   1897 

ValueError: zero-size array to minimum.reduce without identity
... Survey_Strategy finished in 2.69 seconds
... running Fiber_Allocation
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/mpl_toolkits/__init__.py:2: UserWarning: Module readline was already imported from /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/IPython/utils/rlineimpl.pyc, but /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/readline is being added to sys.path
  __import__('pkg_resources').declare_namespace(__name__)

=============================
         fiber allocation         
=============================
... imported data
*** Incoming QA checksum PASSED ***
... combined selection flags
... assigned fiber ids
*** checksum passed for Fiber Allocation***
... gal counts
... ... n total 179
... ... n survey + targ sel 162
... ... n both  159
... output to databank

... Fiber_Allocation finished in 1.0 seconds
[0] Survey_Strategy              2.69s    73.0%
[1] Fiber_Allocation             1.00s    27.0%
Total                        3.69s    100.0%
hpx-public-dock-205-dhcp:Wrapper ben$ 

History

#1 Updated by Santiago Serrano over 7 years ago

Hi Ben,

There is no problem to recreate the database every time and avoids some problems as you had now. I already changed that in my local copy but I forgot to commit. I just updated the svn code and now should do a recreate every time the code runs.

As here the database is just a tool, not for storing data, there is no problem for erasing it at every run.

Sorry I did not commit before, it would have saved you some time.

Santi

#2 Updated by Brian Nord over 7 years ago

  • Due date set to 04/19/2013

Do we want to recreate the SS database every time? I think Ben says "no, we don't," because it takes some time to do that.

Under what circumstances do we want to recreate the SS-specific database? Should we set a button in the param files that we press manually, or should it be automatic?

#3 Updated by Brian Nord over 7 years ago

Hi Brian,

As I said in the previous update of this ticket, you win absolutely nothing by saving the database. The database here is a useful tool that does not have keep data between runs. In the project where we did this module (PAU) the database holds and tracks all the data, and we do not recreate it every time but here it does not, it is the hdf5 file.

I'll try to explain why. So first, the planner adds the empty targets to the database to cover the fields specified (around 1% of the total processing time). Then the scheduler searches over those empty targets (here is useful the database) and fills in with the missing information (time, airmass...). When the scheduler is finished, data is exported to the hdf5 file.

If we do not erase the database and someone reruns the SS code with data from a previous run in the database, confusion between the previous remaining targets (if any) and the ones created will happen. If you try to schedule targets over the same night, it will give you an error because you have selected an observing night that was already scheduled before. For the concept of SPOKES, it does not make sense to keep the database, and indeed is very advisable to start with an empty one.

If you would like to speed up the process and you are not changing SS configuration in each run, a simple solution would be to bypass the SS execution, as data stored in the data bank would be exactly the same from a previous run.

The code can definitely be optimized but not by keeping the database. Believe me! : )

I hope I am not missing anything. Let me know if I do.

Cheers,
Santi

#4 Updated by Brian Nord over 7 years ago

Hi Santi,

Great info.

I think we'd want to bypass the SS execution if it will truly save time. When we find that out, we'll cross that bridge.

#5 Updated by Brian Nord over 7 years ago

  • Tracker changed from Bug to Support
  • Status changed from New to Resolved
  • Assignee changed from Santiago Serrano to Brian Nord

#6 Updated by Brian Nord over 7 years ago

  • Status changed from Resolved to Closed


Also available in: Atom PDF