Project

General

Profile

Catalog Module

The python module catalog.py can be used to interact with catalog files, and to convert data between formats. To use the module, open the python interpreter and type

>>> import catalog

getdata(catalog_file)

The most commonly used method of the catalog module is the getdata method. It creates a dictionary of data from an IQ catalog. The getdata method takes the name of the IQ catalog file as the only argument, and returns a python dictionary. The keys of the python dictionary represent columns, and each corresponds to an array of values, each element representing a row of data. For example, the first element in the array under the "expid" key will be in the same row (from the same exposure) as the first element in the array under the "filter" key.

An example of usage is below:

>>> import catalog
>>> data_dictionary = catalog.getdata('iq_apr19.dat')
# of columns:  118
>>> data_dictionary['expid']
array([ 192742.,  192743.,  192744., ...,  199031.,  199032.,  199033.])
>>> data_dictionary['filter']
array(['z', 'z', 'z', ..., 'r', 'r', 'r'], 
      dtype='|S1')

dict2db(dictionary, filename)

The dict2db method converts a catalog dictionary (a python dictionary where the keys are catalog headers) to an SQLite database. It takes the dictionary to be converted and the name of the database as arguements. For example:

>>> data_dictionary = catalog.getdata('iq_apr19.dat')
# of columns:  118
>>> catalog.dict2db(data_dictionary, 'data_database.db')

The above first fills a dictionary with data using getdata (see above), then puts that data into an SQLite database using dict2db.

write_catalog(cat, filename, header = None)

The write_catalog method converts a catalog dictionary to an SQLite database. It takes two mandatory and one optional argument: The dictionary to be converted, the name of the file being written to, and optionally the header of the columns to be written in a list (it will default to all columns). For example:

>>> catalog.write_catalog(data_dictionary, 'test_file.dat')

or, with headers specified

>>> catalog.write_catalog(data_dictionary, 'test_file.dat', header = ['expid', 'filter'])

for just the columns expid and filter recorded.

convert_date(d)

The convert_date method converts from a string representing a date (such as date-obs) to a datetime object. It takes an array of time stamp strings as an arguement and returns an array of converted datetimes.

For example:

>>> data_dictionary['date-obs']
array(['2013-04-01T00:23:59.939876', '2013-04-01T00:26:10.934967',
       '2013-04-01T00:28:19.940334', ..., '2013-04-19T10:05:26.799933',
       '2013-04-19T10:07:00.240884', '2013-04-19T10:08:32.243648'], 
      dtype='|S26')
>>> dates = catalog.convert_date(data_dictionary['date-obs'])
>>> dates
array([datetime.datetime(2013, 4, 1, 0, 23, 59, 939876),
       datetime.datetime(2013, 4, 1, 0, 26, 10, 934967),
       datetime.datetime(2013, 4, 1, 0, 28, 19, 940334), ...,
       datetime.datetime(2013, 4, 19, 10, 5, 26, 799933),
       datetime.datetime(2013, 4, 19, 10, 7, 0, 240884),
       datetime.datetime(2013, 4, 19, 10, 8, 32, 243648)], dtype=object)

read_csv(csvfile, return_header = false, catalog_file = None)

The read_csv method reads in a csv file and returns a python dictionary. The keys of the python dictionary represent columns, and each corresponds to an array of values, each element representing a row of data. The program can also optionally return the list of columns (the header) if return_header is set to true, and will write and iq_catalog of the data to the location specified by catalog_file.

For example:

>>> d, h = catalog.read_csv('test.txt', return_header = true, catalog_file = 'test2.dat')

In the above example, test.txt is a csv file containing the data wanted. The variable d now contains a python dictionary of the data, h contains a list of the elements, and there is a file test2.dat that has the same data now stored as an IQ catalog.