Project

General

Profile

Yearly update of the Scientist Survey application and start-up data.

This is a description of the necessary yearly updates for the Scientist Survey application (cvdb). It assumes you are starting from a VM which has a configured application, which needs to be cleaned of the previous year's data and updated for the new year's data.

Summary.

Each year, two new data files must be provided:
  1. information regarding the scientists who are to participate in the survey
  2. information regarding the effort reporting categories (called by the application LineItems)

Each year, the years of the survey must be updated.

Each year, it may be necessary to update text presented on some of the web pages, to provide new instructions or new contact information.

Each year, the previous year's data must be removed, and the database reset to its empty state.

Finally, the new data must be loaded into the database.

The sections below describe the steps of the sequence.

Detailed instructions.

All of these steps must be done while logged into the survey.fnal.gov machine, as the user cvdb. All these steps should be executed from the directory /home/cvdb/cvdb. This is the top-level directory of the cvdb application. In these instructions, all filepaths that are not specified as absolute filepaths are relative to this directory.

Get the new data files.

See also Required file formats for Scientist and Line Item data.

Verify that the files that have been provided are in the required format, including the number of columns and their order. If either file is incorrect, the scripts that load the data into the database will not run correctly.

The new CSV file containing the scientist information should be put moved to /home/cvdb/cvdb/data/scientst_list.csv. The new CSV file containing the reporting category (line item) data should be moved to /home/cvdb/cvdb/data/line_items.csv.

Update the years of the survey.

To do this, edit the file /home/cvdb/cvdb/app/models/time_allocation.rb. Modify the two functions shown below to contain the correct dates:

  # Modify the value returned as appropriate for each survey.
  def self.first_valid_year
    2012
  end

  # Modify the value returned as appropriate for each survey.
  def self.last_valid_year
    2017
  end

Do not make any other changes to this file.

Update any text, if necessary.

Updating the "welcome" page.

If it is necessary to update the static text on the "welcome" page, you must edit the file /home/cvdb/cvdb/app/views/welcome/index.html.erb. This is an ERB template file. There are instructions at the top of this file for how to modify some of the programmatically generated parts of the text. Most of the file is normal HTML.

Updating the "estimated efforts" page.

If it is necessary to update the static text on the "estimated efforts" page, you must edit the file /home/cvdb/cvdb/app/views/create_effort.html.erb. This is also an ERB template file. The instructions that might need modification appear near the top of the file. Note that the years that appear on the table are automatically generated by the application (using the code modified above), and do not need to be changed here.

Updating the "regrets" error message page.

If there is a new person to whom errors or problems should be reported, update the file /home/cvdb/cvdb/app/views/welcome/regrets.html. This file is an HTML page fragment. Make whatever corrections are necessary.

Reload the application software.

To reload the application software, modify the timestamp on the file /home/cvdb/cvdb/tmp/restart.txt:

touch tmp/restart.txt

Reloading the application software does not require restarting the Apache web server. Reloading the application software is necessary after any change to application source code (and of the *.rb files) or after any change to any of the application static web pages (*.html files) or any of the application web page templates (*.erb files).

Reloading the application is not necessary after changes to data in the database.

Clear the application logs.

To avoid needlessly large logfiles, the application logs should be cleared periodically, at least at the beginning of each new survey deployment. The command to clear the logs is:

rake log:clear

You may see the following in the output of this command:

rake/rdoctask is deprecated.  Use rdoc/task instead (in RDoc 2.4.2+)

This can be safely ignored.

Reset the database.

Note that this command can lose data. It resets the database to its empty state. Thus is any scientists have entered time allocation data, their data will be lost, and they will have to enter their data again.

The command to reset the database is:

RAILS_ENV=production rake db:reset

The next-to-last line of the output from this command should be:

-- assume_migrated_upto_version(20110204195024)

You may see the following in the output of this command:

rake/rdoctask is deprecated.  Use rdoc/task instead (in RDoc 2.4.2+)

This can be safely ignored.

Load the new year's data.

The command to load the new year's data is:

./script/load_startup_stuff.sh

Dealing with changes before scientists have begun to enter data.

If there are errors in the scientist CSV file or in the line items CSV file, correct the file, reset the database, and reload new data.

Dealing with changes after scientists have begun to enter data.

After scientists have begun to enter the data, do not reset the database without receiving permission from whoever is currently in charge of the Scientist Survey effort.

If it is necessary to add one or more new scientists to the list, it is safe to modify the scientist CSV file (adding the new scientists, in the correct format) and to re-run the script/load_startup_stuff.sh script.
If it is necessary to add one or more new line items to the list, it is safe to modify the line items CSV file (adding the new line items, in the correct format) and to re-run the script/load_startup_stuff.sh script.

How to change an incorrect Service Account username ("principal").

For historical reasons, the cvdb application uses the name "principal" to represent the Service Account username of a person. If the scientists CSV file used to pre-load the database contains an incorrect principal, and if it is no longer possible to correct the file, reset and reload the database, then follow the instructions at Changing an incorrect principal.