How to monitor and troubleshoot offline pubs production

Log in to uboonepro and initialize pubs environment

$ ssh -l uboonepro uboonegpvm01
$ cd /uboone/app/users/uboonepro/pubs_devel
$ source config/

Running the monitoring GUI

$ setup pyqtgraph
$ cd $PUB_TOP_DIR/dstream_prod/prod_gui
$ python

Browse the pubs database using psql

Run psql.

$ psql -h ifdbprod2 -d microboone_prod -p 5437 -U uboone_admin
Password for user uboone_admin: 
psql (9.3.6, server 9.3.5)
Type "help" for help.

Find jobs with error status.

microboone_prod=> select * from prod_swizzle_filter_v2 where status>1000;   

This query will return failed (run,subrun) pairs, as well as failed jobsub job ids.