Quirks, Features, Debug Hints - Operational issues you should be aware of. Many Solved.¶
Slow delivery of exposures to DTS server (09/20/2015) It is taking long for exposures to be delivered to the DTS server(s). This is overloading the IB nodes and it has hapenned that none was available to take new exposures as input. This led to loss of exposures, as they were not digitized and built.
Observer2 versus SISPI (01/25/2015) The computer Observer2 became more and more sluggish one evening, and eventually no longer responded. SISPI was live and images were being accumulated but DQ could not be carried out according to procedure. If Observer2 is rebooted while SISPI is taking data, SISPI will hang, because it copies images over to Observer2. Don't want to hang SISPI. Instead, stop SISPI first, then reboot Observer2, then restart SISPI. This will take about 10 minutes so you will have to decide whether it is worth it. On the other hand, should Observer2 completely stop, SISPI would hang trying to copy an image to it. At that point you have no choice but to restart and to reboot.
Run Manager Cell Phone (08/15/2014) There is an international cell phone passed along by the Run Managers. It can call the US. Instructions are in the box that it comes in. It is to be used in case there is a problem whereby normal communications have failed. In a long sitewide power outage, for instance, the UPS backup for the computers may fail and leave you without internet or (VOIP) phones. To get the cell phone to connect to the phone network, go outside and over by the wall just down the hill from RASICAM. There you will have line of sight to Vicuna. Then turn the phone on (or off and on). If it says "emergency calls only" go a bit further dopwn the hill and turn it off and on again.
CTIO vr filter installed (03/15/2014) CTIO A vr filter is available.
u-band Flat Field LEDs Replaced (03/15/2014) DJJ New LEDs are installed. They are much brighter.
Flat Field LEDs Tweaked (01/30/14) DJJ Powered down the bluest LED in the Dome Flat illumination system (365nm) from 100% to 80%, and changed the NOAO community and DES prenight and postnight calibration scripts to now take 150-second u-filter domeflats. The g-band was very minorly affected, and so the exposure time was left as before. The new 150-seconds u-filter domeflats yield about 11000 ADU/pixel above bias in S4.
PANG - Guider (12/03/13) LO If exposures have noise that looks like this:
on backplane 1 CCDs, along with an interlock GCS failure error, this is most likely a problem with PANG. Stop and start the PANG role and re-configure. The Vsub of PANG will be off after restarting so you will need to turn it on and do flushes. Similar to item 10 below. At this time Crate 3 fan 4 is also bad. Crate 4 fan 4 is masked, though this fan is actually ok.
C5 Dry Gas Purge Not Dry (10/18/13) - HTD
We found that the desiccator for the dry gas purge of C5 had been switched-off since around 01 Aug 2013. That created the possibility for dew or even frost on the outer surface of the camera Dewar depending on the dewpoint of the "dry" gas and the internal temperature of the optical barrel. We need to avoid this. Please check Telemetry>Facility_Utilities>Dry Gas Dewpoint and see that it is at about -60 deg. C. Well, experience then showed we can get marginal humidity as measured by the filter changer (Oct. 26th) even if dry gas purge has a low dewpoint. Watch FCM_telemetry>humidity. And We'll try to tweak up the flow at the barrel. For shifters: are we following the wind screen rules?
look at the telemetry and check if the VSUB is really off, or if it was simply a glitch on the telemetry readout.
If it was a glitch, report the glitch to Marco to see what happened. After enough glitches Marco and the electronics group will try to fix this.
If the telemetry shows VSUB at zero, the observer should look at the corresponding CCDs to see if they are fully depleted. This will be evident by looking at cosmic rays in a dark exposure. If the CCDs are not fully repleted I would recommend doing VSUBOFF and VSUBON and check the telemetry. Then check the telemetry again as in (1). If it is still zero, then there is a real problem with the electronics and should be reported immediately to TELOPs.
Look out for Missing Focus Chips (8/30/13) Last night we started up and because of a software bug the focus chips weren't read out. This was spotted in the Comfort Display. The hexapod corrections were zero. There were no donuts reported in the telemetry. There were expert-level SISPI alarms. The correction is to restart SISPI.
Resetting the Filter Changer in a pinch (8/27/13) - HTD The filter changer can be remotely power-cycled from the Power Distribution Control Chassis, which is in the hexapod rack in the Coude Room. There is a well-labeled switch. The shutter should be closed before this is done. When the filter changer is power-cycled, the filters remain in the same spot (see elog 8806). In August 2013 the FEE crates (1-3) needed to be reset after the FCM was power-cycled, for some reason related to the origin of the problem (see elog 106527).
Imager Vacuum Maintained by Ion Pump (4/8/13) The gate valve on the imager side of the turbo pump roughing pump combination is closed, leaving the imager vacuum on only the ion pump.
SISPI Shutdown Problem (3/12/13) SISPI was unable to be shut down through the GUI (from the Master Console session, neither the button nor manually typing the shutdown command into the System Controls worked; the page had been refreshed and that didn't fix it either). The shutdownSISPI command also failed with "unable to reach Name Server" errors (the instance name was copy/pasted from the Architect's terminal, so it was not simply a typo in the instance name, and the command was retried). Since the safe shutdown methods failed, and the vnc window didn't like passing a ctrl-C to the terminal, and vsub was already off and the block filter was in, SISPI was shut down by sending a SIGINT to the architect's main process. It was then restarted successfully (the first restart had a Labview error, but the second time worked fine). (Editor's Note: This might be on the SISPI News page instead of this one.)
Focal Plane Temperature Excursion (2/15/13) The focal plane temperature had a 1/2 deg C "glitch" on evening of 02/13/13. See Elog 54639. Here's a plot with the temperature on the RTDs over 2 days.
Kentools Update (02/03/13) OLD NEWS From Steve: I have rewritten the critical part of the "center" command in C, and it now can measure these big offsets successfully and much faster than before. If you would like, I will install this on observer2 (and hopefully leave everything else working). Let him know if there are problems.
Dry Gas Purge (12/12/12) OLD NEWS The LN2 tank is now being tapped faster to provide a dry gas purge of the electronics. This is in addition to purge of the C5 lens and high pressure dry gas for the filter changer.
Readout Crate Frequent Fan Error Alarms Masked (12/12/12) OLD NEWS (New) Crate 3 Fan Number 4, (Old) Crate 4 Fan Number 4 The fans are turning but the fan monitors are not reliable.
Ion Pump Controller Turned On (12/11/12) OLD NEWS The ion pump on the camera was enabled. Tests are underway as to whether this puts any additional RO noise on the CCDs. Early indications is that there's insignificant addtional noise. Late ones, too.
DECal Restart Procedure (11/26/2012) DECal stopped after either a network problem or receipt of a bad command. Here is the procedure for starting it back up. Restart procedure server emergency restart procedure.docx
CCD N30: poor CTI, particularly bad behavior. Maybe the problem is connections?
Two of the Focus Chips: one low amplifier on each
Several CCDs: non-linearity at very low light levels. We have an idea tfor a fix that we need to test at FNAL 1st.
Test RO Noise with Ion Pump Controller in new Location
Disable Some RO Crate Fan Monitors
Outstanding question (12/04/12): can we clock the serial registers during an exposure?
Dry gas Purge of Monsoon Crates (12/04/2012) OLD NEWS The electronics crates are cooled by NESLab Chiller with coolant temperature set to 15C. On previous night we got to a dewpoint of 12C. This difference is smaller than desired. We temporarily increased the coolant temperature to 18C. the plan is to implement a dry gas purge using LN2 boil off ASAP and then to set the coolant temperature back to nominal. We use an administrative limit of 6 deg C between the NESLab chiller supply temperature and the dewpoint. That is - we close up when the dewpoint hits 12 C. Actually, TELOPs set the NESLab back to 15 C in the afternoon, so that's where we are at present.
Guider Noise (Sept. 13, 2012) OLD NEWS Clocking the guide CCDs during 2kx4k RO caused noise in the images. This was resolved by changing the SISPI sequence so the the guide CCDs didn't clock while the image CCDs were reading out.