NDOS CommissionMtng 20120924¶
Attending: Leon, Jan, Kurt, Peter, Denis, Xuebing, Jon, Mat
No meeting since Aug 27
8/27 - Detector found noisy following IOC work - disabling cooling drops the noise. Noisy DCS and DSO extra mask correlate.
-Out of sync detector observed for the first time in a while
-Noise level decreases slowly over time (6ish hours)
-8/28-end of week 7 silicone, 2 new parylene, and 25 old parylene installed on 2-2
-8/30 Found the the cooling history was not being logged for DCM 3-2 and 3-3 after trying to view histories on CSS. Notified Gennadiy who resolved the issue
9/4 Detector in a noisy state following reboot .Dropping HV on 3-1 to 300V at source eliminated noise, returned at ~370V. Warming cold channels dropped the noise rate. Dropping the voltage on the HVAdjust also killed the noise.
9/4 Spooky ghosting in the EVD in Hardware view observed (16808, Peter Shanahan (shanahan), Tue, 09/04/2012 18:03:30). Mark has been emailed.
9/5 Stephen ran some tests and try to determine while some TECCs won't start cooling without the start cooling button being pressed multiple times. Pushed out an updated DCM IOC that has better logging and changes that way communication is done with the FEBs. Through 7 cooling cycles I couldn’t reproduce the problem.
9/6 DCS alarm now generates summary emails. DCS email on first, and a hold off. Summary.
9/6 DCM out of sync are run start reported
9/6 Cooling enable on DCM 2-2 APD install from previous week. (7 silicone, 2 new parylene, and 25 old parylene
9/7- Zukai takes over daily DSO scans
9/14- Zukai and León notice that many FEB links fail to produce data in a DSO scan started after data taking without a reboot. This also seems to be accompanied by incorrect data in the FEB files. Ron has reproduced this effect and has a fix. (New DCM progutils on the 8/31)
9/18- High detector rate seem to be associated with DCM reboot. A glitch in the 24V current at reboot was observed indicating momentary cooling during a reboot. Stephen varied that cooling enables occurred before loading of setpt values. This was change in an IOC update and the 24V glitch is gone. Reboot now doesn't see to cause the rate spike (at least in limited testing)
-Production DB machine disk failure and issues
Boot disk failing
-swapped last week
-another failure yesterday
-Migrate production machine on Thursday to a shared machine
-Migrate DCS to NDOS-db01
Leave ECL and HWDB (special Kerberos principles needed) to migrated, working on a plan.
Peter asks about production DB risk with shared machine. (Long term is to use web app to protect the database)
DAQ and DCS will remain insulated. Going to an alarm (more intelligent shipping identical vaules) plus 5 minute update rate.
1. Detector Update
Question on solved reboot issue: Leon can try to reproduce it in the lab. We don't know why breakdown persists.
2. DataCheck Update
Labor day up date - 14088 - subrun 2 was huge. DDS had been restarted myteriously.
14094 - Subrun 3.6 GB again
Max file size is 512 MB
Run control server - dies? Denis sees this and has the "touch". We have a dead RCServer now. Jon will investigating. Is there a file limit for writing to the blue arc
Sept 15, 18 - Hot rans observed 1000 masked that recover
14116 - Thursday died at noon. Recorded empty events for 2.5 subruns. 16946, Leon Mualem (mualem), Thu, 09/20/2012 16:12:57
Run history has the info ECL (probably a DB issue)
4. Installation Plans for the week
Xuebing oil overflow system was test fitted last week.
APD installation not for a few week. I received a box of APDs.
5. DAQ Update and Plans for the week
Ron - ProgUtilites update
Kurt - will change the write setting on the DB today.
6. DCS Update and Plans for the week
Many new monitors in the control room for DCS and Far get monitoring.
Athens- Ask for workshop at collaboration to discuss control alarms (logic). Quantities and frequencies.
Denis ask about monitoring the actually HV with some sense wires. Leon - says know. The current evidence is that the values are reliably set by monitoring the HV current.