Bug #24719

HWR System Intermittent Problems

Added by John Dusatko about 2 months ago. Updated about 1 month ago.

Start date:
Due date:
% Done:


Estimated time:


On Aug 5, 2020, at 9:57 AM, Brian E Chase <> wrote:

Philip, John, Ed,

A quick summary of observations:

Philip and John worked on the system Monday night without reported issues.

Tuesday night strange effects were noted and by the end of the night it seemed that none of the systems were working correctly
-the key issue seems to be that response became slow and some settings were not making it through. By the end of the night at least we could not get FF RF on.
-readbacks froze sometime but by and large waveforms continued okay. Turning off waveforms had no effect. The timing diagram looked normal to me but it has been awhile since I looked at this and might have missed something.

Investigations of LO and other signals ruled out Up or Down converter issues.

The 1320 MHz clock signal going to the controllers read -3.5 to -4 dBm but probably 1 dB of cable loss. The back panel says 2 dBm and I remember a target at the module of 0 dBm.
This is the only measurable problem that we found and should be investigated but I find it unlikely that this is our root issue.

Things that can be done:

In the lab the sensitivity of clock power could be investigated and we could try to repeat the problem there.

John could talk with Ed about the LO and clock chassis( get all drawings and other documents located and a quick walk through. Identify measurement points.

Yes, before we disconnect anything that goes to the warm front end we will need to get permission.

Going back out to repeat some things observed last night might be helpful. We don’t need SSAs on as we can look at the drive monitors. They are before the RF switches so we don’t need permits. (I used to think this was not a good design feature but it does come in handy sometimes)

This feels like a timing race condition problem to me but I can’t identify the cause. We could try to turn off three controllers and see if we can get one to work.



#1 Updated by John Dusatko about 2 months ago

JED: Some additional information / Clock power level:
Philip and I tested how the controller chassis responds to different power levels of the 1.3GHz system clock.
Test: Using RF Synth, change the power level. When the level is changed, the chassis is power cycled. The RF power was varied between -10dBm....+10dBm and the chassis was observed to work (i.e. run LabView OK, respond to controls, update waveform displays with sensible data).

However: IF the RF power level is changed when the chassis is running (e.g. going from 0dBm -> -4dBm), the system will freeze. So this is the corner case where things can fail. One speculation is whether or not the system clock at PIP-II IT is changing its power level intermittently?

#2 Updated by Brian Chase about 2 months ago

I expect that the RF generator in the lab may drop out when levels are changed. If you hear any clicks, that would be a good bet. I think the clock level is not the source of the problems that were seen Tuesday night.

#3 Updated by John Dusatko about 2 months ago

Problem has bifurcated into two tracks:

1) There is something wrong with the cavity selection control: e.g. when Cavity 6 is selected its feed forward magnitude setting is that of #5 -- so the even cavity numbers are getting the settings for their odd numbered partner.
SW needs to be fixed, not sure if its a LabView or ARM code issue

2) Something went wrong with the FW image & ARM object code updates to the controller chassis: #3 was done manually by Philip and worked (we could control Cav5, Cav6 has the selection problem) / as a test, Controller #4 was updated manually, now Cav7 works WRT controls and RF drive is appearing at the monitor point on the upconverter chassis, but there seems to be some issue with possibly the cavity itself (see elog entry) / Cav8 has the selection problem.
Suggest updating controllers #1 & #2 manually to fix.

#4 Updated by Brian Chase about 1 month ago

I have added watchers. Watchers receive email notices.

#5 Updated by Brian Chase about 1 month ago

Was there a further investigation of the clock power level? Are we concluding that the problems seen are not related to the clocks at all? Do we have any understanding of the freezing of the Labview pages?

#6 Updated by Philip Varghese about 1 month ago

  • Status changed from New to Resolved

The clock input was tested in the lab between the range of -10 to + 10 dBm.
The system worked fine at both limits of the clock input specification for the board.
The clock inputs to the HWR controllers at CMTF were measured at no lower than -4 dBm.
They should have been+1 to 2 dBm from the clock distribution module - that is another issue to be looked at.
The clock level is not the reason for Labview locking up.

Also available in: Atom PDF