Project

General

Profile

Bug #24881

PIP2IT System crashes

Added by Brian Chase about 2 months ago.

Status:
New
Priority:
High
Start date:
08/28/2020
Due date:
% Done:

0%

Estimated time:
Duration:

Description

We have several unstable systems. A stable system can run for months or years without a problem, we are rebooting systems multiple times per shift. This is making operations impossible.
We all need to work on this together.

See note from Lionel below.

Thanks, Brian.

You might want to sit in a comfortable chair before starting to read what’s below… 😉

As for the bunchers, I did a little bit of digging this morning and, while I cannot tell exactly what is or has been going on, I think the phase scan program is not the only culprit (and Bill modified the application, so, at least one thing should not happen again… but needs to be tested to be more confident that it is resolved indeed).
As you are aware, for a reason that may or may not be related to having multiple instances of the Labview VIs running on 3 to 4 computers at the same time (even if they are not all ‘talking’ to the same LLRF system), we have had to reboot the RFQ & B1 LLRF front-end quite a few times (Philip is getting good at it!) and even the B2-B3 system yesterday, for which I do not recall having problems with in the past.
Unfortunately, for another reason that I do not understand, the reboots are not always successful and need to be repeated. You probably know (better than me) that, in theory, the most recent values of the parameters are reloaded. But, it may not always be completely true, in particular when the reboot is not successful. I did see in D54 that not always the last ‘set’ value to a device is the one that ends up restored. And maybe, this is what we experience.
I also found out (or at least, realized again) that some of the Sequencer aggregates set certain parameters every time they are run. For instance, the RFQ aggregate ends with setting up some beam compensation devices (if it is a recent addition though, it’s moot), or the proportional and integral gains for B2 are also set to some value. I believe the idea was to make sure that key parameters do not get changed by mistake (or left over from a study). Obviously, if now you tune the cavity a certain way changing the gains, they will be reset back to whatever they are in the aggregate. (but you said you did not tune B2, at least, not last Friday). Note that the setting of the gains in the Sequencer aggregate seems to be happening only in the “B2 turn on” aggregate; nothing is set in the “B1 turn on” aggregate, at least for the pulsed version.

Again, I have to admit that I cannot make complete sense of the various clues. But I have seen enough ‘anomalies’ to believe that “settings can change without our knowledge”. There is one more thing that I think may help get closer to understanding what’s going on sometimes: the interaction of the RFQ & B1 Labview GUI and ACNET. For this older system, we have to press a button on the application for it to synchronize with ACNET. What happened if values are tweaked through the Labview application, which eventually freezes and requires a reboot of the LLRF front-end? Are the ‘tweaked’ values restored if the ACNET synchronization button was never clicked on? That’s one thing that maybe Philip can tell me/us. That might also explain why changes can be made without being recorded by D54.

Anyway. Since I spent quite a bit of time this morning looking into the mystery of those settings seemingly changing without anyone knowledge, I thought I would share my (non)findings.

I’ll be on shift tomorrow afternoon. You are always welcome to spend some time making the LLRF ‘better’ (whatever that means). But I am thinking that if we cannot control 2-3 bunching cavities, how are going to deal with 16 ?!

Lionel.

From: Brian E Chase <>
Sent: Wednesday, August 26, 2020 7:09 PM
To: Lionel R Prost <>
Cc: Brian E Chase <>; Alexander V Shemyakin <>; Darren J Crawford <>; Nathan Eddy <>; Elvin R Harms Jr <>
Subject: Re: Beam loading compensation status

Lionel,

Nathan has identified the source of the phase signal as the digitizer cards and is working on a fix.

As far as beam loading compensation goes there are really four parameters; start time, stop time, amplitude and phase of the correction.

If you can fix a beam start time then the BLC start time does not have to move.

If you determine the longest beam time then the BLC end time can be set to this. If a shorter beam time is in play then there will be a short period of field error after the beam is gone. While it may not look great to the causal observer, the beam will not care because, well, it is gone and can never see this error.

If you change beam current a lot, the BLC amplitude and phase can be changed. There is also an automated learning system that Johnathan put in but I have not tried it recently.

You are correct about B2 and B3. I want to replace this software with what we are working on now to provide pulsed operation and BLC. I don’t remember if I even tuned up B2 because we were looking at BPMs before B2.

I am still finding the bunchers set up in crazy land. I expect it is due to the phase scan software not reseting the phase at the end of the scans.

That’s my story and I’m sticking to it. :-)

Brian

On Aug 26, 2020, at 11:46 AM, Lionel R Prost <> wrote:

Brian:

Last Friday, for investigating the 25-kHz ‘noise’ on the BPM phases, if I understood correctly, you turned on and tuned beam compensation for both the RFQ and B1. This is also my understanding that this tuning is only valid for the beam line parameters that were set that afternoon, namely, all the timing information (triggers that determined the timing of the beam pulse with respect to the RF pulse, basically), pulse length and beam current. This kind of tuning would have to be repeated each time one of those parameters would be changed, right ? However, I would naively think that depending on the magnitude of the changes being made to one or several of those parameters, retuning of the beam compensation would take more or less time.

It is also my understanding that you did not work on beam compensation for B2. Actually, I am not even sure that this is currently possible, or is it? I seem to recall that the beam compensation algorithm was never implemented on the ‘new’ RF system. Is that correct ? If so, is it due to hardware or software, or both ?

Note that I am not complaining. 😉 Just trying to understand what is the current status for beam operation and use our tools appropriately.

Thank you,
Lionel.



Also available in: Atom PDF