Project

General

Profile

Feature #2207

Soft reset for LE Linac

Added by Vitali Tupikov about 8 years ago. Updated almost 8 years ago.

Status:
Feedback
Priority:
Normal
Start date:
11/23/2011
Due date:
% Done:

60%

Estimated time:
Spent time:
Duration:

Description

will require MAXII, FPGA update. (this topic was removed from HINS6)

Linac_SoftReboot.docx (300 KB) Linac_SoftReboot.docx Testing details for soft reboot feature Philip Varghese, 12/21/2011 04:10 PM
slot0.jpg (177 KB) slot0.jpg Vitali Tupikov, 12/22/2011 08:41 AM
Lelpp_MFC_Bootup_Thread.docx (248 KB) Lelpp_MFC_Bootup_Thread.docx Barry Barnes, 01/04/2012 02:08 PM

History

#1 Updated by Philip Varghese about 8 years ago

  • Status changed from New to Assigned
  • Assignee changed from Philip Varghese to Barry Barnes
  • % Done changed from 0 to 50

The MaxII code changes to allow soft reboot were made to the last changed version (dated Jul 2009)
of the LEL MaxII code acrhive stored at Y:\Projects\LLRF\Components\Firmware\MFC\linac\MAXII\prod\Vxi_AHDL.qar
The modified code can be found at Y:\Projects\LLRF\Components\Firmware\MFC\linac\MAXII\dev\MaxII_LEL_WSoftRbt_Dec12_2011.qar
Please try this on the Linac development station and let me know if there are any issues.
Barry could test it on a MFC Rev2 board in the lab first to verify that the feature works before
moving to the development station.

#2 Updated by Barry Barnes about 8 years ago

  • Status changed from Assigned to Feedback
  • % Done changed from 50 to 60

MFC #10, v2, has been programmed with the new MaxII code. Also the CycloneII and the SHARC have been programmed to Vitali's specifications.

#3 Updated by Brian Chase almost 8 years ago

So it looks like progress has been made, but not testing? Vitali and Philip need to follow up here.

#4 Updated by Vitali Tupikov almost 8 years ago

Took MFC #10, with MAXII updated for soft-reset, from Barry and verified it on LEL development station. First I verified that the board is working after powering the crate up: it it is, though waveforms looked very distorted (there is a sticker on the
board saying so). Regardless of DAQ quality the board behaves fine.

Applied Ctrl-X -->
red LED blinked and terminal asked me to press anything to prevent rebooting. At this point I noticed that TRIG2 is not present already (the line stayed low). Since I didn't press any key the boot process begun and went smoothly all the way down to the prompt on the backdoor terminal. The TRIG2 stayed low so no access from slot0 had been initiated (Access LED was off), ---> no readback or waveform updates occurred.

Applied Ctrl-X one more time with the same "success".

Hard reboot through vxi power cycling was OK. TRIG2 started working in few seconds after powering up.

Taking board back to the lab.

#5 Updated by Philip Varghese almost 8 years ago

  • Assignee changed from Barry Barnes to Vitali Tupikov

Following Vitali's test on the development station it appeared that the soft reboot did not work the two times that it was tried. Barry and myself tried to do more extensive testing in the test crate in his area. The linac startup script was used and the 15 Hz trigger was applied. The Trig2 line on the backplane was monitored on a scope. Within 2 to 3 seconds pulses at a 15 Hz rate appeared on Trig2. After a couple of minutes the sloto access light started pulsing at the same rate. The debug terminal was spitting out a lot of messages because of ACNET errors - but both slot0 and the MFC seem to be functioning. A Control X from the shell caused the red reset light to come on and we looked at the scope to see if triggers on the Trig2 line were appearing. Once the soft reboot is initiated you have to wait for more than a minute before the slot0 provides a backplane reset again with a Control X. Out of 30 tries there was a Trig2 seen only on 11 attempts. However, even when the triggers were being issued by the MFC, the slot0 was not responding by accessing the board. Upon examining the RebootHooks function used for CM1 and Hins, it was seen that a DisableTrigSense() call before a sys-reset, which disables the slot0 response to triggers, was missing in the linac code. When this call was added, the slot0 would function correctly to a soft reboot provided the triggers were coming from the MFC. To remove the slot0 from the scenario, we had only the MFC connected to the backplane and simulated the soft reboot by pulling the sys-reset line low manually. In this test, the success was only 13/50 attempts. Success was being measured by the presence of triggers coming out of the FPGA.
This large number of failures prompted me to look at the logic that generates the triggers. The process seems to involve a lot of internal logic in the FPGA and handshaking between the DSP and the FPGA with each trigger. The implication from Vitali's testing was that the configuration did not happen correctly and this
is why there were no triggers. I believe the issue is deeper than this. We need to look for more direct indicators of whether the reboot resulted in code being reloaded from Flash. In the case of the CM1 MFC we did the same test and the success was 10/10 times. In this case it might be OK to look at the trigger showing up as a test of a proper configuration because the data acquisition is directly initiated by the Start trigger.

To summarize, when a backplane reset is received by the MFC, it appears that it attempts to reconfigure but a more direct indicator must be used to confirm this.
There may be a problem with reinitializing the logic of trigger generation which may be the real cause for the missing triggers rather than any issue with the loading from Flash. My notes indicate issues with this triggering scheme from a couple of years ago (01-06-10). Need to study the changes since then.

#6 Updated by Philip Varghese almost 8 years ago

The soft reboot feature was shown to work reliably. It required the removal of a 300 ms delay (DSP code) included in the
DSP initialization process which holds off the FPGA configuration. It appears that this delay may be safely removed as we
do not have it in other projects and it does not appear to be serving any special purpose in the Linac either.
The testing needs to be repeated at the Linac development station.

#7 Updated by Vitali Tupikov almost 8 years ago

Development station Soft-reboot test.

Programmed dev station dsp with fixed code.
I made 4 soft reboots, all of them were successful.

For correct soft reboot operation the following slot0
code change is required (without it, regardless on the
presence of TRIG2 pulse just after Ctrl-X, slot0 never
starts accessing the MFC):
put DisableTrigSence() in lelmain.cpp/LelPPRebootHooks()
how it shown in attached picture.

#8 Updated by Brian Chase almost 8 years ago

It appears that there may be a solution, but it does not appear that there is a real understanding of the problem. Is there any more insight?

#9 Updated by Barry Barnes almost 8 years ago

The attached document contains a DSP reboot code thread which includes the Delay() function mentioned earlier, simulator results showing the number of instruction cycles required to reach various executions, and screen shots of the SYSRESET# and flash CE# waveforms. MFC #10, v2, was the only module in the crate. The SYSRESET# was provided by a pulse generator set to a low pulse width of 250ms. and a period of 5 s.



Also available in: Atom PDF