Project

General

Profile

Task #14512

Support for Rich's MPS transmission loss task

Added by John Diamond over 3 years ago. Updated over 3 years ago.

Status:
Assigned
Priority:
Normal
Assignee:
Category:
MPS
Start date:
11/15/2016
Due date:
% Done:

0%

Estimated time:
16.00 h
Spent time:
Duration:

Description

Add scaling for the SNS Dump to the PXINT front end and add a call-back that Rich can use to attach his task to the 1KHz CW mode interrupt.

PA_D150Roger_T_s_Param.gif (7.59 KB) PA_D150Roger_T_s_Param.gif errors on Z:LOSSxx devices John Diamond, 11/21/2016 02:55 PM
PA_D150Roger_T_s_Param2.gif (7.07 KB) PA_D150Roger_T_s_Param2.gif All devices working now John Diamond, 11/21/2016 04:51 PM
d150_3.gif (7.2 KB) d150_3.gif John Diamond, 11/28/2016 01:34 PM
d150_4.gif (7.33 KB) d150_4.gif John Diamond, 11/29/2016 10:24 AM

History

#2 Updated by John Diamond over 3 years ago

Added Rich's module to the nbeam startup script for testing.

#3 Updated by John Diamond over 3 years ago

Implemented a feature that allows a user to provide a semaphore to be given after each beam pulse interrupt.

VMEInt::pulseListenerAdd( unsigned int devId, SEM_ID semId )

Example:
sem = semBCreate( SEM_Q_PRIORITY, SEM_FULL );
VMEInt::getInstance().pulseListenerAdd( 0x01, sem );

#4 Updated by John Diamond over 3 years ago

Implemented a feature that allows a user to cause an MPS trip through software.

VMEInt::mpsSoftTrip( unsigned int devId )

Example:
VMEInt::getInstance().mpsSoftTrip( 0x01 );

#5 Updated by Richard Neswold over 3 years ago

I'm updating my driver to use these entry points. Which VMEInt.h do I use? There are 18 of them on nova.fnal.gov. The most logical appears to be /export/home/rfies/esd/rfiinst/inc/vmeint/VMEInt.h, but I get:

detector.cpp:222: error: 'class VMEInt' has no member named 'pulseListenerAdd'

so it must be a different VMEInt.h.

#6 Updated by John Diamond over 3 years ago

Use:

/home/rfies/esd/rfiinst/devinc/vmeint/VMEInt.h

(I'm not sure how a VMEInt.h got into inc/vmeint/ as I did not have my Makefile configured to deploy a copy until you mentioned it.)

#7 Updated by Richard Neswold over 3 years ago

It compiles, with warnings, using these directories:

ADDED_C++FLAGS := -mno-strict-align -W -Werror \
    -I /export/misc/bdeelinux/jdiamond/esd/src/vmeint \
    -I /export/misc/bdeelinux/jdiamond/esd/src/timing \
    -I /export/misc/bdeelinux/jdiamond/esd/src/adinsthw \
    -I /home/rfies/esd/rfiinst/inc \
    -I /home/rfies/esd/rfies/inc

Are warnings expected when compiling these header files?

In file included from /export/misc/bdeelinux/jdiamond/esd/src/vmeint/VMEInt.h:30,
                 from rfinst.cpp:5:
/export/misc/bdeelinux/jdiamond/esd/src/vmeint/IIntDAQ.h:712: warning: `virtual short int* IIntDAQ::ToroidGetMemory(int)' was hidden
/export/misc/bdeelinux/jdiamond/esd/src/vmeint/FastTorDrv.h:364: warning:   by `short int* FastTorDrv::ToroidGetMemory(unsigned char)'
In file included from /export/misc/bdeelinux/jdiamond/esd/src/vmeint/VMEInt.h:31,
                 from rfinst.cpp:5:
/export/misc/bdeelinux/jdiamond/esd/src/vmeint/IIntDAQ.h:134: warning: `virtual int IIntDAQ::setThreshold(uint16_t, int8_t)' was hidden
/export/misc/bdeelinux/jdiamond/esd/src/vmeint/DcctDrv.h:332: warning:   by `int DcctDrv::setThreshold(int, int8_t)'
/export/misc/bdeelinux/jdiamond/esd/src/vmeint/IIntDAQ.h:135: warning: `virtual int IIntDAQ::setPulseDir(uint16_t, int)' was hidden
/export/misc/bdeelinux/jdiamond/esd/src/vmeint/DcctDrv.h:333: warning:   by `int DcctDrv::setPulseDir(int, int)'
/export/misc/bdeelinux/jdiamond/esd/src/vmeint/IIntDAQ.h:136: warning: `virtual int IIntDAQ::setDroopCor(uint16_t, int)' was hidden
/export/misc/bdeelinux/jdiamond/esd/src/vmeint/DcctDrv.h:334: warning:   by `int DcctDrv::setDroopCor(int, int)'
In file included from /export/misc/bdeelinux/jdiamond/esd/src/vmeint/DAQPool.h:20,
                 from /export/misc/bdeelinux/jdiamond/esd/src/vmeint/VMEInt.h:32,
                 from rfinst.cpp:5:
/export/misc/bdeelinux/jdiamond/esd/src/vmeint/IIntDAQ.h:712: warning: `virtual short int* IIntDAQ::ToroidGetMemory(int)' was hidden
/export/misc/bdeelinux/jdiamond/esd/src/vmeint/MirrTorDrv.h:655: warning:   by `short int* MirrTorDrv::ToroidGetMemory(unsigned char)'

#8 Updated by Richard Neswold over 3 years ago

I've changed the include paths to:

ADDED_C++FLAGS := -mno-strict-align -W -Werror \
    -I /home/rfies/esd/rfiinst/devinc \
    -I /home/rfies/esd/rfiinst/devinc/vmeint \
    -I /home/rfies/esd/rfiinst/devinc/nmltor \
    -I /home/rfies/esd/rfies/devinc

These feel better since they're more consistent (my previous attempt was more scattered.) However, I'm not finding all the headers anymore. BELMgr.h, BBM.h, ChainFilter.h, PxieTorDrv.h, etc are not found in these paths.

#9 Updated by John Diamond over 3 years ago

Yes, those warnings are to be expected - there's still some cleanup to be done in this code.

All of those missing headers look like they're included by VMEInt. For now I'm going to push out all of the headers in the project to /home/rfies/esd/rfiinst/devinc/vmeint/.

#10 Updated by Richard Neswold over 3 years ago

The PXIE Loss task is ready for testing.

#11 Updated by John Diamond over 3 years ago

I'm getting an error when I call PXIELOSS_create_mooc_instance -

ld 1, 1, "vxworks_boot/v6.4/module/mv5500/pxie-loss-1.0.out" 
*** ALARM DOWNLOAD DISABLED ***
ppcLib: adjusting library for MVME-5500...
value = 40769696 = 0x26e18a0

PXIELOSS_create_mooc_class 0x30
value = 0 = 0x0
PXIELOSS_create_mooc_instance 0xa0, 0
Exception thrown in PXIE-LOSS object init: board not found
value = -1 = 0xffffffff

#12 Updated by Richard Neswold over 3 years ago

The second argument to PXIELOSS_create_mooc_instance is the DIP switch setting of the hardware. I use that to locate the hardware (and I verify it by comparing a board ID located in a register.) Since I'm not actually controlling the hardware, I released a new version (machine-protection:pxieloss|cf75148f) where all the hardware-related stuff is deleted.

#13 Updated by John Diamond over 3 years ago

PXIE loss task control devices:

P:LOSS12 difference between ring 1 and ring 2 (in %)
P:LOSS1D difference between ring 1 and dump (in %)
P:LOSS2D difference between ring 2 and dump (in %)
P:LOSSMX maximum allowed difference (in mA)

nbeam is setup with filter chains identical to PXINT for Ring Pickup 1 (ID 0x03, Z:TOC990), Ring Pickup 2 (ID 0x13, Z:TOC991) and the dump (ID 0x16, Z:TOC992).

However, I'm getting errors for the Z:LOSSxx devices. Z:LOSSMX seems to work.

errors on Z:LOSSxx devices

#14 Updated by Richard Neswold over 3 years ago

All my MOOC methods are wrapped with try-catch clauses. The [57 -106] status is returned when an exception is thrown.

I've released a new version which prints to the console the value of e.what() so we can see what the exception was.

#15 Updated by Richard Neswold over 3 years ago

Got too aggressive at ripping out hardware code. Accidentally ripped out code that sized the reading array. Latest commit (machine-protection:pxieloss|f1a80a40) will fix it.

#16 Updated by John Diamond over 3 years ago

OK, looks like the devices are all working now.

All devices working now

However, I'm not sure about the operation. I have all of the devices set to a maximum 10% loss but they won't trip and the readings show 0% loss when the intensity devices show a loss of about 18% between 990 and 991.

#17 Updated by John Diamond over 3 years ago

Just to note - we have the input to TOC991 and TOC992 are T'd off at the source so we expect them to always read the same.

#18 Updated by Richard Neswold over 3 years ago

I had Ring 2's ID set to 0x1b instead of 0x13.

Added (machine-protection:pxieloss|c97a1361) a function, PXIELOSS_testReadings(), which displays the three inputs' values.

#19 Updated by John Diamond over 3 years ago

Rebooted, but the devices are still showing no loss. Is your code setup to handle a loss between two negative signals?

However, I can make the task trip by setting a loss threshold of 0%.

#20 Updated by John Diamond over 3 years ago

Try building against the headers in -

/home/rfies/esd/rfiinst/testinc/vmeint/

#21 Updated by Richard Neswold over 3 years ago

Makefile has this:

ADDED_C++FLAGS := -mno-strict-align -W -Wno-error \
    -I /home/rfies/esd/rfiinst/testinc \
    -I /home/rfies/esd/rfiinst/testinc/vmeint \
    -I /home/rfies/esd/rfiinst/testinc/nmltor \
    -I /home/rfies/esd/rfies/testinc \
    -I /export/misc/bdeelinux/jdiamond/esd/src/vmeint

but building gives this:

In file included from /home/rfies/esd/rfiinst/testinc/vmeint/ScalingACNET.h:13,
                 from /home/rfies/esd/rfiinst/testinc/vmeint/ACNETInterface.h:19,
                 from /home/rfies/esd/rfiinst/testinc/vmeint/VMEInt.h:22,
                 from rfinst.cpp:5:
/home/rfies/esd/rfiinst/testinc/stdint.h:74:62: targets.h: No such file or directory
In file included from /home/rfies/esd/rfiinst/testinc/vmeint/VMEInt.h:30,
                 from rfinst.cpp:5:
/home/rfies/esd/rfiinst/testinc/vmeint/IIntDAQ.h:712: warning: `virtual short int* IIntDAQ::ToroidGetMemory(int)' was hidden
/home/rfies/esd/rfiinst/testinc/vmeint/FastTorDrv.h:364: warning:   by `short int* FastTorDrv::ToroidGetMemory(unsigned char)'
In file included from /home/rfies/esd/rfiinst/testinc/vmeint/VMEInt.h:31,
                 from rfinst.cpp:5:
/home/rfies/esd/rfiinst/testinc/vmeint/IIntDAQ.h:134: warning: `virtual int IIntDAQ::setThreshold(uint16_t, int8_t)' was hidden
/home/rfies/esd/rfiinst/testinc/vmeint/DcctDrv.h:332: warning:   by `int DcctDrv::setThreshold(int, int8_t)'
/home/rfies/esd/rfiinst/testinc/vmeint/IIntDAQ.h:135: warning: `virtual int IIntDAQ::setPulseDir(uint16_t, int)' was hidden
/home/rfies/esd/rfiinst/testinc/vmeint/DcctDrv.h:333: warning:   by `int DcctDrv::setPulseDir(int, int)'
/home/rfies/esd/rfiinst/testinc/vmeint/IIntDAQ.h:136: warning: `virtual int IIntDAQ::setDroopCor(uint16_t, int)' was hidden
/home/rfies/esd/rfiinst/testinc/vmeint/DcctDrv.h:334: warning:   by `int DcctDrv::setDroopCor(int, int)'
In file included from /home/rfies/esd/rfiinst/testinc/vmeint/DAQPool.h:20,
                 from /home/rfies/esd/rfiinst/testinc/vmeint/VMEInt.h:32,
                 from rfinst.cpp:5:
/home/rfies/esd/rfiinst/testinc/vmeint/IIntDAQ.h:712: warning: `virtual short int* IIntDAQ::ToroidGetMemory(int)' was hidden
/home/rfies/esd/rfiinst/testinc/vmeint/MirrTorDrv.h:655: warning:   by `short int* MirrTorDrv::ToroidGetMemory(unsigned char)'
make: *** [rfinst.o] Error 1

#22 Updated by John Diamond over 3 years ago

Something strange is going on. I'm following up with Duane...

#23 Updated by Richard Neswold over 3 years ago

The include file issue is fixed (machine-protection:pxieloss|2130ebdc).

The previous problem is also fixed (machine-protection:pxieloss|655f9d74). Arden and I were assuming positive values for the current, but they're being scaled as negative.

#24 Updated by John Diamond over 3 years ago

It appears that VMEInt::mpsSoftTrip() is being called (continuously) when the tasks are not reporting as tripped.

#25 Updated by Richard Neswold over 3 years ago

New version has been released (machine-protection:pxieloss|9972e36a).

John says VMEInt::mpsSoftTrip() is only used to trip the system. VMEInt::mpsReset() is used to recover.

#26 Updated by John Diamond over 3 years ago

OK, trips and reset commands seem to be working properly. Last (hopefully) issue, the Z:LOSSMX device shows a 587ma total loss when it really should be about 0.44ma.

#27 Updated by Richard Neswold over 3 years ago

The scaling for Z:LOSSMX was incorrect. I've adjusted it.

#28 Updated by John Diamond over 3 years ago

That looks good now. I've deployed the new code out to pxint. Could you update your P:LOSSxx devices to refer to pxint? Here's how I'm loading your code in my startup script:

ld 1, 1, "vxworks_boot/v6.4/module/mv5500/pxie-loss-1.0.out" 

PXIELOSS_create_mooc_class 0x30
PXIELOSS_create_mooc_instance 0xa0, 0

#29 Updated by Richard Neswold over 3 years ago

I moved P:LOSS* to PXINT. The SDDN OID field was also changed to 0xa0.

Also, PXIELOSS_create_mooc_instance now only has one argument.

#30 Updated by John Diamond over 3 years ago

It appears the scaling is off on P:LOSSMX just like it was on Z:LOSSMX.

#31 Updated by Richard Neswold over 3 years ago

Reload D150.

#32 Updated by Richard Neswold over 3 years ago

How often is the dump's reading (P:MX1EPC) updated?

#33 Updated by John Diamond over 3 years ago

The dump should be updated at the same rate as the other devices. Last night we were out at PXIE double checking that the ring pickup MPS scaling was correct so we had a trigger signal hooked up to the ring pickup digitizer but none of the others. Before we left I was able to verify that the scaling on P:LOSSMX was correct now.

We did notice that mpsSoftTrip would not work when the MPS timing check is disabled, which is the way we are running the ring pickup until we deploy Ning's new firmware. I've modified mpsSoftTrip() to enable the timing check if it's off to force the MPS to trip. I've tested it on nbeam and have staged it for deploying to pxint.

One thing I noticed when testing on nbeam - it appears that the "Reset" function is not working properly once the device is tripped (Z:LOSS12 in the screenshot). I have the method instrumented with a console message so I know that mpsReset() is not getting called. I can reset the trip by changing the loss threshold.

(The signals are setup right now that Z:LOSS12 should be somewhere between 0.5-1%, so you should get a trip every few seconds).

#34 Updated by Richard Neswold over 3 years ago

John Diamond wrote:

One thing I noticed when testing on nbeam - it appears that the "Reset" function is not working properly once the device is tripped (Z:LOSS12 in the screenshot). I have the method instrumented with a console message so I know that mpsReset() is not getting called. I can reset the trip by changing the loss threshold.

The task is always running, always sampling and always making a decision whether the source should be one or off. The decision as to whether the source is off or on is done solely in the task's loop. When the signal exceeds the threshold, it "locks" the value that caused the trigger and shuts off the source. Even if the readings go below the threshold, the locked value keep the source off. There are two ways to re-enable the source:

  1. Raise the threshold setting. If the locked value is less than the new threshold, the task no longer treats the locked value as special and will re-enable the source.
  2. Remove the problem causing the losses and reset the triggered device. The BASIC CONTROL reset only sets the locked value to 0.0. The task's main loop will re-enable the source during its next iteration only if the new readings don't exceed the threshold.

You're expecting to see an mpsReset when you reset the device, but that would cause a potentially damaging beam to be emitted, even if briefly. I only re-enable the source when the losses are acceptable.

#35 Updated by Richard Neswold over 3 years ago

I changed how I calculate the percentages (machine-protection:pxieloss|eba186c2). If the magnitude of the current decreases in the downstream sensor, the percentage will be negative. An increase in magnitude results in a positive percentage. The setting for this device needs to specify a positive or (usually) negative percentage.

#36 Updated by Richard Neswold over 3 years ago

Added another device to the driver, P:LOSSTR. This is a 3-element array device containing the values of Ring 1 (P:M01EPC), Ring 2 (P:M02EPC) and the Dump (P:MX1FPC), respectively, that caused the latest trigger. A basic control reset will reset the entire array to 0.0.

commit: machine-protection:pxieloss|4761dc7d

#37 Updated by John Diamond over 3 years ago

Fixed the scaling filter applied to the dump filter chain (was a CurrentFilter, changed to ScaleFactorFilter).
Found that the semId passed to VMEInt::pulseListenerAdd(..) was not being saved and thus not being given by the interrupt handler.

#38 Updated by Richard Neswold over 3 years ago

Arden and I worked out a few final details and feel the task is working. These details include:

  • I had the wrong ID for Ring 2.
  • The code I wrote to "latch" bad values was making it too confusing to recover from a trip (too many devices needed to be reset.) Now it recovers when the readings are within operating parameters.
  • The two issues mentioned by John in #14512-37.
  • If the task detects it is getting involved longer than 2mS, it'll trip the machine off.

With regards to the last point, the system seemed like it was able to handle a 1.4mS update rate. Unfortunately, we occasionally saw >5mS delays. After some investigating, we found the shell, running at priority 1, could hang onto the CPU for 5mS. We moved the shell's priority to 6 and things seemed very stable. 1.5mS was the largest delay we saw, no matter what we did in the shell.

I would recommend adding to the startup script:

taskPrioritySet(taskNameToId("tShell0"), 6)

#39 Updated by Richard Neswold over 3 years ago

Arden expressed interest in this task also monitoring the scraper charge detection, so this issue shouldn't be closed yet.

#40 Updated by John Diamond over 3 years ago

Updated pxintstartup to set "tShell0" to priority 6.

#41 Updated by John Diamond over 3 years ago

There are 4 scrapers (4 paddles each) that our front end handles. Arden mentioned that there were two scrappers (8 paddles) that you would need to monitor. Do you know which devices those were? The scrapper devices we provide are:
  • P:M01PxC
  • P:M11PxC
  • P:MX2PxC
  • P:MX5PxC

Like the dump these currently use the database for scaling. I'm going to talk with Andrea and Niral about moving the scaling for all of them into the front end and I will get back to you with filter chain IDs.

#42 Updated by John Diamond over 3 years ago

Per Arden's e-mail:

P:M01PRA (right), P:M01PLA (left), P:M01PBA (bottom), P:M01PTA (top)

P:M11PRA (right), P:M11PLA (left), P:M11PBA, (bottom), P:M11PTA (top)

#43 Updated by John Diamond over 3 years ago

The ID's are as follows:
Device Current Filter ID Average Filter ID
P:M01PTx 0x24 0x34
P:M01PRx 0x25 0x35
P:M01PBx 0x26 0x36
P:M01PLx 0x27 0x37
P:M11PTx 0x28 0x38
P:M11PRx 0x29 0x39
P:M11PBx 0x2a 0x3a
P:M11PLx 0x2b 0x3b

Note that the device's Arden gave us are average current devices. They average over a settable number of pulses and will have a slower response. The devices ending in C are not averaged.

I have a startup script that configures the new filter chains ready to go but haven't deployed it yet. I will coordinate with Andrea and Niral to get this done tomorrow morning.



Also available in: Atom PDF