Project

General

Profile

AMC13

AMC13 tool

Run AMC13 tool with our AMC13 using:

AMC13Tool2.exe -c $GM2TRACKERDAQ_CFG/connections.xml -i ${AMC13_ID}

Understanding CMS-like variables

The AMC13 LV1_id variable will mean "fill number" for g-2 (EvN).

In the CMS meaning, OrN (orbit number) and BcN (bunch crossing number) can be combined to give a clock counter ( BcN + OrN*4096 ).

Loading firmware

See here: http://bucms.bu.edu/twiki/bin/view/BUCMSPublic/AMC13Tool2Recipes#UpdateFirmware

g-2 firmware

The firmware directory is here (make sure to use "g-2" versions): http://ohm.bu.edu/~dzou/firmware.cgi

The AMC13 firmware has some changes for g-2, see attached pdf.

There is also useful information in the Rider user manual (http://gm2-docdb.fnal.gov:8080/cgi-bin/RetrieveFile?docid=3409&filename=rider_user_guide.pdf&version=3).

The main change is the re-purposing of the OrN and BX_ID into a more conventional clock counter, which is 44-bit in total. The new format is:

BX_ID = clock counter bits 43:32 (12 bits in total, bits 20-31 of AMC header word 0)
OrN = clock counter bits 31:0 (32 bits in total, bits 16-47 of AMC header word 1)

Note that OrN has been changed from 16-bit to 32-bit to facilitate this, which means than the user space has been reduced from 32-bit to 16-bit.

For a 40MHz clock (25ns period), this means:

  • OrN LSB = 25ns
  • BX_ID LSB = 1FFFFFFFF ns (~215 s)
  • Overall 48-bit counter overflow is ~7E6 s (~81 days)

The tracker electronics thus need to be power cycled at least once every 81 days.

AMC13 IP configuration

The IP addresses of the two FPGAs on the AMC13 are set via the MCH. Make sure the MCH IP address has been set before following these steps (see here: https://cdcvs.fnal.gov/redmine/projects/gm2-tracker-readout-daq/wiki/UTCA#Set-MCH-IP-address)

Information on setting up the IP addresses can be found here: http://bucms.bu.edu/twiki/bin/view/BUCMSPublic/AMC13MmcSoftware

Additional info is here: http://bucms.bu.edu/twiki/bin/view/BUCMSPublic/AMC13DebuggingHints

Note that the AMC13 slot number is 13 (e.g. use --slot 13 argument).

The scripts can be found in $AMC13_STANDALONE_ROOT/dev_tools/amc13Config/. Some must be run from this directory.

If the python scripts don't work, try installing ipmitool (as root: "yum install ipmitool") and try again.

Check current IP addresses

Check current AMC13 FPGA IP addresses using the readIPs.py script (http://bucms.bu.edu/twiki/bin/view/BUCMSPublic/AMC13MmcSoftware#ReadIps)

Set IP addresses

Set them using applyConfig.py (http://bucms.bu.edu/twiki/bin/view/BUCMSPublic/AMC13MmcSoftware#ApplyConfig), store them using storeConfig.py (http://bucms.bu.edu/twiki/bin/view/BUCMSPublic/AMC13MmcSoftware#StoreConfig)

Connecting to PC by fiber

Warning: Updating this as I install it, not yet firm information

Plug the PCIe card with fiber ports into your PC and reboot it.

Out SFP+ (must use SFP+, SFP, e.g. not the ones we use for the logic boards & FC7s) into one of the PCIe card ports and also into DAQ 0 input on the AMC13. Connect them via a fiber.

When plug SFP+ into PCIe card, if network device (ethX) disappears (from ifconfig) and dmesg shows:

ixgbe 0000:04:00.0: failed to initialize because an unsupported SFP+ module type was detected.

then solution (from here) is to (as root) add the following line to /etc/modprobe.d/ixgbe.conf (create it if it doesn't already exist):

options ixgbe allow_unsupported_sfp=1

then restart the drivers using the command:

rmmod ixgbe; modprobe ixgbe

The ethX device should now appear again.

To check if the connection is properly established, run (as root):

ethtool <ethX>

and check that see the following two lines in there:

    Supported link modes:   10000baseT/Full 

which shows that the 10Gb ethernet link is working (e.g. SFP+) and

    Link detected: yes

which shows that it is connected to something and communications are possible.

Note that our PCIe card (at least in UCL and lab 3) is this one. The linked page also includes a list of drivers including linux ones, but I'm not sure we actually need them.

Trouble-shooting

Cannot read AMC13 IP addresses

If get the following response to readIPs.py:

Reading IP addresses of board in slot 13 from host 192.168.1.8
T2 IP Bytes:
ipmitool -H 192.168.1.8 -U '' -P '' -T 0x82 -b 7 -t 0xa4 raw 0x32 0x34 0 11 0 4
Unable to send RAW command (channel=0x7 netfn=0x32 lun=0x0 cmd=0x34 rsp=0xc3): Timeout

This means that the MCH communication is OK, but the AMC13 cannot be contacted. Check the AMC13 is plugged in an the level is pushed in. It was up to a minute to start up after a power cycle.

No data coming out of AMC13 after swapping trigger fiber

After swapping AMC13 input trigger cable (e.g. real trigger for loop-back), need to power cycle the AMC13.

Reset AMC

ws 0x0 0x10

Loading firmware hangs

If loading firmware hangs at the "Programming XXX.mcs to flash address 0x400000..." etc line, it might be that you have the IP address for the two boards the wrong way around. Confusingly the AMC13 seems to default to T2 having an IP address 1 lower than T1.