Project

General

Profile

Grabbing the kernel modules we need for a new machine, or after a yum kernel upgrade

uboonedaq-seb-01> scp /lib/modules/2.6.32-220.23.1.el6.x86_64/kernel/drivers/misc/nevispc* :/lib/modules/2.6.32-358.6.2.el6.x86_64/kernel/drivers/misc/

uboonedaq-seb-01> scp /lib/modules/2.6.32-220.23.1.el6.x86_64/kernel/drivers/misc/windrvr6* :/lib/modules/2.6.32-358.6.2.el6.x86_64/kernel/drivers/misc/

If, instead of scp'ing the .ko's, we need to build them, pause and read this:

  • The history here is important. We bought a 3-seat, 1-node license from Jungo: We have an agreement that Eric, Gennadiy, Georgia only will develop code that uses the Windriver module, and then only on uboonedaq-evb, after which it may be pushed out to any node. What one gets from Jungo, after filling out an online form, is, e.g., /home/uboonedaq/jungo/WD1100LNX86_64.tgz. Unwinding that gives some source code, but most important, a 64 bit ELF (in our case) binary executable at WinDriver/wizard/wdwizard. When this is run it stamps out boilerplate code, which holds a main and a Makefile and a couple header files. This is what the user is intended to use to build out one's code. Please read through to the end of the next bullet point here to see that this whole wdwizard gymnastics is not re-performed unless one downloads a new WDXYZ.tgz. It is not clear we'll ever do this in the life of the MicroBooNE experiment. The official instructions which we use selectively are at http://www.jungo.com/st/support/documentation/windriver/11.0.0/wdpci_manual.mhtml/wd_install_process.html.
  • When wdwizard is run, a window pops up that shows us a number in a box, which one dutifully reports to a Jungo sales rep. They then give us an N digit code which we fill in the box where the wizard asks for a license key. This is what our $10k buys us. I think one may proceed here with no key, in which case the driver sleeps every 20 minutes or so and prompts for some irritating series of actions. Since we can't abide this during running we ponied up for the license. The stamped-out code holds this license key, and reports it to the module driver, which is happy. We have done one extra thing after stamping out our code: we have changed the main in that code into a function and have edited the Jungo Makefile to build a .so, rather than a binary executable. The wizard asks for a project name; we responded with nevispcie, hence one sees a bunch of source files in the wizard/nevispcie directory. The compiled code ends up in wizard/nevispcie/linux including our desired libnevispcie.so. We claim everything through this point does not need to be done again unless a new tgz is obtained from Jungo.
  • Separately, one needs to build the windriver api and then the actual module. The wdapi is built via the script, we mention below. The module building involves going to the redist directory, doing a ./configure; make; make install; That makes the windrvr6.ko -- namely the kernel module that you need. They are found in /etc/rc.local and are thus executed on each boot.
  • Note everything has been cp -r'd over to /uboone/windriver/v11_00_00/. Look at and run /uboone/windriver/v11_00_00/build-windriver.sh to do the steps of building, as described above.
  • Instructions for WinDriver 12.06.00 install and configuration, provided by Gennadiy:
    1. yum installed rgang on ubdaq-prod-ws01
    2. updated /uboone/configureOnBoot-seb.sh, which now installs and configures windriver12.06 on seb0{1,9}
    3. restarted sebs: rgang root@ubdaq-prod-seb0'{2-9}' 'reboot'
    4. checked if the windriver kernel module is loaded: rgang root@ubdaq-prod-seb'{1,10}' 'lsmod |grep -E "(wind|nevis)"'
    5. made changes to ~uboonedaq/.bash_profile, so the new daq release is loaded on seb0{1-9}.
  • We advocate the following when building out this driver on a new cluster of machines, presuming a new tgz is not procured. Have sysadmin's install the full puppet-config'd system. Then mv /uboone/ over to the machine on that cluster which is the exported mount point to be nfs-mounted by the rest of the machines. This will include /uboone/windriver. (All our ups's are there at /uboone.) Run the script there to build the .so and the windrvr6.ko. You are now done compiling/linking things. You shouldn't have to ever do it again for this cluster. Ask the sysadmins to now go cd /uboone/windriver/v11_00_00/source/redist; ./configure; make install; source /uboone/setup; setup windriver v11_00_00 -f Linux64bit+2.6-2.12 -q debug; wdreg windrvr6 auto; chmod 666 /dev/windrvr6
    on each machine in the cluster that holds a PCIe card. Note these commands are not building anything, but basically mv'ing executables and insmod'ing/modprobe'ing the driver. The last two are performed upon each reboot in /etc/rc.local. As yourself, get on each one of the machines and say nevispcie_run: if it works it's cuz you've properly installed the module.
  • Finally, depending on whether you ran the build-windriver.sh, you must go and comment out the line static BOOL IsItemExists(PWDC_DEVICE pDev, ITEM_TYPE item); in /uboone/windriver/v*/include/nevispcie.h. You'll see a lot of compiler errors about this function when you go to build your code later, if you do not.
**Quick tips from Wes during CAPTAIN install:
  • Make sure the WinDriver headers and whatnot are in /usr/local/WinDriver. If they aren't, copy them over from a machine that has them.
  • For each machine you intend to run the pcie cards on, make sure to do: cd /uboone/windriver/v11_00_00/source/redist; ./configure; make install; source /uboone/setup; setup windriver v11_00_00 -f Linux64bit+2.6-2.12 -q debug; wdreg windrvr6 auto; chmod 664 /dev/windrvr6
  • Put everything from "source /uboone/setup" on from the previous line into the /etc/rc.local file, as recommended in tthe windrvr make.

National Instruments drivers

These are needed for configuring the ASICS on seb10. Upon kernel upgrades the NI drivers can disappear, much as is the case with the windriver drivers. The following two commands show the hardware, and then the problem.

# There are as many lines here as there are USB connections, including hub connections. We just show one.
-bash-4.1$ lsusb | grep National | tail -1 
Bus 001 Device 018: ID 3923:718a National Instruments Corp. 
-bash-4.1$ lsdaq
--------------------------------
Detecting National Instruments DAQ Devices
Found the following DAQ Devices:
libnipalu.so failed to initialize
Perhaps you need to run updateNIDrivers
/usr/local/bin/lsdaq: line 7: 31879 Aborted                 /etc/natinst/nidaqmxbase/bin/daqmxbase_listdevices
--------------------------------

# So, we just run that updateNIDrivers

-bash-4.1$ ksu

# we show only some of the output below, as indicated by the elipses
# At some point you must say yes to  the reboot to load the .ko's. (Well, without cowboy'ing the insmod's.)

[root@ubdaq-prod-seb10 echurch]# updateNIDrivers 

Configuring for linux kernel version 2.6.32-642.6.2.el6.x86_64.

********************************* NOTE *********************************
Using kernel headers found in /lib/modules/2.6.32-642.6.2.el6.x86_64/source.
If this does not correspond to the location of the 2.6.32-642.6.2.el6.x86_64 headers,
then define KERNELHEADERS in your environment to point to the location
of the kernel headers, define KERNELTARGET as the version of the
kernel for which to compile, and then rerun ./configure.
********************************* NOTE *********************************

Kernel has reparent_to_init(): no
Number of arguments for do_munmap(): 3
pte_offset function: pte_offset_kernel()
Levels in page table: 4
Kernel has remap_pfn_range: yes
...
Installing NI-KAL:
 NI-KAL successfully updated.
Updating client modules:
 nipalk.ko successfully updated.
 NiViPciK.ko successfully updated.
...
Would you like to reboot [yes|no]

Weird NFS4 problem where files are owned by nobody.nobody

nfsidmap -c

rsyncing /uboone from test-stand as products

bash-4.1$ uname -a
Linux ubdaq-prod-evb.fnal.gov 2.6.32-358.23.2.el6.x86_64 #1 SMP Wed Oct 16 11:13:47 CDT 2013 x86_64 x86_64 x86_64 GNU/Linux
bash-4.1$ whoami
products
bash-4.1$ pwd
/uboone
bash-4.1$ nohup rsync -avzr  -e ssh uboonedaq-evb:/uboone/* . >& nohup_rsync_12Feb2014.out &

similarly, rsyncing up uboonepab02 (executed as ksu ) from uboonedaq-evb when uboonepab02's gotten crufty.

Note this puts 'em at /home/uboone, which is a mount point for /uboone on uboonepab02, where the build really needs 'em. We may only need to rsync particular packages. rsync preserves dates and permissions, etc.

uboonepab02> nohup rsync -avz -e ssh -K root@uboonedaq-evb.fnal.gov:/uboone /home/ > /uboone/rsync.txt &