Embedded Linux Software Development¶
About¶
Buildroot is a set of Makefiles and patches for building embedded Linux kernels and custom root file systems based on Busybox and ucLibc. This project customizes Buildroot by adding packages for Fermi specific products, board support for single-board computers (SBCs) and System-on-Module chips (SoM) used in Accelerator Division and tools to assist in developing embedded Linux applications on a shared development system (adlinux.fnal.gov).
Building¶
Quick Start¶
- Configure buildroot with make menuconfig. Fermi packages are under Package Selection / Fermi.
- If you need to configure the kernel run make linux-menuconfig. When you're done, copy the kernel config file from output/build/linux-custom/.config into ees-buildroot/.linux-config.
- Run make clean if you want your build to start from scratch, but this takes a long time. If you want to rebuild just a specific package then delete it's directory from ees-buildroot/output/build and if necessary, it's source package from ees-buildroot/dl.
- Run make
- From ees-buildroot/output/images, copy bzImage to a bootable flash drive with grub installed. A sample menu.lst file can be found in ees-buildroot/fs/skeleton/boot/grub. If booting with TFTP, copy bzImage over to /fecode-bd/linux_boot/ees/project on nova.fnal.gov. Alternatively, a target is provided in the buildroot Makefile for installing the boot images to /fecode-bd/linux_boot/ees for you:
make install_target_images BOARD=<board> [VARIANT=<variant>]
Build Using an Extracted Tool-chain¶
Buildroot allows you the option of using an external tool-chain, which can speed up the build process considerably. The tool-chain can be vendor supplied or extracted from another Buildroot build (extracting a tool chain is discussed below) . Using an external tool-chain will also allow you to share a tool-chain between multiple developers.
To configure Buildroot to use an external tool-chain installed on adlinux:- make menuconfig
- From toolchain menu:
- Select external toolchain / pre-installed
- Toolchain directory: /usr/local/products/elsd/<board>/<linux-X.Y_variant>/output/host/usr
- Select Kernel header version
- Select glibc for toolchain library (mvme8100) or uclibc (conc405x)
- Enable RPC support, C++ support, whatever else buildroot asks for
- Save configuration, exit, make clean
Node Configuration¶
The root filesystem image generated by ees-buildroot is intended to be generic and loaded into a ramfs at bootup, but there are certain files that need to be customized for each node, e.g. the network configuration. The solution here is to mount a node-specific flash storage or NFS file system that contains these files and use symlinks to map them to the image that is loaded into the ramdisk. The mount point for this filesystem is /rfs. You can specify if this is an NFS mount or a local drive in board/fermi/<board_variant>/rootsfs_overlay/etc/fstab.The files replaced with symlinks are:
- /etc/hostname -> /rfs/etc/hostname
- /etc/hosts -> /rfs/etc/hosts
- /etc/krb5.keytab -> /rfs/etc/krb5.keytab
- /etc/network/interfaces -> /rfs/etc/network/interfaces
- /etc/ssh_host_key -> /rfs/etc/ssh_host_key
- /etc/ssh_host_key.pub -> /rfs/etc/ssh_host_key.pub
- /etc/ssh_host_dsa_key -> /rfs/etc/ssh_host_dsa_key
- /etc/ssh_host_dsa_key.pub -> /rfs/etc/ssh_host_dsa_key.pub
- /etc/ssh_host_ecdsa_key -> /rfs/etc/ssh_host_ecdsa_key
- /etc/ssh_host_ecdsa_key.pub -> /rfs/etc/ssh_host_ecdsa_key.pub
- /etc/ssh_host_rsa_key -> /rfs/etc/ssh_host_rsa_key
- /etc/ssh_host_rsa_key.pub -> /rfs/etc/ssh_host_rsa_key.pub
- /usr/local/products/data -> /rfs/usr/local/products/data
Booting¶
The boot process is somewhat board-specific and you should refer to your board's documentation for more details. That said, we have developed some documentation that could help you out:A Makefile target is provided for deploying boot images to the proper location on fecode-bd:
make install_target_images BOARD=<board> VARIANT=<variant>
Board Support¶
Board support has been developed for several popular SBCs and SoMs used in Accelerator Division. If your board is not listed, don't panic! Adding a board support isn't that hard and you can probably get away with copying the board support package for a similar board and tweaking some configuration settings.
Supported SBCs and SoMs¶
Product | defconfigs | Architecture | Kernels Supported | Deployments |
---|---|---|---|---|
MVME 5500 | fermi_mvme5500_generic_defconfig | VME / PowerPC | 2.6.20 | |
MVME 8100 | fermi_mvme8100_generic_defconfig | VME / QorIQ (PowerPC) | 3.0.6 | Booster BPMs |
MVME 8105 | fermi_mvme8105_generic_defconfig | VME / QorIQ (PowerPC) | 3.0.6 | |
Concurrent 405x | fermi_conc405x_generic_defconfig | VME / Intel Core Duo | 3.16.7 | IOTA BPMs |
VMIC VMIVME7805 | fermi_vmivme7805_generic_defconfig | VME / Intel ??? | 3.16.7 | |
MitySOM 5CSX | fermi_mitysom5csx_lnmxmd_defconfig | SOM / ARM | 4.1.33-ltsi-rt38 | Linac Marx Modulators |
Novtech NOVSOM CV | fermi_socfpga_generic_defconfig | SOM / ARM | 4.1.22-rt | Test Stand |
Reflex CES Achilles SOM | fermi_achilles_generic_defconfig | SOM / ARM | 4.1.22-ltsi-rt23 | Mu2e SRS |
PC i486 | fermi_pc_i486_generic_defconfig | PC / i486 | 2.6.32.11 | PC104 |
PC i686 | fermi_pc_i686_generic_defconfig | PC / i686 | 4.19.50-rt22 | PCI104/PCIe104/PC |
Board Support Explained¶
There are two components to a board support package in ees-buildroot:- A buildroot defconfig file - this file contains the Buildroot configuration values that deviate from their default settings. It mostly consists of high-level configuration settings such as the architecture type, enabled packages, tool-chain location and location of the Linux kernel config, patches and other files. This file is located in the configs/ and has the form fermi_<board>_<variant>_defconfig. To load the configuration into Buildroot run
make fermi_<board>_<variant>_defconfig
- Board support directory - this directory contains all of the ancillary configuration files (Linux kernel config, Busybox config, gclibc/uclibc config), patch files and post-build scripts. Board directories are located in board/fermi/ and have the following structure:
board/ Main board support directory fermi/ Fermi board support directory common/ Configurations/patches/rootfs files/etc. common to all Fermilab board support packages patches/ Patches that are fermi-specific but not board-specific rootfs_overlay/ Fermilab-specific root filesystem overlay etc/ passwd Users that can login to Fermilab boards root/ .k5login Users that can ksu to root <board_variant>/ linux.config Linux kernel configuration for <board_variant> busybox.config Busybox configuration for <board_variant> uclibc.config ucLibc / glibc configuration for <board_variant> patches/ Patch files specific for <board_variant> rootfs_overlay/ Root filesystem overlay specific to <board_variant>
Fermilab Common Board Support¶
Notice the common/ directory structure under board/fermi/. This directory contains board support files that are common to all of the Fermilab board-support packages. For example, the /etc/passwd rootfs-overlay file contains the usernames of Fermilab personnel that are allowed to login to all embedded nodes.
Creating a New Board Support Package¶
If you're building Buildroot for a totally new board (not a variant of an existing board) then you probably want to build a new tool-chain as well. If a tool-chain that you wish to use already exists (i.e. if you are creating a variant support package for an existing board), when you go to configure Buildroot make sure that you select the external tool-chain for the board from:
/usr/local/products/elsd/Otherwise, have buildroot build a toolchain for you and extract it for external use later. Steps for creating a new board support package:
- Grab ees-buildroot from Git
- Copy board/fermi/someboard_variant to board/fermi/newboard_variant
- Copy configs/fermi_someboard_variant_defconfig to fermi_newboard_variant_defconfig
- Open configs/fermi_newboard_variant_defconfig, replace refernces to board/fermi/someboard_variant with board/fermi/newboard_variant
- Run make fermi_newboard_variant_defconfig to load new configuration
- Run make menuconfig
- Run make linux-menuconfig to configure kernel (optional) and make linux-update-defconfig to save
- Run make busybox-menuconfig to configure busybox (optional) and make busybox-update-defconfig to save
- Run make uclibc-menuconfig to configure uclinc (optional) and make uclibc-update-defconfig to save
- Any board_variant-specific root filesystem changes should go in board/fermi/newboard_variant/rootfs_overlay (any fermi-specific / board-agnostic changes should go in board/fermi/common/rootfs_overlay)
- Place any board-specific kernel or package patches in board/fermi/newboard_variant/patches (if you have fermi-specific / board-agnostic patches they should go in board/fermi/common/patches)
- Run make, test new kernel
- (optional) If you built a new toolchain now is a good time to extract the toolchain, reconfigure Buildroot to build with the external toolchain you just extracted, re-build and test again
- Run make savedefconfig to save configuration to defconfig.
make savedefconfig
- Copy defconfig to configs/fermi_newboard_variant_defconfig
cp defconfig configs/fermi_<board>_<variant>_defconfig
- Commit new board configuration to Git:
git add board/fermi/newboard_variant git add configs/fermi_newboard_variant
Extracting a Toolchain¶
Once you have created a board support package, you should consider extracting a tool-chain to speed-up re-builds and to allow other developers to build for your target.
make install_toolchain BOARD=<board> VARIANT=<variant>
The BOARD argument is required and should identify your board properly. The VARIANT argument is optional - generic will be used if you omit it.
Using a pre-built External Toolchain¶
Some toolchains can be downloaded as prebuilt, ready to use, toolchains. An example of this is the Linaro ARM toolchain used by the MitySOM 5CSX System On Module (Intel Cyclone V w/ ARM HPS SOM). The Linaro ARM toolchain was downloaded from:
https://releases.linaro.org/archive/14.09/components/toolchain/binaries/gcc-linaro-arm-linux-gnueabihf-4.9-2014.09_linux.tar.xz and installed on adlinux.fnal.gov at /usr/local/products/elsd/mitysom5csx/gcc-linaro-arm-Linux-gnueabihf-4.9.
The Buildroot toolchain configuration is then setup as follows:
Toolchain Type -> External toolchain Toolchain -> Linaro ARM 2014.09 Toolchain origin -> Pre-installed toolchain Toolchain path -> /usr/local/products/elsd/mitysom5csx/gcc-linaro-arm-Linux-gnueabihf-4.9
Development Tools¶
Once a tool-chain is extracted you can use it to develop your applications from your sandbox directory. We provide a Makefile include that you can use for building C/C++ applications located at/usr/local/products/elsd/include/elsd-latest.mk
To use it:
- Define global variables for your project in your Makefile -
- OS (EES_TARGET_OS)
- OS version (EES_TARGET_OS_VERSION)
- OS variant, optional - generic assumed (EES_TARGET_OS_VARIANT)
- Target board name (EES_TARGET_BOARD)
- Target board architecture (EES_TARGET_ARCH)
- Project name (EES_PROJECT)
- Include /usr/local/products/elsd/include/elsd-latest.mk in your makefile
Example usage from adinstbpm:
# EES_OUT = EES_PREFIX = $(prefix) EES_PROJECT = bpmd EES_TARGET_OS = linux ifeq ($(EES_TARGET_BOARD),mvme8100) EES_TARGET_BOARD = mvme8100 EES_TARGET_OS_VERSION = 3.0.6 else EES_TARGET_BOARD = conc405x EES_TARGET_OS_VERSION = 3.16.7 endif include /usr/local/products/elsd/include/elsd-latest.mk EES_CFLAGS = -I$(EES_ERL_LIBS)/cdev-1.2/include -I$(EES_ERL_LIBS)/acnet-2.1/include $(EES_PREF_CFLAGS) $(EES_PREF_LXRT_CFLAGS) -fno-strict-aliasing -D'BOARD_SUPPORT=$(BPMD_BOARD_ID)' EES_CPPFLAGS = -I$(EES_ERL_LIBS)/cdev-1.2/include -I$(EES_ERL_LIBS)/acnet-2.1/include $(EES_PREF_CPPFLAGS) $(EES_PREF_LXRT_CPPFLAGS) -fno-strict-aliasing -std=c++0x -I$(EES_INC) -I$(EES_ERL_LIBS)/cdev-1.2/include -I$(EES_ERL_LIBS)/acnet-2.1/include -I$(E ES_PREFIX)/usr/include/libnl3 -D'BOARD_SUPPORT=$(BPMD_BOARD_ID)' EES_LDFLAGS = $(EES_PREF_LDFLAGS) $(EES_PREF_LXRT_LDFLAGS) -lbbpm250x12 -lbtime -lnl-genl-3 -lnl-3 -lrt -lconfig++ ERLANG_INT = erl_interface-3.10.4 MYLIBS = $(EES_ERLANG_LIBDIR)/lib/$(ERLANG_INT)/lib MODULES = \ BBPM250x12DAQDeviceAdapter.o \ BBPM250x12DAQReader.o \ BoosterStateMachine.o \ BPM.o \ BPMError.o \ BPMSample.o \ BTimeTSGDeviceAdapter.o \ CLIAbortCommand.o \ CLIArmCommand.o \ CLIClearReadingsCommand.o \ CLICommand.o \ CLIConfigReloadCommand.o \ CLIDumpCommand.o \ CLIDumpFlashCommand.o \ CLIReadoutCommand.o \ CLIHelpCommand.o \ CLIListCommand.o \ CLIShowCommand.o \ CLIShutdownCommand.o \ CLISMDisableCommand.o \ CLISMEnableCommand.o \ ClosedOrbitReading.o \ ConfigurationManager.o \ Controller.o \ ControlMQClient.o \ DAQClosedOrbitReadoutRequest.o \ DAQFlashOrbitReadoutRequest.o \ DAQFlashTurnReadoutRequest.o \ DAQRawReadoutRequest.o \ DAQReader.o \ DAQReadoutRequest.o \ DAQReadoutRequestQueue.o \ DAQSmoothOrbitReadoutRequest.o \ DAQSystem.o \ DAQSystemArmed.o \ DAQSystemFailure.o \ DAQSystemState.o \ DAQSystemStatus.o \ DAQSystemInitialization.o \ DAQSystemReadout.o \ DAQSystemReady.o \ DAQSystemTriggered.o \ DAQTBTReadoutRequest.o \ FlashReading.o \ IReading.o \ MapCLICommands.o \ MachineContext.o \ Measurement.o \ MeasurementSpecification.o \ NoMachineStateMachine.o \ RawReading.o \ RawSample.o \ ReadoutSpecification.o \ SharedMemoryClient.o \ SharedMemoryHeader.o \ SharedMemoryManager.o \ SmoothOrbitReading.o \ StateMachine.o \ TBTReading.o \ TCLKTimestamp.o \ TriggerSpecification.o all: all_buildroot all_buildroot: bpmd bpmcli boosterbpm_acsys boostercrate_acsys clean: -rm -f *.o bpmd bpmcli boosterbpm_acsys boostercrate_acsys boosterbpm_acsys: $(MODULES) BoosterBPMACNET.o boosterbpm_fef_init.o $(CXX) $(MODULES) BoosterBPMACNET.o boosterbpm_fef_init.o -o boosterbpm_acsys $(EES_LDFLAGS) \ $(EES_ERL_LIBS)/cdev-1.2/priv/fef_driver_lib.o -L${MYLIBS} -lpthread -lerl_interface -lei boostercrate_acsys: $(MODULES) BoosterCrateACNET.o boostercrate_fef_init.o $(CXX) $(MODULES) BoosterCrateACNET.o boostercrate_fef_init.o -o boostercrate_acsys $(EES_LDFLAGS) \ $(EES_ERL_LIBS)/cdev-1.2/priv/fef_driver_lib.o -L${MYLIBS} -lpthread -lerl_interface -lei bpmcli: $(MODULES) cli.o $(CXX) $(MODULES) cli.o -o bpmcli $(EES_LDFLAGS) bpmd: $(MODULES) main.o $(CXX) $(MODULES) main.o -o bpmd $(EES_LDFLAGS) install_buildroot_target: install -d $(EES_BIN) install -D -m 0755 $(@D)/bpmd $(EES_BIN) install -D -m 0755 $(@D)/bpmcli $(EES_BIN) install -D -m 0755 $(@D)/boosterbpm_acsys $(EES_PREFIX)/usr/local/products/acsysfe/lib/cdev-1.2/priv install -D -m 0755 $(@D)/boostercrate_acsys $(EES_PREFIX)/usr/local/products/acsysfe/lib/cdev-1.2/priv install -D -m 0755 $(@D)/example-bpmd.conf $(EES_PREFIX)/etc/bpmd.conf install -D -m 0755 $(@D)/startacsys-bpmd $(EES_PREFIX)/usr/local/products/acsysfe/bin/startacsys install -D -m 0755 $(@D)/S70bpmd $(EES_PREFIX)/etc/init.d/ install_test: all scp -q bpmd nova:/fecode-bd/vxworks_write/fe/ees/bbpmts scp -q bpmcli nova:/fecode-bd/vxworks_write/fe/ees/bbpmts scp -q boosterbpm_acsys nova:/fecode-bd/vxworks_write/fe/ees/bbpmts/ scp -q boostercrate_acsys nova:/fecode-bd/vxworks_write/fe/ees/bbpmts/ scp -q example-bpmd.conf nova:/fecode-bd/vxworks_write/fe/ees/bbpmts/bpmd.conf scp -q S70bpmd nova:/fecode-bd/vxworks_write/fe/ees/bbpmts install_bbpmt2: all scp -q bpmd nova:/fecode-bd/vxworks_write/fe/ees/bbpmt2 scp -q bpmcli nova:/fecode-bd/vxworks_write/fe/ees/bbpmt2 scp -q boosterbpm_acsys nova:/fecode-bd/vxworks_write/fe/ees/bbpmt2 scp -q boostercrate_acsys nova:/fecode-bd/vxworks_write/fe/ees/bbpmt2 scp -q S70bpmd nova:/fecode-bd/vxworks_write/fe/ees/bbpmt2
Board/Architecture/OS Conditional Compilation¶
The ELSD tools make some pre-processor defines that you can use in your code to conditionally compile code based on the target configuration. Values for the pre-processor defines are in elsd_defines-latest.h.
Some useful defines are:
Name | Values | Description |
---|---|---|
ELSD_ARCH | ELSD_ARCH_x86 ELSD_ARCH_POWERPC ELSD_ARCH_ARM |
Target Architecture |
ELSD_OS | ELSD_OS_LINUX ELSD_OS_VXWORKS |
Target Operating System |
ELSD_BOARD | ELSD_BOARD_CONC405X ELSD_BOARD_MVME8100 ELSD_BOARD_MVME81XX ELSD_BOARD_MVME8105 ELSD_BOARD_MVME5500 ELSD_BOARD_VMIVME7805 |
Target Board |
Changes to elsd.mk¶
The elsd.mk Makefile is tracked in the ees-buildroot project Git repository under the elsd/ directory. To deploy changes so other developers on adlinux can use them, use the install_elsd target provided in the Buildroot Makefile:
make install_elsd VERSION=<version>
The VERSION argument is required. If you omit it, a directory listing of the current elsd include directory will be displayed to remind you of the current version number.
Creating new Buildroot packages for Fermi software¶
You may want to consider making a Buildroot package for your software or for software out there that your application depends on.
Remote Deployment¶
You can transfer new builds to your target using scp. Most targets live inside the controls firewall so in order to scp you must first configure ssh to proxy through outland.fnal.gov. Add the following to your ~/.ssh/config file:
Host target.fnal.gov ProxyCommand ssh user@outland.fnal.gov nc %h %p
Then to copy your_program to the target:
# scp -q your_program root@target.fnal.gov:/usr/local/products/bin/your_program
Keep in mind that the root filesystem is in a ramdisk, so the new file that you copy over will disappear after you reboot unless you rebuild and deploy your kernel+ramfs with the new changes. To deploy a new kernel, you can add an entry for nova.fnal.gov to your ~/.ssh/config file and copy the kernel to the linux_boot area:
# scp -q output/images/bzImage nova.fnal.gov:/fecode-bd/linux_boot/ees/project/bzImage.dev
Remote Debugging¶
You can debug executables on your target using GDB's remote features. Start the application with gdbserver:
$ gdbserver :2345 /usr/local/products/bin/your_program
Then, from the host that you built ees-buildroot on, establish a secure tunnel for GDB into the controls vlan:
$ ssh -f -N -L 2345:target.fnal.gov:2345 outland
Alternatively, you could log into the controls VPN and address your target directly.
Move into the build directory and connect to the target via the tunnel:
$ cd output/build/your_program $ gdb your_program (gdb) target remote localhost:2345