Project

General

Profile

History

Embedded Linux Software Development

About

Buildroot is a set of Makefiles and patches for building embedded Linux kernels and custom root file systems based on Busybox and ucLibc. This project customizes Buildroot by adding packages for Fermi specific products, board support for single-board computers (SBCs) and System-on-Module chips (SoM) used in Accelerator Division and tools to assist in developing embedded Linux applications on a shared development system (adlinux.fnal.gov).

Building

Quick Start

  1. Configure buildroot with make menuconfig. Fermi packages are under Package Selection / Fermi.
  2. If you need to configure the kernel run make linux-menuconfig. When you're done, copy the kernel config file from output/build/linux-custom/.config into ees-buildroot/.linux-config.
  3. Run make clean if you want your build to start from scratch, but this takes a long time. If you want to rebuild just a specific package then delete it's directory from ees-buildroot/output/build and if necessary, it's source package from ees-buildroot/dl.
  4. Run make
  5. From ees-buildroot/output/images, copy bzImage to a bootable flash drive with grub installed. A sample menu.lst file can be found in ees-buildroot/fs/skeleton/boot/grub. If booting with TFTP, copy bzImage over to /fecode-bd/linux_boot/ees/project on nova.fnal.gov. Alternatively, a target is provided in the buildroot Makefile for installing the boot images to /fecode-bd/linux_boot/ees for you:
    make install_target_images BOARD=<board> [VARIANT=<variant>]

Build Using an Extracted Tool-chain

Buildroot allows you the option of using an external tool-chain, which can speed up the build process considerably. The tool-chain can be vendor supplied or extracted from another Buildroot build (extracting a tool chain is discussed below) . Using an external tool-chain will also allow you to share a tool-chain between multiple developers.

To configure Buildroot to use an external tool-chain installed on adlinux:
  1. make menuconfig
  2. From toolchain menu:
    1. Select external toolchain / pre-installed
    2. Toolchain directory: /usr/local/products/elsd/<board>/<linux-X.Y_variant>/output/host/usr
    3. Select Kernel header version
    4. Select glibc for toolchain library (mvme8100) or uclibc (conc405x)
    5. Enable RPC support, C++ support, whatever else buildroot asks for
  3. Save configuration, exit, make clean

Node Configuration

The root filesystem image generated by ees-buildroot is intended to be generic and loaded into a ramfs at bootup, but there are certain files that need to be customized for each node, e.g. the network configuration. The solution here is to mount a node-specific flash storage or NFS file system that contains these files and use symlinks to map them to the image that is loaded into the ramdisk. The mount point for this filesystem is /rfs. You can specify if this is an NFS mount or a local drive in board/fermi/<board_variant>/rootsfs_overlay/etc/fstab.
The files replaced with symlinks are:
  • /etc/hostname -> /rfs/etc/hostname
  • /etc/hosts -> /rfs/etc/hosts
  • /etc/krb5.keytab -> /rfs/etc/krb5.keytab
  • /etc/network/interfaces -> /rfs/etc/network/interfaces
  • /etc/ssh_host_key -> /rfs/etc/ssh_host_key
  • /etc/ssh_host_key.pub -> /rfs/etc/ssh_host_key.pub
  • /etc/ssh_host_dsa_key -> /rfs/etc/ssh_host_dsa_key
  • /etc/ssh_host_dsa_key.pub -> /rfs/etc/ssh_host_dsa_key.pub
  • /etc/ssh_host_ecdsa_key -> /rfs/etc/ssh_host_ecdsa_key
  • /etc/ssh_host_ecdsa_key.pub -> /rfs/etc/ssh_host_ecdsa_key.pub
  • /etc/ssh_host_rsa_key -> /rfs/etc/ssh_host_rsa_key
  • /etc/ssh_host_rsa_key.pub -> /rfs/etc/ssh_host_rsa_key.pub
  • /usr/local/products/data -> /rfs/usr/local/products/data

Booting

The boot process is somewhat board-specific and you should refer to your board's documentation for more details. That said, we have developed some documentation that could help you out:

A Makefile target is provided for deploying boot images to the proper location on fecode-bd:

make install_target_images BOARD=<board> VARIANT=<variant>

Board Support

Board support has been developed for several popular SBCs and SoMs used in Accelerator Division. If your board is not listed, don't panic! Adding a board support isn't that hard and you can probably get away with copying the board support package for a similar board and tweaking some configuration settings.

Supported SBCs and SoMs

Product defconfigs Architecture Kernels Supported Deployments
MVME 5500 fermi_mvme5500_generic_defconfig VME / PowerPC 2.6.20
MVME 8100 fermi_mvme8100_generic_defconfig VME / QorIQ (PowerPC) 3.0.6 Booster BPMs
MVME 8105 fermi_mvme8105_generic_defconfig VME / QorIQ (PowerPC) 3.0.6
Concurrent 405x fermi_conc405x_generic_defconfig VME / Intel Core Duo 3.16.7 IOTA BPMs
VMIC VMIVME7805 fermi_vmivme7805_generic_defconfig VME / Intel ??? 3.16.7
MitySOM 5CSX fermi_mitysom5csx_lnmxmd_defconfig SOM / ARM 4.1.33-ltsi-rt38 Linac Marx Modulators
Novtech NOVSOM CV fermi_socfpga_generic_defconfig SOM / ARM 4.1.22-rt Test Stand
Reflex CES Achilles SOM fermi_achilles_generic_defconfig SOM / ARM 4.1.22-ltsi-rt23 Mu2e SRS
PC i486 fermi_pc_i486_generic_defconfig PC / i486 2.6.32.11 PC104
PC i686 fermi_pc_i686_generic_defconfig PC / i686 4.19.50-rt22 PCI104/PCIe104/PC

Board Support Explained

There are two components to a board support package in ees-buildroot:
  1. A buildroot defconfig file - this file contains the Buildroot configuration values that deviate from their default settings. It mostly consists of high-level configuration settings such as the architecture type, enabled packages, tool-chain location and location of the Linux kernel config, patches and other files. This file is located in the configs/ and has the form fermi_<board>_<variant>_defconfig. To load the configuration into Buildroot run 
    make fermi_<board>_<variant>_defconfig
  2. Board support directory - this directory contains all of the ancillary configuration files (Linux kernel config, Busybox config, gclibc/uclibc config), patch files and post-build scripts. Board directories are located in board/fermi/ and have the following structure:
    board/                         Main board support directory
  fermi/                       Fermi board support directory
    common/                    Configurations/patches/rootfs files/etc. common to all Fermilab board support packages
      patches/                 Patches that are fermi-specific but not board-specific
        rootfs_overlay/        Fermilab-specific root filesystem overlay
          etc/
            passwd             Users that can login to Fermilab boards
          root/
            .k5login           Users that can ksu to root
    <board_variant>/
      linux.config             Linux kernel configuration for <board_variant>
      busybox.config           Busybox configuration for <board_variant>
      uclibc.config            ucLibc / glibc configuration for <board_variant>
      patches/                 Patch files specific for <board_variant>
      rootfs_overlay/          Root filesystem overlay specific to <board_variant>

Fermilab Common Board Support

Notice the common/ directory structure under board/fermi/. This directory contains board support files that are common to all of the Fermilab board-support packages. For example, the /etc/passwd rootfs-overlay file contains the usernames of Fermilab personnel that are allowed to login to all embedded nodes.

Creating a New Board Support Package

If you're building Buildroot for a totally new board (not a variant of an existing board) then you probably want to build a new tool-chain as well. If a tool-chain that you wish to use already exists (i.e. if you are creating a variant support package for an existing board), when you go to configure Buildroot make sure that you select the external tool-chain for the board from: 

/usr/local/products/elsd/
Otherwise, have buildroot build a toolchain for you and extract it for external use later.

Steps for creating a new board support package:
  1. Grab ees-buildroot from Git
  2. Copy board/fermi/someboard_variant to board/fermi/newboard_variant
  3. Copy configs/fermi_someboard_variant_defconfig to fermi_newboard_variant_defconfig
  4. Open configs/fermi_newboard_variant_defconfig, replace refernces to board/fermi/someboard_variant with board/fermi/newboard_variant
  5. Run make fermi_newboard_variant_defconfig to load new configuration
  6. Run make menuconfig
  7. Run make linux-menuconfig to configure kernel (optional) and make linux-update-defconfig to save
  8. Run make busybox-menuconfig to configure busybox (optional) and make busybox-update-defconfig to save
  9. Run make uclibc-menuconfig to configure uclinc (optional) and make uclibc-update-defconfig to save
  10. Any board_variant-specific root filesystem changes should go in board/fermi/newboard_variant/rootfs_overlay (any fermi-specific / board-agnostic changes should go in board/fermi/common/rootfs_overlay)
  11. Place any board-specific kernel or package patches in board/fermi/newboard_variant/patches (if you have fermi-specific / board-agnostic patches they should go in board/fermi/common/patches)
  12. Run make, test new kernel
  13. (optional) If you built a new toolchain now is a good time to extract the toolchain, reconfigure Buildroot to build with the external toolchain you just extracted, re-build and test again
  14. Run make savedefconfig to save configuration to defconfig.
    make savedefconfig
  15. Copy defconfig to configs/fermi_newboard_variant_defconfig
    cp defconfig configs/fermi_<board>_<variant>_defconfig
  16. Commit new board configuration to Git:
           git add board/fermi/newboard_variant
       git add configs/fermi_newboard_variant

Extracting a Toolchain

Once you have created a board support package, you should consider extracting a tool-chain to speed-up re-builds and to allow other developers to build for your target.

make install_toolchain BOARD=<board> VARIANT=<variant>

The BOARD argument is required and should identify your board properly. The VARIANT argument is optional - generic will be used if you omit it.

Using a pre-built External Toolchain

Some toolchains can be downloaded as prebuilt, ready to use, toolchains. An example of this is the Linaro ARM toolchain used by the MitySOM 5CSX System On Module (Intel Cyclone V w/ ARM HPS SOM). The Linaro ARM toolchain was downloaded from:
https://releases.linaro.org/archive/14.09/components/toolchain/binaries/gcc-linaro-arm-linux-gnueabihf-4.9-2014.09_linux.tar.xz and installed on adlinux.fnal.gov at /usr/local/products/elsd/mitysom5csx/gcc-linaro-arm-Linux-gnueabihf-4.9.

The Buildroot toolchain configuration is then setup as follows:

Toolchain Type -> External toolchain
Toolchain -> Linaro ARM 2014.09
Toolchain origin -> Pre-installed toolchain
Toolchain path -> /usr/local/products/elsd/mitysom5csx/gcc-linaro-arm-Linux-gnueabihf-4.9

Development Tools

Once a tool-chain is extracted you can use it to develop your applications from your sandbox directory. We provide a Makefile include that you can use for building C/C++ applications located at 
/usr/local/products/elsd/include/elsd-latest.mk

To use it:
  1. Define global variables for your project in your Makefile -
    1. OS (EES_TARGET_OS)
    2. OS version (EES_TARGET_OS_VERSION)
    3. OS variant, optional - generic assumed (EES_TARGET_OS_VARIANT)
    4. Target board name (EES_TARGET_BOARD)
    5. Target board architecture (EES_TARGET_ARCH)
    6. Project name (EES_PROJECT)
  2. Include /usr/local/products/elsd/include/elsd-latest.mk in your makefile

Example usage from adinstbpm:

# EES_OUT         =
EES_PREFIX      = $(prefix)
EES_PROJECT     = bpmd

EES_TARGET_OS = linux
ifeq ($(EES_TARGET_BOARD),mvme8100)
EES_TARGET_BOARD = mvme8100
EES_TARGET_OS_VERSION = 3.0.6
else
EES_TARGET_BOARD = conc405x
EES_TARGET_OS_VERSION = 3.16.7
endif

include /usr/local/products/elsd/include/elsd-latest.mk

EES_CFLAGS      = -I$(EES_ERL_LIBS)/cdev-1.2/include -I$(EES_ERL_LIBS)/acnet-2.1/include $(EES_PREF_CFLAGS)     $(EES_PREF_LXRT_CFLAGS) -fno-strict-aliasing -D'BOARD_SUPPORT=$(BPMD_BOARD_ID)'
EES_CPPFLAGS    = -I$(EES_ERL_LIBS)/cdev-1.2/include -I$(EES_ERL_LIBS)/acnet-2.1/include $(EES_PREF_CPPFLAGS)   $(EES_PREF_LXRT_CPPFLAGS) -fno-strict-aliasing -std=c++0x -I$(EES_INC) -I$(EES_ERL_LIBS)/cdev-1.2/include -I$(EES_ERL_LIBS)/acnet-2.1/include -I$(E
ES_PREFIX)/usr/include/libnl3 -D'BOARD_SUPPORT=$(BPMD_BOARD_ID)'
EES_LDFLAGS     = $(EES_PREF_LDFLAGS)   $(EES_PREF_LXRT_LDFLAGS) -lbbpm250x12 -lbtime -lnl-genl-3 -lnl-3 -lrt -lconfig++
ERLANG_INT      = erl_interface-3.10.4
MYLIBS          = $(EES_ERLANG_LIBDIR)/lib/$(ERLANG_INT)/lib

MODULES = \
        BBPM250x12DAQDeviceAdapter.o \
        BBPM250x12DAQReader.o \
        BoosterStateMachine.o \
        BPM.o \
        BPMError.o \
        BPMSample.o \
        BTimeTSGDeviceAdapter.o \
        CLIAbortCommand.o \
        CLIArmCommand.o \
        CLIClearReadingsCommand.o \
        CLICommand.o \
        CLIConfigReloadCommand.o \
        CLIDumpCommand.o \
        CLIDumpFlashCommand.o \
        CLIReadoutCommand.o \
        CLIHelpCommand.o \
        CLIListCommand.o \
        CLIShowCommand.o \
        CLIShutdownCommand.o \
        CLISMDisableCommand.o \
        CLISMEnableCommand.o \
        ClosedOrbitReading.o \
        ConfigurationManager.o \
        Controller.o \
        ControlMQClient.o \
        DAQClosedOrbitReadoutRequest.o \
        DAQFlashOrbitReadoutRequest.o \
        DAQFlashTurnReadoutRequest.o \
        DAQRawReadoutRequest.o \
        DAQReader.o \
        DAQReadoutRequest.o \
        DAQReadoutRequestQueue.o \
        DAQSmoothOrbitReadoutRequest.o \
        DAQSystem.o \
        DAQSystemArmed.o \
        DAQSystemFailure.o \
        DAQSystemState.o \
        DAQSystemStatus.o \
        DAQSystemInitialization.o \
        DAQSystemReadout.o \
        DAQSystemReady.o \
        DAQSystemTriggered.o \
        DAQTBTReadoutRequest.o \
        FlashReading.o \
        IReading.o \
        MapCLICommands.o \
        MachineContext.o \
        Measurement.o \
        MeasurementSpecification.o \
        NoMachineStateMachine.o \
        RawReading.o \
        RawSample.o \
        ReadoutSpecification.o \
        SharedMemoryClient.o \
        SharedMemoryHeader.o \
        SharedMemoryManager.o \
        SmoothOrbitReading.o \
        StateMachine.o \
        TBTReading.o \
        TCLKTimestamp.o \
        TriggerSpecification.o

all: all_buildroot

all_buildroot: bpmd bpmcli boosterbpm_acsys boostercrate_acsys

clean:
        -rm -f *.o bpmd bpmcli boosterbpm_acsys boostercrate_acsys

boosterbpm_acsys: $(MODULES) BoosterBPMACNET.o boosterbpm_fef_init.o
        $(CXX) $(MODULES) BoosterBPMACNET.o boosterbpm_fef_init.o -o boosterbpm_acsys $(EES_LDFLAGS) \
        $(EES_ERL_LIBS)/cdev-1.2/priv/fef_driver_lib.o -L${MYLIBS} -lpthread -lerl_interface -lei

boostercrate_acsys: $(MODULES) BoosterCrateACNET.o boostercrate_fef_init.o
        $(CXX) $(MODULES) BoosterCrateACNET.o boostercrate_fef_init.o -o boostercrate_acsys $(EES_LDFLAGS) \
        $(EES_ERL_LIBS)/cdev-1.2/priv/fef_driver_lib.o -L${MYLIBS} -lpthread -lerl_interface -lei

bpmcli: $(MODULES) cli.o
        $(CXX) $(MODULES) cli.o -o bpmcli $(EES_LDFLAGS)

bpmd: $(MODULES) main.o
        $(CXX) $(MODULES) main.o -o bpmd $(EES_LDFLAGS)

install_buildroot_target:
        install -d $(EES_BIN)
        install -D -m 0755 $(@D)/bpmd $(EES_BIN)
        install -D -m 0755 $(@D)/bpmcli $(EES_BIN)
        install -D -m 0755 $(@D)/boosterbpm_acsys $(EES_PREFIX)/usr/local/products/acsysfe/lib/cdev-1.2/priv
        install -D -m 0755 $(@D)/boostercrate_acsys $(EES_PREFIX)/usr/local/products/acsysfe/lib/cdev-1.2/priv
        install -D -m 0755 $(@D)/example-bpmd.conf $(EES_PREFIX)/etc/bpmd.conf
        install -D -m 0755 $(@D)/startacsys-bpmd $(EES_PREFIX)/usr/local/products/acsysfe/bin/startacsys
        install -D -m 0755 $(@D)/S70bpmd $(EES_PREFIX)/etc/init.d/

install_test: all
        scp -q bpmd nova:/fecode-bd/vxworks_write/fe/ees/bbpmts
        scp -q bpmcli nova:/fecode-bd/vxworks_write/fe/ees/bbpmts
        scp -q boosterbpm_acsys nova:/fecode-bd/vxworks_write/fe/ees/bbpmts/
        scp -q boostercrate_acsys nova:/fecode-bd/vxworks_write/fe/ees/bbpmts/
        scp -q example-bpmd.conf nova:/fecode-bd/vxworks_write/fe/ees/bbpmts/bpmd.conf
        scp -q S70bpmd nova:/fecode-bd/vxworks_write/fe/ees/bbpmts

install_bbpmt2: all
        scp -q bpmd nova:/fecode-bd/vxworks_write/fe/ees/bbpmt2
        scp -q bpmcli nova:/fecode-bd/vxworks_write/fe/ees/bbpmt2
        scp -q boosterbpm_acsys nova:/fecode-bd/vxworks_write/fe/ees/bbpmt2
        scp -q boostercrate_acsys nova:/fecode-bd/vxworks_write/fe/ees/bbpmt2
        scp -q S70bpmd nova:/fecode-bd/vxworks_write/fe/ees/bbpmt2

Board/Architecture/OS Conditional Compilation

The ELSD tools make some pre-processor defines that you can use in your code to conditionally compile code based on the target configuration. Values for the pre-processor defines are in elsd_defines-latest.h.

Some useful defines are:

Name Values Description
ELSD_ARCH ELSD_ARCH_x86
ELSD_ARCH_POWERPC
ELSD_ARCH_ARM		 Target Architecture
ELSD_OS ELSD_OS_LINUX
ELSD_OS_VXWORKS		 Target Operating System
ELSD_BOARD ELSD_BOARD_CONC405X
ELSD_BOARD_MVME8100
ELSD_BOARD_MVME81XX
ELSD_BOARD_MVME8105
ELSD_BOARD_MVME5500
ELSD_BOARD_VMIVME7805		 Target Board

Changes to elsd.mk

The elsd.mk Makefile is tracked in the ees-buildroot project Git repository under the elsd/ directory. To deploy changes so other developers on adlinux can use them, use the install_elsd target provided in the Buildroot Makefile:

make install_elsd VERSION=<version>

The VERSION argument is required. If you omit it, a directory listing of the current elsd include directory will be displayed to remind you of the current version number.

Creating new Buildroot packages for Fermi software

You may want to consider making a Buildroot package for your software or for software out there that your application depends on.

Buildroot Package How-to

Remote Deployment

You can transfer new builds to your target using scp. Most targets live inside the controls firewall so in order to scp you must first configure ssh to proxy through outland.fnal.gov. Add the following to your ~/.ssh/config file:

Host target.fnal.gov
      ProxyCommand ssh user@outland.fnal.gov nc %h %p

Then to copy your_program to the target:

# scp -q your_program root@target.fnal.gov:/usr/local/products/bin/your_program

Keep in mind that the root filesystem is in a ramdisk, so the new file that you copy over will disappear after you reboot unless you rebuild and deploy your kernel+ramfs with the new changes. To deploy a new kernel, you can add an entry for nova.fnal.gov to your ~/.ssh/config file and copy the kernel to the linux_boot area:

# scp -q output/images/bzImage nova.fnal.gov:/fecode-bd/linux_boot/ees/project/bzImage.dev

Remote Debugging

You can debug executables on your target using GDB's remote features. Start the application with gdbserver:

$ gdbserver :2345 /usr/local/products/bin/your_program

Then, from the host that you built ees-buildroot on, establish a secure tunnel for GDB into the controls vlan:

$ ssh -f -N -L 2345:target.fnal.gov:2345 outland

Alternatively, you could log into the controls VPN and address your target directly.
Move into the build directory and connect to the target via the tunnel:

$ cd output/build/your_program
$ gdb your_program
(gdb) target remote localhost:2345

Other Documentation