Cross-Compiling

FRR is capable of being cross-compiled to a number of different architectures. With an adequate toolchain this process is fairly straightforward, though one must exercise caution to validate this toolchain’s correctness before attempting to compile FRR or its dependencies; small oversights in the construction of the build tools may lead to problems which quickly become difficult to diagnose.

Toolchain Preliminary

The first step to cross-compiling any program is to identify the system which the program (FRR) will run on. From here on this will be called the “host” machine, following autotools’ convention, while the machine building FRR will be called the “build” machine. The toolchain will of course be installed onto the build machine and be leveraged to build FRR for the host machine to run.

Note

The build machine used while writing this guide was x86_64-pc-linux-gnu and the target machine was arm-linux-gnueabihf (a Raspberry Pi 3B+). Replace this with your targeted tuple below if you plan on running the commands from this guide:

export HOST_ARCH=”arm-linux-gnueabihf”

For your given target, the build system’s OS may have some support for building cross compilers natively, or may even offer binary toolchains built upstream for the target architecture. Check your package manager or OS documentation before committing to building a toolchain from scratch.

This guide will not detail how to build a cross-compiling toolchain but will instead assume one already exists and is installed on the build system. The methods for building the toolchain itself may differ between operating systems so consult the OS documentation for any particulars regarding cross-compilers. The OSDev wiki has a pleasant tutorial on cross-compiling in the context of operating system development which bootstraps from only the native GCC and binutils on the build machine. This may be useful if the build machine’s OS does not offer existing tools to build a cross-compiler targeting the host.

This guide will also not demonstrate how to build all of FRR’s dependencies for the target architecture. Instead, general instructions for using a cross-compiling toolchain to compile packages using CMake, Autotools, and Makefiles are provided; these three cases apply to almost all FRR dependencies.

Warning

Ensure the versions and implementations of the C standard library (glibc or what have you) match on the host and the build toolchain. ldd --version will help you here. Upgrade one or the other if the they do not match.

Testing the Toolchain

Before any cross-compilation begins it would be prudent to test the new toolchain by writing, compiling and linking a simple program.

# A small program
cat > nothing.c <<EOF
int main() { return 0; }
EOF

# Build and link with the cross-compiler
${HOST_ARCH}-gcc -o nothing nothing.c

# Inspect the resulting binary, results may vary
file ./nothing

# nothing: ELF 32-bit LSB pie executable, ARM, EABI5 version 1 (SYSV),
# dynamically linked, interpreter /lib/ld-linux-armhf.so.3,
# for GNU/Linux 3.2.0, not stripped

If this produced no errors then the installed toolchain is probably ready to start compiling the build dependencies and eventually FRR itself. There still may be lurking issues but fundamentally the toolchain can produce binaries and that’s good enough to start working with it.

Warning

If any errors occurred during the previous functional test please look back and address them before moving on; this indicates your cross-compiling toolchain is not in a position to build FRR or its dependencies. Even if everything was fine, keep in mind that many errors from here on may still be related to your toolchain (e.g. libstdc++.so or other components) and this small test is not a guarantee of complete toolchain coherence.

Cross-compiling Dependencies

When compiling FRR it is necessary to compile some of its dependencies alongside it on the build machine. This is so symbols from the shared libraries (which will be loaded at run-time on the host machine) can be linked to the FRR binaries at compile time; additionally, headers for these libraries are needed during the compile stage for a successful build.

Sysroot Overview

All build dependencies should be installed into a “root” directory on the build computer, hereafter called the “sysroot”. This directory will be prefixed to paths while searching for requisite libraries and headers during the build process. Often this may be set via a --prefix flag when building the dependent packages, meaning a make install will copy compiled libraries into (e.g.) /usr/${HOST_ARCH}/usr.

If the toolchain was built on the build machine then there is likely already a sysroot where those tools and standard libraries were installed; it may be helpful to use that directory as the sysroot for this build as well.

Basic Workflow

Before compiling or building any dependencies, make note of which daemons are being targeted and which libraries will be needed. Not all dependencies are necessary if only building with a subset of the daemons.

The following workflow will compile and install any libraries which can be built with Autotools. The resultant library will be installed into the sysroot /usr/${HOST_ARCH}.

./configure \
   CC=${HOST_ARCH}-gcc \
   CXX=${HOST_ARCH}-g++ \
   --build=${HOST_ARCH} \
   --prefix=/usr/${HOST_ARCH}
make
make install

Some libraries like json-c and libyang are packaged with CMake and can be built and installed generally like:

mkdir build
cd build
CC=${HOST_ARCH}-gcc \
CXX=${HOST_ARCH}-g++ \
cmake \
    -DCMAKE_INSTALL_PREFIX=/usr/${HOST_ARCH} \
    ..
make
make install

For programs with only a Makefile (e.g. libcap) the process may look still a little different:

CC=${HOST_ARCH}-gcc make
make install DESTDIR=/usr/${HOST_ARCH}

These three workflows should handle the bulk of building and installing the build-time dependencies for FRR. Verify that the installed files are being placed correctly into the sysroot and were actually built using the cross-compile toolchain, not by the native toolchain by accident.

Dependency Notes

There are a lot of things that can go wrong during a cross-compilation. Some of the more common errors and a few special considerations are collected below for reference.

libyang

-DENABLE_LYD_PRIV=ON should be provided during the CMake step.

Ensure also that the version of libyang being installed corresponds to the version required by the targeted FRR version.

gRPC

This piece is requisite only if the --enable-grpc flag will be passed later on to FRR. One may get burned when compiling gRPC if the protoc version on the build machine differs from the version of protoc being linked to during a gRPC build. The error messages from this defect look like:

gens/src/proto/grpc/channelz/channelz.pb.h: In member function ‘void grpc::channelz::v1::ServerRef::set_name(const char*, size_t)’:
gens/src/proto/grpc/channelz/channelz.pb.h:9127:64: error: ‘EmptyDefault’ is not a member of ‘google::protobuf::internal::ArenaStringPtr’
 9127 |   name_.Set(::PROTOBUF_NAMESPACE_ID::internal::ArenaStringPtr::EmptyDefault{}, ::std::string(

This happens because protocol buffer code generation uses protoc to create classes with different getters and setters corresponding to the protobuf data defined by the source tree’s .proto files. Clearly the cross-compiled protoc cannot be used for this code generation because that binary is built for a different CPU.

The solution is to install matching versions of native and cross-compiled protocol buffers; this way the native binary will generate code and the cross-compiled library will be linked to by gRPC and these versions will not disagree.

The -latomic linker flag may also be necessary here if using libstdc++ since GCC’s C++11 implementation makes library calls in certain cases for <atomic> so -latomic cannot be assumed.

Cross-compiling FRR Itself

With all the necessary libraries cross-compiled and installed into the sysroot, the last thing to actually build is FRR itself:

# Clone and bootstrap the build
git clone 'https://github.com/FRRouting/frr.git'
# (e.g.) git checkout stable/7.5
./bootstrap.sh

# Build clippy using the native toolchain
mkdir build-clippy
cd build-clippy
../configure --enable-clippy-only
make clippy-only
cd ..

# Next, configure FRR and use the clippy we just built
./configure \
   CC=${HOST_ARCH}-gcc \
   CXX=${HOST_ARCH}-g++ \
   --host=${HOST_ARCH} \
   --with-sysroot=/usr/${HOST_ARCH} \
   --with-clippy=./build-clippy/lib/clippy \
   --sysconfdir=/etc \
   --localstatedir=/var \
   --sbindir="\${prefix}/lib/frr" \
   --prefix=/usr \
   --enable-user=frr \
   --enable-group=frr \
   --enable-vty-group=frrvty \
   --disable-doc \
   --enable-grpc

# Send it
make

Installation to Host Machine

If no errors were observed during the previous steps it is safe to make install FRR into its own directory.

# Install FRR its own "sysroot"
make install DESTDIR=/some/path/to/sysroot

After running the above command, FRR binaries, modules and example configuration files will be installed into some path on the build machine. The directory will have folders like /usr and /etc; this “root” should now be copied to the host and installed on top of the root directory there.

# Tar this sysroot (preserving permissions)
tar -C /some/path/to/sysroot -cpvf frr-${HOST_ARCH}.tar .

# Transfer tar file to host machine
scp frr-${HOST_ARCH}.tar me@host-machine:

# Overlay the tarred sysroot on top of the host machine's root
ssh me@host-machine <<-EOF
   # May need to elevate permissions here
   tar -C / -xpvf frr-${HOST_ARCH}.tar.gz .
EOF

Now FRR should be installed just as if make install had been run on the host machine. Create configuration files and assign permissions as needed. Lastly, ensure the correct users and groups exist for FRR on the host machine.

Troubleshooting

Even when every precaution has been taken some things may still go wrong! This section details some common runtime problems.

Mismatched Libraries

If you see something like this after installing on the host:

/usr/lib/frr/zebra: error while loading shared libraries: libyang.so.1: cannot open shared object file: No such file or directory

… at least one of FRR’s dependencies which was linked to the binary earlier is not available on the host OS. Even if it has been installed the host repository’s version may lag what is needed for more recent FRR builds (this is likely to happen with YANG at the moment).

If the matching library is not available from the host OS package manager it may be possible to compile them using the same toolchain used to compile FRR. The library may have already been built earlier when compiling FRR on the build machine, in which case it may be as simple as following the same workflow laid out during the Installation to Host Machine.

Mismatched Glibc Versions

The version and implementation of the C standard library must match on both the host and build toolchain. The error corresponding to this misconfiguration will look like:

/usr/lib/frr/zebra: /lib/${HOST_ARCH}/libc.so.6: version `GLIBC_2.32' not found (required by /usr/lib/libfrr.so.0)

See the earlier warning about preventing a glibc mismatch.