Cross Compiling Kernel, Kernel Module, and User Program for PYNQ

Check PYNQ Version

First, we need to check the PYNQ version, architecture, and kernel version on the target board.

On the target board, check the PYNQ version:

xilinx@pynq:~/workspace$ pynq --version
PYNQ version: 3.0.1
Path: /usr/local/share/pynq-venv/lib/python3.10/site-packages/pynq
Git Id: 16022d5f2c61c7e5e1d4aabcfc9b3e4c91b491b6

Then, check the kernel version and CPU architecture:

xilinx@pynq:~/workspace$ uname -r -m
5.15.19-xilinx-v2022.1 armv7l

If it says armv7l, then it is 32-bit, but if it is aarch64, then it is 64-bit.

Setup Build Machine

On the build machine, we need to install a cross-compilation toolchain and several tools. I am using Ubuntu 18.04.4 running on a virtual machine.

First, update the system and install the required tools.

sudo apt-get update
sudo apt-get install make build-essential libncurses-dev bison flex libssl-dev libelf-dev

Then, install the cross-compiler depending on your target board architecture.

Install this for 32-bit:

sudo apt-get install gcc-arm-linux-gnueabihf

Install this for 64-bit:

sudo apt-get install gcc-aarch64-linux-gnu

Download the Kernel Source

For my PYNQ version, this kernel version is used: https://github.com/Xilinx/linux-xlnx/tree/xlnx_rebase_v5.15_LTS_2022.1_update

So, download that via GitHub or git clone on the build machine.

Get the Kernel Configuration

Now we compile the kernel with a configuration that is close to what was used on our target board. To do that, we need to extract configuration from the target board by reading the config.gz.

On the target board, do the following command:

xilinx@pynq:~$ cd /proc
xilinx@pynq:/proc$ zcat config.gz > ~/.config
xilinx@pynq:/proc$ cd ~
xilinx@pynq:~$ ls -a
.              .bashrc            .local    .sudo_as_admin_successful
..             .cache             .profile  workspace
.bash_history  .config            pynq
.bash_logout   jupyter_notebooks  REVISION

We will use the .config file to configure and compile the kernel source that we have downloaded before.

Go to the following folder and get the Module.symvers file for later use when compiling the kernel.

xilinx@pynq:~$ cd /usr/src/kernel
xilinx@pynq:/usr/src/kernel$ ls
arch           include   mm              System.map-5.15.19-xilinx-v2022.1
block          init      Module.symvers  tools
certs          ipc       net             usr
crypto         Kconfig   samples         virt
Documentation  kernel    scripts
drivers        lib       security
fs             Makefile  sound
xilinx@pynq:/usr/src/kernel$

So, we have two files .config and Module.symvres that are required when we compile the kernel.

Compile the Kernel

On your build machine, extract the kernel source file that we have downloaded, linux-xlnx-xlnx_rebase_v5.15_LTS_2022.1_update.zip.

Copy the following files to the main location of extracted folder:

• .config

• Module.symvers

We first need to set the ARCH and CROSS_COMPILE variables.

For 32-bit system:

export ARCH=arm
export CROSS_COMPILE=arm-linux-gnueabihf-

For 64-bit system:

export ARCH=arm64
export CROSS_COMPILE=aarch64-linux-gnu-

Run this command to read the existing .config file that was used for an old kernel. Then, it sets every option (that are not found in the file) to their default value without asking interactively.

make olddefconfig

Run this command to build the kernel for 32-bit system:

make -j8 uImage UIMAGE_LOADADDR=0x8000

Or for 64-bit system:

make -j8 Image UIMAGE_LOADADDR=0x8000

You will find the kernel image file in this location for 32-bit system:

linux-xlnx-xlnx_rebase_v5.15_LTS_2022.1_update32/arch/arm/boot

Or for 64-bit system:

linux-xlnx-xlnx_rebase_v5.15_LTS_2022.1_update32/arch/arm64/boot

Use zImage for 32-bit and Image for 64-bit.

Convert Image File

PYNQ uses an image.ub file, so we need to convert the image file and devicetree to image.ub.

Install the required tools:

sudo apt-get install u-boot-tools

Dump Image

Copy the original image.ub file from the target board to build the machine.

Use this command to list the content of image.ub:

dumpimage -l image.ub

The contents are the kernel (parition 0) and devicetree (parition 1).

erwin@vm:~/Workspace/image/$ dumpimage -l image.ub
FIT description: U-Boot fitImage for PYNQ arm kernel
Created:         Tue Oct 15 23:49:29 2024
 Image 0 (kernel-0)
  Description:  Linux Kernel
  Created:      Tue Oct 15 23:49:29 2024
  Type:         Kernel Image
  Compression:  uncompressed
  Data Size:    6442720 Bytes = 6291.72 KiB = 6.14 MiB
  Architecture: ARM
  OS:           Linux
  Load Address: 0x00080000
  Entry Point:  0x00080000
  Hash algo:    sha1
  Hash value:   b056acb229d05b806f975396dacae4ad04002e0c
 Image 1 (fdt-0)
  Description:  Flattened Device Tree blob
  Created:      Tue Oct 15 23:49:29 2024
  Type:         Flat Device Tree
  Compression:  uncompressed
  Data Size:    17130 Bytes = 16.73 KiB = 0.02 MiB
  Architecture: ARM
  Hash algo:    sha1
  Hash value:   e14b2e9b9a54d8a045283bee083b9661164d207a
 Default Configuration: 'conf-1'
 Configuration 0 (conf-1)
  Description:  Boot Linux kernel with FDT blob
  Kernel:       kernel-0
  FDT:          fdt-0
  Hash algo:    sha1
  Hash value:   unavailable

To dump the kernel we can use the following command:

dumpimage -T flat_dt -p 0 image.ub -o zImage

To dump the devicetree we can use the following command:

dumpimage -T flat_dt -p 1 image.ub -o system.dtb

Modify Devicetree

To convert the devicetree from .dtb to .dts, we can use the following command:

dtc -I dtb -O dts system.dtb -o system.dts

Open the devicetree source (.dts) using a text editor, modify the setting that you need, and compile it back to binary (.dtb).

To convert the devicetree from .dts back to .dtb, we can use the following command:

dtc -I dts -O dtb system.dts -o system.dtb

Make Image

Download the .its from https://github.com/Xilinx/PYNQ/blob/master/sdbuild/boot/image_arm.its for 32-bit or https://github.com/Xilinx/PYNQ/blob/master/sdbuild/boot/image_aarch64.its for 64-bit.

Put the required files in one folder: zImage, system.dtb, image.its. Then, run the following command to build new image.ub.

mkimage -f image.its image.ub

Compile the Kernel Module

On the build machine, create an example C kernel module file named hello.c:

#include <linux/module.h>
#include <linux/kernel.h>

MODULE_LICENSE("GPL");
MODULE_AUTHOR("Erwin");
MODULE_VERSION("0.01");

int init_module(void)
{
	printk(KERN_INFO "Hello!\n");
	return 0;
}

void cleanup_module(void)
{
	printk(KERN_INFO "Good Bye!\n");
}

Create a Makefile (must be named as this with capital M) for compiling the hello.c. Put this file on the same folder with hello.c.

This Makefile is for 32-bit system. Change the KERNELDIR to your respective location.

ARCH = arm
COMPILER = arm-linux-gnueabihf-
obj-m := hello.o
KERNELDIR := /home/erwin/Workspace/linux-xlnx-xlnx_rebase_v5.15_LTS_2022.1_update
PWD := $(shell pwd)

default:
	$(MAKE) -C $(KERNELDIR) M=$(PWD) ARCH=$(ARCH) CROSS_COMPILE=$(COMPILER) modules

clean:
	$(MAKE) -C $(KERNELDIR) M=$(PWD) ARCH=$(ARCH) clean

This Makefile is for 64-bit system. Change the KERNELDIR to your respective location.

ARCH = arm64
COMPILER = aarch64-linux-gnu-
obj-m := hello.o
KERNELDIR := /home/erwin/Workspace/linux-xlnx-xlnx_rebase_v5.15_LTS_2022.1_update
PWD := $(shell pwd)

default:
	$(MAKE) -C $(KERNELDIR) M=$(PWD) ARCH=$(ARCH) CROSS_COMPILE=$(COMPILER) modules

clean:
	$(MAKE) -C $(KERNELDIR) M=$(PWD) ARCH=$(ARCH) clean

Run the make command, and the result should be similar to this:

erwin@vm:~/Workspace/driver/hello$ make
make -C /home/erwin/Workspace/linux-xlnx-xlnx_rebase_v5.15_LTS_2022.1_update M=/home/erwin/Workspace/driver/hello ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- modules
make[1]: Entering directory '/home/erwin/Workspace/linux-xlnx-xlnx_rebase_v5.15_LTS_2022.1_update'
  CC [M]  /home/erwin/Workspace/driver/hello/hello.o
  MODPOST /home/erwin/Workspace/driver/hello/Module.symvers
  CC [M]  /home/erwin/Workspace/driver/hello/hello.mod.o
  LD [M]  /home/erwin/Workspace/driver/hello/hello.ko
make[1]: Leaving directory '/home/erwin/Workspace/linux-xlnx-xlnx_rebase_v5.15_LTS_2022.1_update'
erwin@vm:~/Workspace/driver/hello$ ls
hello.c  hello.ko  hello.mod  hello.mod.c  hello.mod.o  hello.o  Makefile  modules.order  Module.symvers

You can find the hello.ko. Copy this file to your target board.

Run using insmod and check the result on the target board:

xilinx@pynq:~/workspace$ sudo insmod hello.ko
xilinx@pynq:~/workspace$ sudo lsmod
Module                  Size  Used by
hello                  16384  0
zocl                  143360  0
uio_pdrv_genirq        16384  0
xilinx@pynq:~/workspace$ dmesg
[ 2908.449994] Hello!
xilinx@pynq:~/workspace$ sudo rmmod hello.ko
xilinx@pynq:~/workspace$ dmesg
[ 2908.449994] Hello!
[ 2933.774419] Good Bye!

If you find an error like this, then your compiled kernel version or configuration is different from the one that is running on your board.

xilinx@pynq:~/workspace$ sudo insmod hello.ko
insmod: ERROR: could not insert module hello.ko: Invalid module format

Compile the User Program

Create the C user space program named hello.c.

#include <stdio.h>

int main()
{
	printf("Hello, World!\n");
	return 0;
}

Compile the code in 32-bit system.

erwin@vm:~/Workspace/user-app/hello$ arm-linux-gnueabihf-gcc -o hello hello.c
erwin@vm:~/Workspace/user-app/hello$ readelf -h hello
ELF Header:
  Magic:   7f 45 4c 46 01 01 01 00 00 00 00 00 00 00 00 00 
  Class:                             ELF32
  Data:                              2's complement, little endian
  Version:                           1 (current)
  OS/ABI:                            UNIX - System V
  ABI Version:                       0
  Type:                              DYN (Shared object file)
  Machine:                           ARM
  Version:                           0x1
  Entry point address:               0x3fd
  Start of program headers:          52 (bytes into file)
  Start of section headers:          6996 (bytes into file)
  Flags:                             0x5000400, Version5 EABI, hard-float ABI
  Size of this header:               52 (bytes)
  Size of program headers:           32 (bytes)
  Number of program headers:         9
  Size of section headers:           40 (bytes)
  Number of section headers:         29
  Section header string table index: 28

Compile the code in 64-bit system.

erwin@vm:~/Workspace/user-app/hello$ aarch64-linux-gnu-gcc -o hello hello.c
erwin@vm:~/Workspace/user-app/hello$ readelf -h hello
ELF Header:
  Magic:   7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00 
  Class:                             ELF64
  Data:                              2's complement, little endian
  Version:                           1 (current)
  OS/ABI:                            UNIX - System V
  ABI Version:                       0
  Type:                              DYN (Shared object file)
  Machine:                           AArch64
  Version:                           0x1
  Entry point address:               0x620
  Start of program headers:          64 (bytes into file)
  Start of section headers:          7352 (bytes into file)
  Flags:                             0x0
  Size of this header:               64 (bytes)
  Size of program headers:           56 (bytes)
  Number of program headers:         8
  Size of section headers:           64 (bytes)
  Number of section headers:         27
  Section header string table index: 26

On the target board, change the permission to hello, and rund the progam as root:

root@pynq:/home/xilinx/workspace# chmod +x hello
root@pynq:/home/xilinx/workspace# ./hello
Hello, World!

References

• Cross compiling a kernel module, https://embear.ch/blog/compiling-a-kernel-module

• Cross compiling a kernel module, https://stackoverflow.com/questions/3467850/cross-compiling-a-kernel-module

• openwifi, https://github.com/open-sdr/openwifi/blob/master/user_space/prepare_kernel.sh

• openwifi, https://github.com/open-sdr/openwifi/blob/master/driver/make_all.sh