> For the complete documentation index, see [llms.txt](https://weenslab.gitbook.io/pages/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://weenslab.gitbook.io/pages/fpga-tutorials/zynq-fpga-linux-kernel-module/cross-compiling-kernel-kernel-module-and-user-program-for-pynq.md).
# Cross Compiling Kernel, Kernel Module, and User Program for PYNQ
## Check PYNQ Version
First, we need to check the PYNQ version, architecture, and kernel version on the target board.
On the target board, check the PYNQ version:
```
xilinx@pynq:~/workspace$ pynq --version
PYNQ version: 3.0.1
Path: /usr/local/share/pynq-venv/lib/python3.10/site-packages/pynq
Git Id: 16022d5f2c61c7e5e1d4aabcfc9b3e4c91b491b6
```
Then, check the kernel version and CPU architecture:
```
xilinx@pynq:~/workspace$ uname -r -m
5.15.19-xilinx-v2022.1 armv7l
```
If it says `armv7l`, then it is 32-bit, but if it is `aarch64`, then it is 64-bit.
## Setup Build Machine
On the build machine, we need to install a cross-compilation toolchain and several tools. I am using Ubuntu 18.04.4 running on a virtual machine.
First, update the system and install the required tools.
```
sudo apt-get update
sudo apt-get install make build-essential libncurses-dev bison flex libssl-dev libelf-dev
```
Then, install the cross-compiler depending on your target board architecture.
Install this for 32-bit:
sudo apt-get install gcc-arm-linux-gnueabihf
Install this for 64-bit:
```
sudo apt-get install gcc-aarch64-linux-gnu
```
## Download the Kernel Source
For my PYNQ version, this kernel version is used:
So, download that via GitHub or git clone on the build machine.
## Get the Kernel Configuration
Now we compile the kernel with a configuration that is close to what was used on our target board. To do that, we need to extract configuration from the target board by reading the `config.gz`.
On the target board, do the following command:
```
xilinx@pynq:~$ cd /proc
xilinx@pynq:/proc$ zcat config.gz > ~/.config
xilinx@pynq:/proc$ cd ~
xilinx@pynq:~$ ls -a
. .bashrc .local .sudo_as_admin_successful
.. .cache .profile workspace
.bash_history .config pynq
.bash_logout jupyter_notebooks REVISION
```
We will use the `.config` file to configure and compile the kernel source that we have downloaded before.
Go to the following folder and get the `Module.symvers` file for later use when compiling the kernel.
```
xilinx@pynq:~$ cd /usr/src/kernel
xilinx@pynq:/usr/src/kernel$ ls
arch include mm System.map-5.15.19-xilinx-v2022.1
block init Module.symvers tools
certs ipc net usr
crypto Kconfig samples virt
Documentation kernel scripts
drivers lib security
fs Makefile sound
xilinx@pynq:/usr/src/kernel$
```
So, we have two files `.config` and `Module.symvres` that are required when we compile the kernel.
## Compile the Kernel
On your build machine, extract the kernel source file that we have downloaded, `linux-xlnx-xlnx_rebase_v5.15_LTS_2022.1_update.zip`.
Copy the following files to the main location of extracted folder:
* `.config`
* `Module.symvers`
We first need to set the `ARCH` and `CROSS_COMPILE` variables.
For 32-bit system:
```
export ARCH=arm
export CROSS_COMPILE=arm-linux-gnueabihf-
```
For 64-bit system:
```
export ARCH=arm64
export CROSS_COMPILE=aarch64-linux-gnu-
```
Run this command to read the existing `.config` file that was used for an old kernel. Then, it sets every option (that are not found in the file) to their default value without asking interactively.
```
make olddefconfig
```
Run this command to build the kernel for 32-bit system:
```
make -j8 uImage UIMAGE_LOADADDR=0x8000
```
Or for 64-bit system:
```
make -j8 Image UIMAGE_LOADADDR=0x8000
```
You will find the kernel image file in this location for 32-bit system:
```
linux-xlnx-xlnx_rebase_v5.15_LTS_2022.1_update32/arch/arm/boot
```
Or for 64-bit system:
linux-xlnx-xlnx_rebase_v5.15_LTS_2022.1_update32/arch/arm64/boot
Use `zImage` for 32-bit and `Image` for 64-bit.
## Convert Image File
PYNQ uses an `image.ub` file, so we need to convert the image file and devicetree to `image.ub`.
Install the required tools:
```
sudo apt-get install u-boot-tools
```
### Dump Image
Copy the original `image.ub` file from the target board to build the machine.
Use this command to list the content of `image.ub`:
```
dumpimage -l image.ub
```
The contents are the kernel (parition 0) and devicetree (parition 1).
```
erwin@vm:~/Workspace/image/$ dumpimage -l image.ub
FIT description: U-Boot fitImage for PYNQ arm kernel
Created: Tue Oct 15 23:49:29 2024
Image 0 (kernel-0)
Description: Linux Kernel
Created: Tue Oct 15 23:49:29 2024
Type: Kernel Image
Compression: uncompressed
Data Size: 6442720 Bytes = 6291.72 KiB = 6.14 MiB
Architecture: ARM
OS: Linux
Load Address: 0x00080000
Entry Point: 0x00080000
Hash algo: sha1
Hash value: b056acb229d05b806f975396dacae4ad04002e0c
Image 1 (fdt-0)
Description: Flattened Device Tree blob
Created: Tue Oct 15 23:49:29 2024
Type: Flat Device Tree
Compression: uncompressed
Data Size: 17130 Bytes = 16.73 KiB = 0.02 MiB
Architecture: ARM
Hash algo: sha1
Hash value: e14b2e9b9a54d8a045283bee083b9661164d207a
Default Configuration: 'conf-1'
Configuration 0 (conf-1)
Description: Boot Linux kernel with FDT blob
Kernel: kernel-0
FDT: fdt-0
Hash algo: sha1
Hash value: unavailable
```
To dump the kernel we can use the following command:
```
dumpimage -T flat_dt -p 0 image.ub -o zImage
```
To dump the devicetree we can use the following command:
dumpimage -T flat_dt -p 1 image.ub -o system.dtb
### Modify Devicetree
To convert the devicetree from `.dtb` to `.dts`, we can use the following command:
```
dtc -I dtb -O dts system.dtb -o system.dts
```
Open the devicetree source (`.dts`) using a text editor, modify the setting that you need, and compile it back to binary (`.dtb`).
To convert the devicetree from `.dts` back to `.dtb`, we can use the following command:
```
dtc -I dts -O dtb system.dts -o system.dtb
```
### Make Image
Download the `.its` from for 32-bit or for 64-bit.
Put the required files in one folder: `zImage`, `system.dtb`, `image.its`. Then, run the following command to build new `image.ub`.
```
mkimage -f image.its image.ub
```
## Compile the Kernel Module
On the build machine, create an example C kernel module file named `hello.c`:
```c
#include
#include
MODULE_LICENSE("GPL");
MODULE_AUTHOR("Erwin");
MODULE_VERSION("0.01");
int init_module(void)
{
printk(KERN_INFO "Hello!\n");
return 0;
}
void cleanup_module(void)
{
printk(KERN_INFO "Good Bye!\n");
}
```
Create a Makefile (must be named as this with capital M) for compiling the `hello.c`. Put this file on the same folder with `hello.c`.
This Makefile is for 32-bit system. Change the `KERNELDIR` to your respective location.
```makefile
ARCH = arm
COMPILER = arm-linux-gnueabihf-
obj-m := hello.o
KERNELDIR := /home/erwin/Workspace/linux-xlnx-xlnx_rebase_v5.15_LTS_2022.1_update
PWD := $(shell pwd)
default:
$(MAKE) -C $(KERNELDIR) M=$(PWD) ARCH=$(ARCH) CROSS_COMPILE=$(COMPILER) modules
clean:
$(MAKE) -C $(KERNELDIR) M=$(PWD) ARCH=$(ARCH) clean
```
This Makefile is for 64-bit system. Change the `KERNELDIR` to your respective location.
```makefile
ARCH = arm64
COMPILER = aarch64-linux-gnu-
obj-m := hello.o
KERNELDIR := /home/erwin/Workspace/linux-xlnx-xlnx_rebase_v5.15_LTS_2022.1_update
PWD := $(shell pwd)
default:
$(MAKE) -C $(KERNELDIR) M=$(PWD) ARCH=$(ARCH) CROSS_COMPILE=$(COMPILER) modules
clean:
$(MAKE) -C $(KERNELDIR) M=$(PWD) ARCH=$(ARCH) clean
```
Run the make command, and the result should be similar to this:
```
erwin@vm:~/Workspace/driver/hello$ make
make -C /home/erwin/Workspace/linux-xlnx-xlnx_rebase_v5.15_LTS_2022.1_update M=/home/erwin/Workspace/driver/hello ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- modules
make[1]: Entering directory '/home/erwin/Workspace/linux-xlnx-xlnx_rebase_v5.15_LTS_2022.1_update'
CC [M] /home/erwin/Workspace/driver/hello/hello.o
MODPOST /home/erwin/Workspace/driver/hello/Module.symvers
CC [M] /home/erwin/Workspace/driver/hello/hello.mod.o
LD [M] /home/erwin/Workspace/driver/hello/hello.ko
make[1]: Leaving directory '/home/erwin/Workspace/linux-xlnx-xlnx_rebase_v5.15_LTS_2022.1_update'
erwin@vm:~/Workspace/driver/hello$ ls
hello.c hello.ko hello.mod hello.mod.c hello.mod.o hello.o Makefile modules.order Module.symvers
```
You can find the `hello.ko`. Copy this file to your target board.
Run using `insmod` and check the result on the target board:
```
xilinx@pynq:~/workspace$ sudo insmod hello.ko
xilinx@pynq:~/workspace$ sudo lsmod
Module Size Used by
hello 16384 0
zocl 143360 0
uio_pdrv_genirq 16384 0
xilinx@pynq:~/workspace$ dmesg
[ 2908.449994] Hello!
xilinx@pynq:~/workspace$ sudo rmmod hello.ko
xilinx@pynq:~/workspace$ dmesg
[ 2908.449994] Hello!
[ 2933.774419] Good Bye!
```
If you find an error like this, then your compiled kernel version or configuration is different from the one that is running on your board.
```
xilinx@pynq:~/workspace$ sudo insmod hello.ko
insmod: ERROR: could not insert module hello.ko: Invalid module format
```
## Compile the User Program
Create the C user space program named `hello.c`.
```c
#include
int main()
{
printf("Hello, World!\n");
return 0;
}
```
Compile the code in 32-bit system.
```
erwin@vm:~/Workspace/user-app/hello$ arm-linux-gnueabihf-gcc -o hello hello.c
erwin@vm:~/Workspace/user-app/hello$ readelf -h hello
ELF Header:
Magic: 7f 45 4c 46 01 01 01 00 00 00 00 00 00 00 00 00
Class: ELF32
Data: 2's complement, little endian
Version: 1 (current)
OS/ABI: UNIX - System V
ABI Version: 0
Type: DYN (Shared object file)
Machine: ARM
Version: 0x1
Entry point address: 0x3fd
Start of program headers: 52 (bytes into file)
Start of section headers: 6996 (bytes into file)
Flags: 0x5000400, Version5 EABI, hard-float ABI
Size of this header: 52 (bytes)
Size of program headers: 32 (bytes)
Number of program headers: 9
Size of section headers: 40 (bytes)
Number of section headers: 29
Section header string table index: 28
```
Compile the code in 64-bit system.
```
erwin@vm:~/Workspace/user-app/hello$ aarch64-linux-gnu-gcc -o hello hello.c
erwin@vm:~/Workspace/user-app/hello$ readelf -h hello
ELF Header:
Magic: 7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00
Class: ELF64
Data: 2's complement, little endian
Version: 1 (current)
OS/ABI: UNIX - System V
ABI Version: 0
Type: DYN (Shared object file)
Machine: AArch64
Version: 0x1
Entry point address: 0x620
Start of program headers: 64 (bytes into file)
Start of section headers: 7352 (bytes into file)
Flags: 0x0
Size of this header: 64 (bytes)
Size of program headers: 56 (bytes)
Number of program headers: 8
Size of section headers: 64 (bytes)
Number of section headers: 27
Section header string table index: 26
```
On the target board, change the permission to `hello`, and rund the progam as root:
```
root@pynq:/home/xilinx/workspace# chmod +x hello
root@pynq:/home/xilinx/workspace# ./hello
Hello, World!
```
## References
* Cross compiling a kernel module,
* Cross compiling a kernel module,
* openwifi,
* openwifi,