Yesterday we’ve discussed one guy’s problem in the chat. He had a device based on old ARM SoC with installed Linux on it. His target was to launch his own Qt application on it, however the Qt Framework on his device was too old.
He asked the device vendor for cross-toolchain, kernel sources or at
least instruction for building rootfs but his request was not
satisified. Hopefully, at least there was
gcc compiler on the
device. In such case the option of building newer Qt Framework
“in-place” on the device is not so bad.
Of course, cross-toolchain is the best solution for this problem. But sometimes you don’t have anything else but the working device with the operating system on SD card. It’s almost miraculous if you also have a compiler on this system.
However, you can’t build fat software on the low-end ARM system, it will take days to build something like Qt Framework. Therefore, in this case you have only two options:
build your own cross-toolchain using the appropriate
Linux kernelversions in accordance with the device’s rootfs;
dump the rootfs of the device and launch the building process inside it on your host system.
I want to discuss here the second option. So, in this case you can
make virtual machine with full virtualization using
and put the dumped rootfs and kernel into it. For full virtualization
you will emulate ARM processor and all needed hardware and run kernel
code on it along with the code of all drivers, file systems and so
on. It requires too many resources of your host system and still it
will work very slow.
However, there is another way to do it: you can emulate only ARM
instructions for syscalls and “convert” it to the x86 syscalls. For
this purpose there are several
qemu utilties for all popular
CPU architectures. In order to test it you can download any
statically compiled ARM binary for Linux (e.g.
busybox-armv7) and launch it on your host system
Moreover, you can
chroot into your ARM rootfs if you put
qemu-arm-static into this rootfs.
qemu-arm-static should be
statically compiled because it can’t access to your host libraries in
runtime while being in chroot. Usually, there is
package in your Linux repository (at least there are in Arch Linux,
Alpine Linux and all Debian derivatives).
So, if you try to chroot into it, the
/bin/bash compiled for ARM
systems will be launched via
qemu-arm-static on your host x86
system. It’s just a miracle!
In addition, in chroot by default you will have an access to all the CPU cores, all available RAM and disk space.
I’ve compared full virtualization performance with performance of
syscalls-only emulating. For this I run Qt 5.15.0
for the both variants.
./configure took much time because in the
process of configuring it also builds
time ./configure -skip qt3d -no-warnings-are-errors -release \ -recheck-all -opensource \ -confirm-license -nomake examples -nomake tests \ -c++std c++17 -I /usr/include/xcb/ -xcb-xlib -xcb \ -feature-thread -feature-xkbcommon -qt-libpng -qt-libjpeg \ -qt-zlib -I /usr/include/xcb/ --recheck-all -skip wayland \ -skip qtwebengine -skip qtwayland ... real 91m49.471s user 78m43.608s sys 7m22.934s
time ./configure -skip qt3d -no-warnings-are-errors -release \ -recheck-all -opensource \ -confirm-license -nomake examples -nomake tests \ -c++std c++17 -I /usr/include/xcb/ -xcb-xlib -xcb \ -feature-thread -feature-xkbcommon -qt-libpng -qt-libjpeg \ -qt-zlib -I /usr/include/xcb/ --recheck-all -skip wayland \ -skip qtwebengine -skip qtwayland ... real 17m22.799s user 16m59.336s sys 0m23.783s
As you see, the second variant was faster by the factor of 5 (17
minutes vs 91 minutes) in comparison with the first one. Both
variants used only one CPU core. Of course, it can be highly improved
if we make
./configure use all the CPU cores while building
Despite this improvement, 17 minutes is still very slow, if we
compare it with the native build: the same test on x86-based system
took only 2 minutes.
If you are interested, you can try this trick by yourself. This is a
small instruction for getting chroot’ed into ARM rootfs on the example
Raspberry Pi OS:
Raspberry Pi OSlite image and unzip it:
wget -O image.zip http://downloads.raspberrypi.org/raspios_lite_armhf_latest unzip image.zip rm -f image.zip
Extract rootfs from OS image:
sudo su modprobe loop losetup -fP --show 2020-05-27-raspios-buster-lite-armhf.img mount /dev/loop0p2 /mnt cd /mnt cp -r * /home/xxx/rootfs
Install additional packages on your host system (it is for Debian; however, for Arch Linux there is
qemu-user-staticpackage in AUR):
sudo apt-get install qemu-user-static qemu-user-static
qemu-arm-staticinto target rootfs:
cp "$(which qemu-arm-static)" /home/xxx/rootfs/usr/bin
Mount dev, sys, proc, run, tmp into target rootfs (don’t forget to unmount it afterwards):
mount --bind /dev /home/xxx/rootfs/dev mount --bind /sys /home/xxx/rootfs/sys mount --bind /proc /home/xxx/rootfs/proc mount -t tmpfs tmp /home/xxx/rootfs/tmp mount -t tmpfs run /home/xxx/rootfs/run
Chroot into target rootfs:
chroot /home/xxx/rootfs/ export PATH=/bin:/sbin:/usr/bin:/usr/sbin echo "nameserver 188.8.131.52" > /etc/resolv.conf
Success! Now you are chroot’ed in ARM-based rootfs. Good luck!