5. QEMU TCG Tests

QEMU TCG Tests


  • Ever since I started working on QEMU, a small directory called tests/tcg has been in a perpetually broken state. It contains tests that exercise QEMU’s ability to work across architectures using the power of the Tiny Code Generator. However as these tests needed to be compiled for the guest architectures and not the host architecture—this is known as cross-compiling—most developers never ran them. As the tests were hardly ever built inevitably a certain amount of bit-rot set in.

    Cross Compilers

    In the old days, cross-compilation setups were almost all hand-crafted affairs which involved building versions of binutils, gcc and a basic libc. If you couldn’t get someone to give you a pre-built tarball, it was something you laboured through once and hopefully never had to touch again. There were even dedicated scripts like crosstool-ng which attempted to make the process of patching and configuring your toolchain easier.

    While the distributions have improved their support for cross compilers over the years, there are still plenty of variations in how they are deployed. It is hard for a project like QEMU which has to build on a wide range of operating systems and architectures to seamlessly use any given distributions compiler setup. However for those with cross compilers at hand configure now accepts two additional flags:

    --cross-cc-$(ARCH) --cross-cc-flags-$(ARCH)

    With a compiler specified for each guest architecture you want to test the build system can now build and run the tests. For developers that don’t have cross compilers around, they can take advantage of QEMU’s docker images.

    Enter Docker Containers

    If you work in IT you would be hard pressed not to have noticed the hype around Docker and the concept of containerisation over the last few years. Put simply containers allow you to define a known working set of software that gets run in an isolated environment for a given task. While this has many uses for QEMU it allows us to define build environments that any developer can run without having to mess around with their preferred host setup.

    Over the last few years QEMU’s build system has been expanding the number of docker images it supports. Most of this has been in service of our CI testing such as Patchew and Shippable but any developer with a docker setup can run the exact same images. For example if you want to check your patches won’t break when compiled on a 32 bit ARM system you can run:

    make docker-test-build@debian-armhf-cross J=n

    instead of tracking down a piece of ARM hardware to actually build on. Run make docker in your source tree to see the range of builds and tests it can support.

    make check-tcg

    With the latest work merged into master we can now take advantage of either hand-configured and Docker-based cross compilers to build test cases for TCG again. To run the TCG tests after you have built QEMU:

    make check-tcg

    and the build system will build and run all the tests it can for your configured targets.

    Rules for tests/tcg

    So now we have the infrastructure in place to add new tests what rules need to be followed to add new tests?

    Well the first thing to note is currently all the tests are for the linux-user variant of QEMU. This means the tests are all currently user-space tests that have access to the Linux syscall ABI.

    Another thing to note is the tests are separate from the rest of the QEMU test infrastructure. To keep things simple they are compiled as standalone “static” binaries. As the cross-compilation setup can be quite rudimentary for some of the rarer architectures we only compile against a standard libc. There is no support for linking to other libraries like for example glib. Thread and maths support is part of glibc so shouldn’t be a problem.

    Finally when writing new tests consider if it really is architecture specific or can be added to tests/tcg/multiarch. The multiarch tests are re-built for every supported architecture and should be the default place for anything that tests syscalls or other common parts of the code base.

    What’s next

    My hope with this work is we can start adding more tests to systematically defend functionality in linux-user. In fact I hope the first port of call to reproducing a crash would be writing a test case that can be added to our growing library of tests.

    Another thing that needs sorting out is getting toolchains for all of the less common architectures. The current work relies heavily on the excellent work of the Debian toolchain team in making multiarch aware cross compilers available in their distribution. However QEMU supports a lot more architectures than QEMU, some only as system emulations. In principle supporting them is as easy as adding another docker recipe but it might be these recipes end up having to compile the compilers from source.

    The tests/tcg directory still contains a number of source files we don’t build.

    The cris and openrisc directories contain user-space tests which just need the support of a toolchain and the relevant Makefile plumbing to be added.

    The lm32, mips and xtensa targets have a set of tests that need a system emulator. Aside from adding the compilers as docker images some additional work is needed to handle the differences between plain linux-user tests which can simply return an exit code to getting the results from a qemu-system emulation. Some architectures have semi-hosting support already for this while others report their test status over a simple serial link which will need to be parsed and handled in custom versions of the run-%: rule.

    /2018/06/21/tcg-testing/

Log in to reply
 

© Lightnetics 2024