Add a runner for libtock2 examples. #365

jrvanwhy · 2022-01-21T19:59:48Z

This replaces tools/flash.sh. In the future, it will have functionality to process test output, and will therefore replace test_runner as well.

It can currently deploy to HiFive1 on QEMU, and has untested logic to deploy via tockloader.

It is no longer possible to deploy Tock 1.0 libtock-rs apps using cargo run, as cargo run will now invoke the new tool. This prevents us from running the Tock 1.0 libtock-rs integration test, so I have removed it from make test.

I also make the following changes to the submodule setup:

I added a setup-qemu-2 target to build QEMU in the tock2/ submodule. When I remove the tock/ submodule, I'll also remove the setup-qemu target and rename setup-qemu-2 to setup-qemu to replace it.
I updated the tock2/ submodule to a newer kernel. Because libtock-rs depends on the new Allow API implementation that is not in a released Tock version, I used the latest master branch commit.

This replaces tools/flash.sh. In the future, it will have functionality to process test output, and will therefore replace test_runner as well. It can currently deploy to Hifive1 on QEMU, and has untested logic to deploy via `tockloader`. It is no longer possible to use `cargo run` to deploy a Tock 1.0 libtock-rs process binary, as that will now use the Tock 2.0 runner. Additional changes: 1. added a `setup-qemu-2` build target to build the tock2 submodule's QEMU module. I intend to rename this to `setup-qemu` after the Tock 1.0 targets and submodules are removed. 2. I updated the tock2 submodule to a newer kernel. Because the new libtock-rs crates depend on post-2.0 Allow API guarantees, I updated to latest master.

jrvanwhy · 2022-01-21T21:07:26Z

Oh, hang on a bit while I debug the GitHub Actions workflows live...

jrvanwhy · 2022-01-21T21:11:23Z

Oh, I think the --help output provides a good overview of what the tool can do, so here it is:

runner 
Converts ELF binaries into Tock Binary Format binaries and runs them on a Tock system

USAGE:
    runner [OPTIONS] <ELF>

ARGS:
    <ELF>    The executable to convert into Tock Binary Format and run

OPTIONS:
    -d, --deploy <DEPLOY>    Where to deploy the process binary. If not specified, runner will only
                             make a TBF file and not attempt to run it [possible values: qemu,
                             tockloader]
    -h, --help               Print help information
    -v, --verbose            Whether to output verbose debugging information to the console

Also, here is an example invocation of the tool:

jrvanwhy@penguin:~/libtock-rs$ make qemu-example EXAMPLE=low_level_debug
make -C tock2/boards/hifive1 \
        /home/jrvanwhy/libtock-rs/tock2/target/riscv32imac-unknown-none-elf/release/hifive1.elf
make[1]: Entering directory '/home/jrvanwhy/libtock-rs/tock2/boards/hifive1'
    Finished release [optimized + debuginfo] target(s) in 0.07s
   text    data     bss     dec     hex filename
  66916      12    6128   73056   11d60 /home/jrvanwhy/libtock-rs/tock2/target/riscv32imac-unknown-none-elf/release/hifive1
make[1]: Leaving directory '/home/jrvanwhy/libtock-rs/tock2/boards/hifive1'
LIBTOCK_PLATFORM="hifive1" cargo run --example "low_level_debug" -p libtock2 \
        --release --target=riscv32imac-unknown-none-elf -- --deploy qemu
    Finished release [optimized + debuginfo] target(s) in 0.25s
     Running `cargo run -p runner --release target/riscv32imac-unknown-none-elf/release/examples/low_level_debug --deploy qemu`
    Finished release [optimized + debuginfo] target(s) in 0.04s
     Running `target/release/runner target/riscv32imac-unknown-none-elf/release/examples/low_level_debug --deploy qemu`
HiFive1 initialization complete.
Entering main loop.
LowLevelDebug: App 0x0 prints 0x1
LowLevelDebug: App 0x0 prints 0x2 0x3
^Cqemu-system-riscv32: terminating on signal 2
make: *** [Makefile:81: qemu-example] Interrupt

jrvanwhy@penguin:~/libtock-rs$

runner/src/qemu.rs

jrvanwhy · 2022-01-21T21:31:02Z

Update on the CI: Because cargo run no longer invokes flash.sh, libtock_test is not being built, resulting in test failures.

This didn't occur on my system because I already had a working libtock_test TBF file in my target/ directory, and the libtock-rs 1.0 tests were picking up that binary.

I'm going so see how much I need to disable to get CI to pass.

…out having `flash.sh` as the runner.

jrvanwhy · 2022-01-21T21:48:52Z

I needed to disable the Tock 1.0 libtock-rs integration test in order for make test to work.

I'd prefer to keep those tests disabled rather than add a hack to support both.

alexandruradovici · 2022-01-22T11:16:36Z

I'm not sure of this is the right place for this questions, but what about having the possibility to specify where the app should be loaded? Right now the runner assumes the app will be the only one (or the first) app running on the device.

For this I assume that this utility should write a temporary linker file before building the app.

jrvanwhy · 2022-01-22T17:53:21Z

I'm not sure of this is the right place for this questions, but what about having the possibility to specify where the app should be loaded? Right now the runner assumes the app will be the only one (or the first) app running on the device.

For this I assume that this utility should write a temporary linker file before building the app.

This runner runs after cargo builds the ELF file, which means it is too late for it to write the process binary's linker script. Changing this tool to write out the linker script and then invoke cargo would effectively make this tool become libtock-rs' build system, which adds more complexity than I would like to add.

alexandruradovici · 2022-01-23T12:18:50Z

This runner runs after cargo builds the ELF file, which means it is too late for it to write the process binary's linker script. Changing this tool to write out the linker script and then invoke cargo would effectively make this tool become libtock-rs' build system, which adds more complexity than I would like to add.

Do you see this dynamic layout generation as a separate tool that would be run by build.rs maybe?

jrvanwhy · 2022-01-23T17:15:12Z

This runner runs after cargo builds the ELF file, which means it is too late for it to write the process binary's linker script. Changing this tool to write out the linker script and then invoke cargo would effectively make this tool become libtock-rs' build system, which adds more complexity than I would like to add.

Do you see this dynamic layout generation as a separate tool that would be run by build.rs maybe?

My initial thought is yes, but upon further thought I realized that may exacerbate a race condition we may have: #366. So I agree with "maybe".

runner/src/qemu.rs

Makefile

runner/src/elf2tab.rs

alexandruradovici · 2022-01-24T11:31:43Z

Do you see this dynamic layout generation as a separate tool that would be run by build.rs maybe?

My initial thought is yes, but upon further thought I realized that may exacerbate a race condition we may have: #366. So I agree with "maybe".

I can try sketching something to build binaries for multiple slots and we can start experimenting from there. My question sis how is a slot defined? Would it be something like RAM_BASE and FLASH_BASE and (maybe) with their assigned length?

jrvanwhy · 2022-01-24T17:47:31Z

Do you see this dynamic layout generation as a separate tool that would be run by build.rs maybe?

My initial thought is yes, but upon further thought I realized that may exacerbate a race condition we may have: #366. So I agree with "maybe".

I can try sketching something to build binaries for multiple slots and we can start experimenting from there. My question sis how is a slot defined? Would it be something like RAM_BASE and FLASH_BASE and (maybe) with their assigned length?

I don't know how a slot should be defined, that would be part of the design for the slotted layout integration.

hudson-ayers · 2022-01-25T19:41:58Z

I took a stab at testing the tockloader version of this and was unable to get it to work:

(.venv) hudson: ~/libtock-rs (runner) $ LIBTOCK_PLATFORM="nrf52840" cargo run --example "low_level_debug" -p libtock2 --release --target=thumbv7em-none-eabi -- --deploy tockloader --verbose
    Finished release [optimized + debuginfo] target(s) in 0.03s
     Running `cargo run -p runner --release target/thumbv7em-none-eabi/release/examples/low_level_debug --deploy tockloader --verbose`
    Finished release [optimized + debuginfo] target(s) in 0.03s
     Running `target/release/runner target/thumbv7em-none-eabi/release/examples/low_level_debug --deploy tockloader --verbose`
Package name: "low_level_debug"
TAB path: target/thumbv7em-none-eabi/release/examples/low_level_debug.tab
Protected region size: 72
Found .stack section, size: 256
ELF file: "target/thumbv7em-none-eabi/release/examples/low_level_debug"
TBF path: target/thumbv7em-none-eabi/release/examples/low_level_debug.tbf
elf2tab command: "elf2tab" "--kernel-major" "2" "--kernel-minor" "0" "-n" "low_level_debug" "-o" "target/thumbv7em-none-eabi/release/examples/low_level_debug.tab" "--protected-region-size" "72" "--stack" "256" "target/thumbv7em-none-eabi/release/examples/low_level_debug" "-v"
Spawning elf2tab
Creating "target/thumbv7em-none-eabi/release/examples/low_level_debug.tbf"
Min RAM size from segments in ELF: 0 bytes
Number of writeable flash regions: 0
Kernel version: 2.0
Entry point is in .start section
  Adding .start section. Offset: 72 (0x48). Length: 110 (0x6e) bytes.
  Adding 2 bytes of padding between sections
  Adding .text section. Offset: 184 (0xb8). Length: 48 (0x30) bytes.
Searching for .rel.X sections to add.
TBF Header:
               version:        2        0x2
           header_size:       72       0x48
            total_size:      512      0x200
                 flags:        1        0x1

        init_fn_offset:       33       0x21
        protected_size:        0        0x0
      minimum_ram_size:     2304      0x900

     start_process_ram: 536887296 0x20004000
   start_process_flash:   196680    0x30048

    kernel version: ^2.0

elf2tab finished. exit status: 0
Detected platform nrf52840
Tockloader flags: ["--jlink", "--arch", "cortex-m4", "--board", "nrf52dk", "--jtag-device", "nrf52"]
Warning: tockloader listen may miss early messages on platform nrf52840
tockloader uninstall command: "tockloader" "uninstall" "--jlink" "--arch" "cortex-m4" "--board" "nrf52dk" "--jtag-device" "nrf52"
[INFO   ] Using settings from KNOWN_BOARDS["nrf52dk"]
[STATUS ] Preparing to uninstall apps...
[ERROR  ] No apps are installed on the board
tockloader uninstall finished. exit status: 1
tockloader install command: "tockloader" "install" "--jlink" "--arch" "cortex-m4" "--board" "nrf52dk" "--jtag-device" "nrf52" "target/thumbv7em-none-eabi/release/examples/low_level_debug.tab"
[INFO   ] Using settings from KNOWN_BOARDS["nrf52dk"]
[STATUS ] Installing app on the board...
[ERROR  ] ('Size only valid with one TBF, found: {}', 0)
tockloader install finished. exit status: 1
thread 'main' panicked at 'tockloader install returned unsuccessful status exit status: 1', runner/src/tockloader.rs:105:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Aborted (core dumped)

It seems that tockloader (I am using tockloader master with a tiny patch to print the number of tbfs found, and the latest elf2tab) is not detecting a tbf header in the tab file. I haven't had a chance to dig into why this is

1. Forward stdin to QEMU. 2. Add -serial mon:stdio to QEMU's flags. 3. Print the full IO error if we are unable to delete the TBF file before elf2tab invocation.

…for use with QEMU.

hudson-ayers

Though the tockloader support doesn't work, this PR documents it as untested, and I think it is fine if this is merged so people can use the (useful) qemu runner until someone (hopefully me) has time to dig into the tockloader support issues

runner/src/elf2tab.rs

runner/src/qemu.rs

…rify runner's TBF_HEADER_SIZE value.

hudson-ayers · 2022-02-03T03:14:53Z

bors r+

bors · 2022-02-03T03:35:10Z

Build succeeded:

ci
size-diff

The only major change from upstream that this preserves is customized external assembly linkage. This should be mostly removed once libtock switches to using `global_asm!` - now that that will be stabilized. This CL is a squash merge of the following commits: 28ebb70 Merge #365 a0440ba Merge #370 abff51c Merge #359 85242dc Merge #364 f1bd0d4 Merge #361 b0f8593 Merge #355 4c7ecb6 Merge #358 6f8b512 Merge #356 73cbbde Merge #357 022984f Merge #353 b2e6471 Merge #352 9eb12c0 Merge #351 This was retrieved with the following command: ``` git log ti50-internal/upstream/master --not \ $(git merge-base --all ti50-internal/upstream/master m/main) \ --first-parent ``` BUG=None TEST=ti50/common make build SOURCE=UPSTREAM(28ebb70) Change-Id: I0aa1606f407a0b4bcf0c76b62261414903479092

jrvanwhy force-pushed the runner branch 4 times, most recently from dea7e41 to 3530195 Compare January 21, 2022 20:40

jrvanwhy force-pushed the runner branch from 3530195 to 2c45730 Compare January 21, 2022 20:52

jrvanwhy marked this pull request as ready for review January 21, 2022 20:55

jrvanwhy added the significant Indicates a PR is significant as defined by the code review policy. label Jan 21, 2022

Install Ninja in the CI workflow, as QEMU now requires it.

4f34598

hudson-ayers reviewed Jan 21, 2022

View reviewed changes

runner/src/qemu.rs Outdated Show resolved Hide resolved

Remove the libtock-rs 1.0 integration tests, as they do not pass with…

e2505b5

…out having `flash.sh` as the runner.

alistair23 requested changes Jan 24, 2022

View reviewed changes

runner/src/qemu.rs Outdated Show resolved Hide resolved

Makefile Show resolved Hide resolved

runner/src/elf2tab.rs Outdated Show resolved Hide resolved

jrvanwhy added 3 commits January 25, 2022 16:02

Address review feedback:

dc96125

1. Forward stdin to QEMU. 2. Add -serial mon:stdio to QEMU's flags. 3. Print the full IO error if we are unable to delete the TBF file before elf2tab invocation.

runner: Add support for multiple platforms in the QEMU deployer.

22b0cc6

Add a kernel-opentitan make action that builds an OpenTitan kernel …

a67900f

…for use with QEMU.

hudson-ayers approved these changes Jan 31, 2022

View reviewed changes

alistair23 requested changes Feb 2, 2022

View reviewed changes

runner/src/elf2tab.rs Show resolved Hide resolved

runner/src/qemu.rs Outdated Show resolved Hide resolved

Fix variable name in the QEMU deployer: tab_path -> tbf_path, and cla…

46916e1

…rify runner's TBF_HEADER_SIZE value.

jrvanwhy mentioned this pull request Feb 2, 2022

Removing Tock 1.0 crates (timing?) #327

Closed

alistair23 approved these changes Feb 3, 2022

View reviewed changes

bors bot merged commit 28ebb70 into tock:master Feb 3, 2022

jrvanwhy mentioned this pull request Feb 18, 2022

process_console: Add missing '\r' on command output strings tock/tock#2976

Merged

2 tasks

jrvanwhy deleted the runner branch June 1, 2022 22:09

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add a runner for libtock2 examples. #365

Add a runner for libtock2 examples. #365

jrvanwhy commented Jan 21, 2022 •

edited

Loading

jrvanwhy commented Jan 21, 2022

jrvanwhy commented Jan 21, 2022

jrvanwhy commented Jan 21, 2022

jrvanwhy commented Jan 21, 2022

alexandruradovici commented Jan 22, 2022

jrvanwhy commented Jan 22, 2022

alexandruradovici commented Jan 23, 2022

jrvanwhy commented Jan 23, 2022

alexandruradovici commented Jan 24, 2022 •

edited

Loading

jrvanwhy commented Jan 24, 2022

hudson-ayers commented Jan 25, 2022

hudson-ayers left a comment

hudson-ayers commented Feb 3, 2022

bors bot commented Feb 3, 2022

Add a runner for libtock2 examples. #365

Add a runner for libtock2 examples. #365

Conversation

jrvanwhy commented Jan 21, 2022 • edited Loading

jrvanwhy commented Jan 21, 2022

jrvanwhy commented Jan 21, 2022

jrvanwhy commented Jan 21, 2022

jrvanwhy commented Jan 21, 2022

alexandruradovici commented Jan 22, 2022

jrvanwhy commented Jan 22, 2022

alexandruradovici commented Jan 23, 2022

jrvanwhy commented Jan 23, 2022

alexandruradovici commented Jan 24, 2022 • edited Loading

jrvanwhy commented Jan 24, 2022

hudson-ayers commented Jan 25, 2022

hudson-ayers left a comment

Choose a reason for hiding this comment

hudson-ayers commented Feb 3, 2022

bors bot commented Feb 3, 2022

jrvanwhy commented Jan 21, 2022 •

edited

Loading

alexandruradovici commented Jan 24, 2022 •

edited

Loading