Why is MTIME M-mode only?

[Indanz]

Since commit 59a08cd, which is part of Split region permissions into M-mode and SU-mode MTIME is M-mode only.

This breaks OSes that read MTIME directly to avoid the overhead of emulated rdtime instructions on hardware that don't have native support for the instruction. The emulated rdtime has huge overhead. MTIME is well defined by the RISC-V standard, so we thought we could rely on it.

Is MTIME off-limits for OS code now, and why? Is that documented somewhere? Could read-only access to the CLINT/MTIME memory region be added back so low overhead timer reads are possible again on all RISC-V platforms?

Please advise how we should proceed.

[tswaehn]

MTIME is by spec only readable by machine mode. currently there is no function from supervisor mode to read MTIME. I suggest to implement a timer extension to read the MTIME from supervisor mode.

[SiFiveHolland]

FWIW, some implementations (e.g. C906) restrict access to the CLINT to M-mode in hardware, so relaxing the PMP restrictions will not accomplish what you want portably. There are also (non-compliant) CLINT implementations that omit the mtime MMIO register because they implement the time CSR.

@tswaehn Most of the overhead in emulating rdtime is in the context switching, so it's not clear that an ecall would be any faster than instruction emulation.

[tswaehn]

It wouldnt probably be faster, but an option to read mtime directly in supervisor.

additionally rdtime or mtime could be not implmented. instead custom timers are possible.

So wondering why OSes should use mtime which is implementation specific.

Probably its more generic to use rdtime if implemented - still this is optional.

So an openSBI extension that will provide the time in all custom implementations (mtime, rdtime, custom timer) would be most generic (with the downside of performance).

[Indanz]

We used to use only rdtime, but found out that those have erratic performance overhead up to 2 us when emulated. Considering our syscalls take around 1 us, that is unacceptable overhead. That's when we added support for MTIME. However, shortly after we did that, read-only access to MTIME was taken away in a newer OpenSBI version, and here we are.

All I want is a fast, non-emulated rdtime. When that is missing, what is the best work-around?

Or should we just accept that performance is not a priority for current RISC-V implementations and use rdtime, however horribly slow?

[avpatel]

It is expected that performant HW will implement time CSR which is a read-only copy of mtime.

The time CSR is expected to be available for both HS-mode and VS-mode. If it is not implemented by HW then M-mode will trap-n-emulate it.

[Indanz]

In the meantime, why can't S-mode have a read-only mapping of MTIME like it used to have, on hardware where that makes sense?

The portable solution is rdtime, the fast non-portable solution is read-only MTIME and there is no alternative.

[tswaehn]

just my 2 cents: ecall (the ultimate generic option) on an 100MHz cpu should be around 0.7usec

[tswaehn]

In the meantime, why can't S-mode have a read-only mapping of MTIME like it used to have, on hardware where that makes sense?

that would require a special OS running in supervisor mode, that "knows" it can use mtime even so its not in machine mode.

[Indanz]

just my 2 cents: ecall (the ultimate generic option) on an 100MHz cpu should be around 0.7usec

Well, it was bad enough to occasionally fail a clock synchronisation test with a threshold of 2 us between cores on a HiFive Unleashed (the spec says time should be within 1us). In hindsight, if the ecall takes a global lock on SMP systems, that may explain it.

Slowdown thanks to emulated rtdtime is about 40% for simple syscalls. (The tests showing no big change don't check the time.)

that would require a special OS running in supervisor mode, that "knows" it can use mtime even so its not in machine mode.

Agreed, but seL4 is already build per board, so that isn't a big deal. But that knowledge could be passed by DTS too.

[tswaehn]

I see, but as this seems to be some very special case to make mtime available in supervisor mode and needs knowledge about presence of mtime in the used riscv hardware. the actual request might be to add an openSBI extension that "opens" the mtime register directly and make it accessible to supervisor. then by default we would rely on a generic (and maybe slightly slower) indirect ecall getTime() and if needed you could switch to direct access of mtime.

side note 1:

as sel4 is some kind of secure abstraction layer, ... just wondering if on a secure system I want to have 2 timers. 1 timer super secure and probably in machine mode that is used to get some pseudo random numbers. and 1 timer to actually run scheduler and tasks and so on. so maybe at the end of the day you might not make mtime available, ... just thinking about it.

side note 2:

maybe this could also be an option to integrate more drivers of hardware TRNG, crypto, UART, ... directly in openSBI and then have (a) generic way of accessing security modules and drivers for all hardware - completely hardware independant kernel and (b) split security - in machine mode from actual kernel task - in supervisor mode. -- that would open the door to come with other kernels that can rely on same security built into machine mode openSBI.

[Indanz]

The only thing special is rdtime being emulated and extremely slow, hence the need for falling back to MTIME. If a hardware implementation of rdtime was mandatory we wouldn't have this mess. The CLINT registers are standard and defined, it's just their address that differs per hardware.

Before commit 8b56980 supervisor had read-only access to the CLINT memory region, which we used. What harm can there be in giving S-mode read-only access to MTIME on platforms that have slow rdtime?

Note 1: seL4 is a secure microkernel, it is not an abstraction layer. The timer we need is a fast timer for the seL4 kernel itself for scheduling, not to expose to user space. User space will have to do with rdtime, however slow it may be. For some workloads high performance and accurate timing in user space is very important to have, for those you do need proper rdtime.

Note 2: What you seem to be describing is similar to UEFI. Please just give proper well defined hardware interfaces to such things instead of a software emulation layer.

Most of seL4 code is mathematically proven to be correct, the last thing seL4 wants to do is depend on unverified OpenSBI code after bootup (but that's what it does now). I think having seL4 run in M-mode might make more sense, but currently it runs in S-mode. seL4 doesn't have hardware abstraction as a design goal, while that's probably the whole point of OpenSBI.

[tswaehn]

actually I like your comparison to UEFI. because also things like TPM could be implemented in machine mode (openSBI). storing secrets in machine mode and accessing them through openSBI would be great. ex. I do have a device with secure boot in machine mode. even the kernel in supervisor mode should not have access to most secure things. I see openSBI as a low-level driver/hardware/crypto interface. so thats why mtime is something really special, because it actually belongs to openSBI only, but for performance reason should be made available to supervisor - I understand why. just wondering how to consent and do the generic way.