[general] Rendering regression(s) after FBInk upgrade

[sre3]

After upgrading to the new FBInk (just threw the new KoboRoot.tgz into .kobo), the screen does not refresh when exiting out of the remux menu to Nickel, so you end up with something like this:

Normal home screen for comparison:

Somewhat relatedly, the experience in general seems slightly less... polished, I suppose. For instance, upon loading harmony, the framebuffer seems to have stored the previous frame from when the app was exited, and that frame is briefly displayed on opening the app.

There are also many other (less severe) ghosting examples everywhere. Mines, for instance, will leave a nice grid of ghosty squares that is very noticeable - but pretty much every app will eventually leave ghosting all over the screen, and the only way to clear it is to exit to Nickel and force a full screen refresh.

In fact, Nickel itself will leave a faint ghost of your home screen that is particularly noticeable when opening harmony.

I should note that I do recall harmony having some minor ghosting issues pre-FBInk, specifically when erasing,
but I don't recall any other apps having any issues.

I did go digging and found the Nickel home screen ghosting also did exist in some capacity, but not as badly. It's visible on my remux menu image in the guide.

P.S.
I did notice that eclear is installed by default by the KoboRoot and is available in remux, but it does not seem to refresh or get rid of the ghosting in any way.

P.P.S.
@raisjn
I unsuccessfully tried testing rotation... nothing would rotate....

[raisjn]

thanks for filing and thorough details! I will investigate this week - sorry for the delay

[sre3]

No worries, I'm not in any hurry. I can do some more thorough testing if you need, this was done fairly quickly.

[raisjn]

i pushed a diff that changes the waveform mode when switching apps through remux (b818e01), it could be that using WAVEFORM_MODE_AUTO stopped working when i switched to fbink. i changed it to always use GC16 and double checked that the rendering looks better on the color kobos. can you try it out? thanks!

https://build.rmkit.dev/master/latest/kobo/

and to confirm: the issue mostly appeared when switching between apps, right? if there is still ghosting, i can try out other waveforms (maybe there's more appropriate ones than GC16 that reduce ghosting even more)

[raisjn]

@Szybet and @NiLuJe helped me debug this, i'm using the constants from MXCFB for the waveform modes, but fbink has its own constants (WFM). in this case, WFM_AUTO = 0, while WAVEFORM_MODE_AUTO = 257 (which wraps around to 1 which is DU - this is what causes the ghosting / weird drawing)

[NiLuJe]

Interestingly enough, GC16, which you mentioned earlier, is probably going to be the only exception that is mostly always set to 2 everywhere (except sunxi, but we don't talk about sunxi :D).

[sre3]

Aha! That makes sense. I was digging around in the fbink header file and checking waveforms - I did notice the constants mismatch, and was about to ask about it!

So there were two issues:

1. Nickel would not refresh when you returned from the remux menu (has been fixed from forced G16)
2. General ghosting problems (not fixed)

In terms of 2:
No full screen refreshing or "flashing" occurs once you have exited nickel. This is particularly noticeable when changing from mines to harmony (or any other app) it will ghost quite severely (this seems to be unique to mines!).

There is also minor ghosting present in other cases: eg.

1. opening up remux from the home tab leave ghosts of your currently reading covers (though not nearly as severely),
2. the remux menu will leave minor ghosting that is particularly visible on a blank harmony canvas
3. erasing stuff at the same location repeatedly in harmony will eventually accumulate enough to ghost

These were present in the non-fbink build, so I am assuming this is a limitation of G16 without flashing...

I am pretty clueless about FBInk, but would it be possible to implement a flash when launching an app through remux?

[NiLuJe]

Yeah, you almost exclusively always want to flash on GC16 (unlike some other stronger pairings, it does do something halfway sensible when not, but it's not really ever done in practice).

[sre3]

I was pretty impressed with the performance without flashing, to be honest. Nickel seems to flash incredibly aggressively when using notebooks, and it's impressive how well harmony can work without any flashing at all...

[raisjn]

In terms of 2:
No full screen refreshing or "flashing" occurs once you have exited nickel. This is particularly noticeable when changing from mines to harmony (or any other app) it will ghost quite severely (this seems to be unique to mines!).

i wonder if this is somewhat helped by forcing GC16 (instead of WAVEFORM_MODE_AUTO) when using remux to switch apps or redraw the screen when returning to an app. if you can report back your experiences, that would be awesome! really appreciate the depth you are digging in and helping :-D

Aha! That makes sense. I was digging around in the fbink header file and checking waveforms - I did notice the constants mismatch, and was about to ask about it!

nice - i love when people read code :-D i feel a bit embarrassed that it was only a one line change but took me a few weeks to get to. on a side note: i've installed the rmkit apps on kobo color and they seem to work fine

[NiLuJe]

Nickel seems to flash incredibly aggressively when using notebooks

I haven't looked at how Nickel handles the pen on MTK yet; but on sunxi it had to do a bunch of gymnastics on pen lifts because the driver is oh so very buggy (as in, fatally so: hanging the kernel with pen mode is extremely easy). That might just be a remnant of all that mess.

What's visible from the Elipsa 2E kernel sources looks like a somewhat different approach than what lab126 did for the Scribe, though.
IIRC, the lab126 implementation essentially does most of the heavy lifting in-kernel, userland just has a single ioctl to bracket "pen mode" stuff (which matches how it's done on sunxi, FWIW).
On NTX, it looks like you instead need to do a bunch of gymnastics with extra per-refresh A2 flags or something.

[NiLuJe]

it does do something halfway sensible when not

Speaking of, it has the notable feature of not doing anything for pixels that haven't changed. That obviously goes out the window once you ask for a flash, but is otherwise a fairly interesting feature.

e.g., if you display the same image multiple times via fbink -W GC16 -i foo.jpg, only the first call will actually refresh the screen; further requests will be entirely optimized out.

This becomes more obvious if you then try to display something based on that initial image, but with only some content having been modified: only those pixels will flip ;).

[sre3]

i wonder if this is somewhat helped by forcing GC16 (instead of WAVEFORM_MODE_AUTO) when using remux to switch apps or redraw the screen when returning to an app. if you can report back your experiences, that would be awesome! really appreciate the depth you are digging in and helping :-D

So.. that was with the forced G16... It did seem a little better? Like I noted earlier, it did definitely fix issue 1. I unfortunately forgot to test the old DU version before installing the new KoboRoot... I can reinstall the old version to test if needed.

e.g., if you display the same image multiple times via fbink -W GC16 -i foo.jpg, only the first call will actually refresh the screen; further requests will be entirely optimized out.

Huh, cool! I don't have that much experience with e-ink development, to be honest, and I'm a little scared to do any digging with the 2E with soldered eMMC...

That might just be a remnant of all that mess.

It does make sense that they would have tried to reuse code from the sunxi implementation, to save R&D costs or whatever. I have certainly noticed some odd behavior with the pen in Nickel, some funny ones in basic notebooks:

• If you fiddle with the pen tip, it will occasionally start writing without coming into direct contact with the screen (breaks something to do with how the MPP stuff recognizes distance to the digitizer, perhaps?) It's fixable by tapping the screen a few times while trying to write without contact.
• Occasionally the palm rejection will freak out and draw a line (a shape recognized, straightened one) directly to your hand. This will then break the flash upon erasing, so everything you erase ghosts. You can fix it by opening the menu. (sometimes as well as ghosting, the eraser just doesn't erase anything)

Also, Kobo's pdf annotation is incredibly buggy:

• Occasionally the rendering just goes wild and comes up will all sorts of ways to render things very slowly, sometimes spiralling out from a center point, or curling up from the bottom of every letter etc. etc.
• Occasionally it randomly decides to connect every letter that you wrote in the past little while after a flash
• Occasionally it randomly decides that the eraser is actually also the stylus tip and starts writing instead of erasing (I love this one)

I enabled native annotation with dev options, but the rendering speed was just unusable... Oh well, it only happens occasionally. 😅

The one bug that has been consistently annoying me, to be honest, is the one where:

• If you open up a new basic notebook, and let it sit without writing for a few minutes (haven't been able to figure out exactly when) when you start writing, it immediately crashes and reboots.

Also it used to just crash randomly and whatever wasn't autosaved got deleted, but that hasn't happened in a while, thank god...