Bricked Device after Toltec install

[emmawilby]

Not entirely sure what happened, but I managed to brick my device installing with toltec. I installed over usb ssh and as soon as the install completed my screen blanked; I was unable to start xochitl via systemctl because the process was exiting (w/ status 255 IIRC) but I didn't get to dig any deeper than that. After uninstalling I was still unable to start xochitl but after a restart things went back to normal.

Since the blanked screen didn't interrupt my ssh access the first time, I decided to try installing again and rebooting to see if it would work. Unfortunately after doing this, my device won't show up on the network at all (and the screen is still blank). The startup screen shows briefly after restarting but otherwise nothing shows up.

Not sure if it's related but I had just updated to 2.8 before setting up toltec and running the install.

I probably won't be able to give many more details on this for a while since I just ordered the parts for https://github.com/ddvk/remarkable2-recovery from alibaba, but can circle back when those arrive.

[mattbeshara]

I have run into what I think is the same problem after installing the new 2.8 update last night. I can still SSH into the RM2 if I plug it in via USB, but it doesn't show up on the wifi, and when it boots the screen is blank. However, I have set up Genie so that it responds to a 3 finger long press by opening KOReader, and that still works, as does KOReader. When I exit KOReader, all I get is a blank screen though.

[consp]

Same issue, but I was still in session and everything went beck to normal after removing.

systemctl messages, not very helpful though:

Jun 24 09:43:59 reMarkable systemd[1]: [[0;1;38;5;185m[[0;1;39m[[0;1;38;5;185mxochitl.service: Start request repeated too quickly.[[0m
Jun 24 09:43:59 reMarkable systemd[1]: [[0;1;38;5;185m[[0;1;39m[[0;1;38;5;185mxochitl.service: Failed with result 'exit-code'.[[0m
Jun 24 09:43:59 reMarkable systemd[1]: [[0;1;31m[[0;1;39m[[0;1;31mFailed to start reMarkable main application.[[0m

Nothing in syslog related to this.

I did have remarkable-hacks installed which might be the reason for recovery.

[mattbeshara]

By doing mv /home/root/.entware /home/root/dot-entware and rebooting, the display does start showing the stock user interface again, but the colours are inverted (black is white and vice versa) and it seems to never clear the screen completely.

[mattbeshara]

..but after pressing the button to sleep the device, and waking it back up again, it seems like it's back to normal!

[matteodelabre]

@emmawilby

I installed over usb ssh and as soon as the install completed my screen blanked; I was unable to start xochitl via systemctl because the process was exiting (w/ status 255 IIRC) but I didn't get to dig any deeper than that. After uninstalling I was still unable to start xochitl but after a restart things went back to normal.

@mattbeshara

I can still SSH into the RM2 if I plug it in via USB, but it doesn't show up on the wifi, and when it boots the screen is blank.

If you’re using the remarkable2-framebuffer release from Toltec stable (i.e. package version 1.0.1-1), it’s expected that you’ll get a blank screen when starting Xochitl. For this to work properly, you should install the 2.8-compatible release that’s currently in the testing branch (package version 1.0.1-5), or manually install remarkable2-framebuffer from this repo’s 0.0.5 release.

Installing an incompatible remarkable2-framebuffer release should not result in your device getting bricked though, since you should still be able to access SSH from the USB interface that is managed by the system and not by Xochitl. You can simply uninstall the rm2fb package to get back to normal.

@consp

systemctl messages, not very helpful though:

You can use journalctl --unit xochitl --follow to get the Xochitl logs. Or you can also stop the service and start xochitl manually from your SSH session by typing xochitl at the prompt, then the logs will be displayed on your terminal directly.

@emmawilby

Since the blanked screen didn't interrupt my ssh access the first time, I decided to try installing again and rebooting to see if it would work. Unfortunately after doing this, my device won't show up on the network at all (and the screen is still blank). The startup screen shows briefly after restarting but otherwise nothing shows up.

To clarify: Were you accessing SSH from the USB network interface or via Wi-Fi?

[matteodelabre]

By doing mv /home/root/.entware /home/root/dot-entware and rebooting, the display does start showing the stock user interface again, but the colours are inverted (black is white and vice versa) and it seems to never clear the screen completely.

Note that while this may work, there may remain some references to /home/root/.entware in your system configuration that may cause issues in the future. Please try to either restore the .entware folder to where it was and uninstall the rm2fb package, or completely remove Toltec (see toltec-dev/toltec#295).

[mattbeshara]

Note that while this may work, there may remain some references to /home/root/.entware in your system configuration that may cause issues in the future. Please try to either restore the .entware folder to where it was and uninstall the rm2fb package, or completely remove Toltec (see toltec-dev/toltec#295).

Absolutely, this was just a temporary measure to see if it would get me the standard UI and wifi back. Thanks for your help, I'm downloading remarkable2-framebuffer 0.0.5 now.

[emmawilby]

@matteodelabre I'm using the usb interface to ssh. I tried to pick up open ssh ports using nmap in case it had changed ips and nothing came up there. I've tried rebooting several times without success, not sure if there's more debugging I can do before I get hardware (the device is showing up on lsusb as it should).

Edit: No insights from dmesg; checked the usb network's settings again and it had swapped to an entirely different ip range that I wasn't scanning before. Found the correct ip and was able to ssh in and get the device working again.

[matteodelabre]

You can also inspect the kernel logs (dmesg -w) to see if there’s more info there. Here’s what I see when I plug in my rM2:

[ 2983.952294] usb 1-2: new high-speed USB device number 4 using xhci_hcd
[ 2984.080604] usb 1-2: New USB device found, idVendor=04b3, idProduct=4010, bcdDevice= 4.14
[ 2984.080624] usb 1-2: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[ 2984.080633] usb 1-2: Product: RNDIS/Ethernet Gadget
[ 2984.080640] usb 1-2: Manufacturer: Linux 4.14.78 with 30b20000.usb
[ 2984.089690] cdc_ether 1-2:1.0 usb0: register 'cdc_ether' at usb-0000:00:14.0-2, CDC Ethernet Device, XX:XX:XX:XX:XX:XX
[ 2984.127985] cdc_ether 1-2:1.0 enp0s20f0u2: renamed from usb0

[nov1n]

I ran into the same problem. Blank screen without ssh access (nor through wifi nor through usb). dmesg -w shows the Remarkable was configured as an ethernet device, but the IP is different from what it was before. Trying to ssh that IP does connect, but the ip and fingerprint are different and the password no longer works. I'm not sure what to make of this :/

[Eeems]

I ran into the same problem. Blank screen without ssh access (nor through wifi nor through usb). dmesg -w shows the Remarkable was configured as an ethernet device, but the IP is different from what it was before. Trying to ssh that IP does connect, but the ip and fingerprint are different and the password no longer works. I'm not sure what to make of this :/

For the reMarkable 1: https://github.com/ddvk/remarkable-uuuflash
For the reMarkable 2: https://github.com/ddvk/remarkable2-recovery

[nov1n]

I ran into the same problem. Blank screen without ssh access (nor through wifi nor through usb). dmesg -w shows the Remarkable was configured as an ethernet device, but the IP is different from what it was before. Trying to ssh that IP does connect, but the ip and fingerprint are different and the password no longer works. I'm not sure what to make of this :/

For the reMarkable 1: https://github.com/ddvk/remarkable-uuuflash
For the reMarkable 2: https://github.com/ddvk/remarkable2-recovery

Thanksfor the links. I'll have a look!

I'm still a bit confused as to how a framebuffer-related package seems to have regenerated the ssh password. 😕

For my understanding: is there any way to see if a toltec package is safe to install for a specific version of the remarkable OS?

[Eeems]

I'm still a bit confused as to how a framebuffer-related package seems to have regenerated the ssh password. 😕

It doesn't. Perhaps the device isn't booting into a normal boot mode? Or, did you try to factory reset?

For my understanding: is there any way to see if a toltec package is safe to install for a specific version of the remarkable OS?

Not currently. We have had a couple discussions on how to properly handle this without much luck yet. For now, if you want to use 2.6+ you can grab the toltec packages from the build artifacts here: toltec-dev/toltec#366

[nov1n]

It doesn't. Perhaps the device isn't booting into a normal boot mode? Or, did you try to factory reset?

No, I did nothing of the sort.

Not currently. We have had a couple discussions on how to properly handle this without much luck yet. For now, if you want to use 2.6+ you can grab the toltec packages from the build artifacts here: toltec-dev/toltec#366

I see, just my luck! I will try to follow the recovery steps, thanks for your help (:

[consp]

@consp

systemctl messages, not very helpful though:

You can use journalctl --unit xochitl --follow to get the Xochitl logs. Or you can also stop the service and start xochitl manually from your SSH session by typing xochitl at the prompt, then the logs will be displayed on your terminal directly.

Late reply sorry for that.

reMarkable: ~/ journalctl --unit xochitl --follow
-- Logs begin at Thu 2021-06-24 09:37:36 UTC. --
Jul 01 11:05:12 reMarkable xochitl[2754]: Unable to find update_fn
Jul 01 11:05:12 reMarkable xochitl[2754]: PLEASE SEE https://github.com/ddvk/remarkable2-framebuffer/issues/18
Jul 01 11:05:12 reMarkable systemd[1]: xochitl.service: Main process exited, code=exited, status=255/EXCEPTION
Jul 01 11:05:12 reMarkable systemd[1]: xochitl.service: Failed with result 'exit-code'.
Jul 01 11:05:12 reMarkable systemd[1]: xochitl.service: Scheduled restart job, restart counter is at 4.
Jul 01 11:05:12 reMarkable systemd[1]: Stopped reMarkable main application.
Jul 01 11:05:12 reMarkable systemd[1]: xochitl.service: Start request repeated too quickly.
Jul 01 11:05:12 reMarkable systemd[1]: xochitl.service: Failed with result 'exit-code'.
Jul 01 11:05:12 reMarkable systemd[1]: Failed to start reMarkable main application.
Jul 01 11:05:12 reMarkable systemd[1]: xochitl.service: Triggering OnFailure= dependencies.

Manual start starts fine. Already mentioned the digests in the other issue.

Logs:

reMarkable: ~/ xochitl 
09:15.434 default                  2021-06-11T11:42:05Z heads/releases/bertwhistle 264f47ba0 (int main(int, char**) /usr/src/debug/xochitl/override+gitAUTOINC+264f47ba03-r0/git/src/main.cpp:294)
Registering exit handlers
09:15.435 default                  we're running on an epaper device (int main(int, char**) /usr/src/debug/xochitl/override+gitAUTOINC+264f47ba03-r0/git/src/main.cpp:301)
Reading waveforms from /usr/share/remarkable/320_R327_AFEC21_ED103TC2M1_VB3300-KCD_TC.wbf
Running INIT (111 phases)
09:17.133 default                  SWTCON initialized \o/
09:17.144 default                  EPD platform plugin loaded!
09:17.147 default                  QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-root'
09:17.172 qt.qpa.input             evdevtouch: Using device discovery
09:17.179 qt.qpa.input             evdevtouch: Adding device at /dev/input/event2
09:17.180 qt.qpa.input             evdevtouch: Using device /dev/input/event2
09:17.181 qt.qpa.input             evdevtouch: /dev/input/event2: Protocol type B  (multi), filtered=no
09:17.182 qt.qpa.input             evdevtouch: /dev/input/event2: min X: 0 max X: 1403
09:17.184 qt.qpa.input             evdevtouch: /dev/input/event2: min Y: 0 max Y: 1871
09:17.185 qt.qpa.input             evdevtouch: /dev/input/event2: min pressure: 0 max pressure: 0
09:17.185 qt.qpa.input             evdevtouch: /dev/input/event2: device name: pt_mt
09:17.657 default                  XoApp: Change to sync 1.0 requested...
09:17.708 default                  Lister task was rejected from the queue 0
09:17.713 default                  > Loading (LoadingBlock::LoadingBlock() /usr/src/debug/xochitl/override+gitAUTOINC+264f47ba03-r0/git/src/shared/include/shared/loadingthread.h:51)
09:17.714 default                  Starting loading (LoadingBlock::LoadingBlock() /usr/src/debug/xochitl/override+gitAUTOINC+264f47ba03-r0/git/src/shared/include/shared/loadingthread.h:54)
09:19.002 default                  Could not create scene graph context for backend 'epaper' - check that plugins are installed correctly in /usr/lib/plugins
09:19.120 default                  qrc:/qml/device/system/SleepScreen.qml:16:5: QML Image: Cannot open: file:///usr/share/remarkable/sleeping.png (qrc:/qml/device/system/SleepScreen.qml:16)
09:19.192 default                  qrc:/qml/device/main.qml:425:13: QML Connections: Implicitly defined onFoo properties in Connections are deprecated. Use this syntax instead: function onFoo(<arguments>) { ... } (qrc:/qml/device/main.qml:425)
09:19.361 default                  qrc:/qml/device/main.qml:197:9: QML Connections: Detected function "onTimeout" in Connections element. This is probably intended to be a signal handler but no signal of the target matches the name. (qrc:/qml/device/main.qml:197)
09:19.362 default                  qrc:/qml/device/main.qml:197:9: QML Connections: Detected function "onResetDone" in Connections element. This is probably intended to be a signal handler but no signal of the target matches the name. (qrc:/qml/device/main.qml:197)
09:19.363 default                  Creating window
09:19.447 default                  < Loading (LoadingBlock::~LoadingBlock() /usr/src/debug/xochitl/override+gitAUTOINC+264f47ba03-r0/git/src/shared/include/shared/loadingthread.h:62)
09:19.447 default                  Stopping loading (LoadingBlock::~LoadingBlock() /usr/src/debug/xochitl/override+gitAUTOINC+264f47ba03-r0/git/src/shared/include/shared/loadingthread.h:66)
09:19.448 default                  Stopped loading (LoadingBlock::~LoadingBlock() /usr/src/debug/xochitl/override+gitAUTOINC+264f47ba03-r0/git/src/shared/include/shared/loadingthread.h:68)
09:19.449 qt.qpa.input             evdevtouch: Updating QInputDeviceManager device count: 1 touch devices, 0 pending handler(s)
09:19.533 default                  	 virtual Entry::SharedPtr Library::loadEntry(const QString&) 11 ms  *** ON GUI THREAD ***
09:19.730 default                  	 virtual Entry::SharedPtr Library::loadEntry(const QString&) 12 ms  *** ON GUI THREAD ***
09:19.745 default                  XMLHttpRequest: Using GET on a local file is dangerous and will be disabled by default in a future Qt version.Set QML_XHR_ALLOW_FILE_READ to 1 if you wish to continue using this feature.
09:19.763 default                  	 void Library::load() 313 ms  *** ON GUI THREAD ***
09:19.778 xochitl.wifi             wlan0: Link is up
09:19.860 default                  FIXME: mipmap filtering not supported
09:19.860 default                  FIXME: wrapmode not supported
09:19.861 default                  FIXME: wrapmode not supported
09:19.861 default                  FIXME, setting node inner target rect for image node not supported: QRectF(0,0 60x60)
09:19.861 default                  FIXME: mirroring not supported
09:20.386 xochitl.drawingarea      No worker!
09:20.386 xochitl.drawingarea      No worker!
09:21.029 xochitl.library          Asked for invalid document "28fcced7-1ccb-432b-ab46-0b2f6384cf1e"
09:21.038 xochitl.wifi             wlan0: Address added: 192.168.0.104
09:21.039 xochitl.wifi             wlan0: Address added: fe80::d612:43ff:fed5:f0ec
09:21.099 xochitl.wifi             Scanning: true
09:21.100 xochitl.wifi             Got gateway: 192.168.0.1
09:21.109 xochitl.wifi             We probably have Internet
09:21.502 xochitl.updates          Request finished. Was dispatched at: 2021-07-01T11:09:21.108
09:24.422 xochitl.wifi             Scanning: false

[Eeems]

@emmawilby any luck with recovery?

[emmawilby]

@Eeems Yeah, I actually figured out that for whatever reason the remarkable had chosen a particularly poor moment to change ip addresses and I had been trying to connect to its previous one. Once I had the correct address ssh was working and I was able to restore xochitl (haven't had a chance to try the new version of rm2fb yet)

[Eeems]

@emmawilby Awesome! I guess this issue can be closed then.

[HalosGhost]

Incidentally, I'm somewhat sure I'm affected by this issue, and I was foolish enough to not write down my ssh password before installing rm2fb and nao. I've ordered the parts necessary to make a jig for recovery and will report back when I get everything i need. ☺

[Eeems]

Incidentally, I'm somewhat sure I'm affected by this issue, and I was foolish enough to not write down my ssh password before installing rm2fb and nao. I've ordered the parts necessary to make a jig for recovery and will report back when I get everything i need. ☺

Sorry to hear that :( If you need help with the pogo connector after you get the parts, I'd suggest reaching out on the community discord server as there are a number of people who've had good success with it so far.

[HalosGhost]

No worries (there was some serious silly on my part for not writing things down the way I should have). And I'll keep that in-mind. Should I get the jig functional, I intend to heatshrink-wrap it and try to make it as streamlined as possible so I can just keep it around (will generally make me feel much more confident when fiddling with it).

[matteodelabre]

I’ll be closing this issue: Toltec now supports version 2.8, with support for 2.9 coming soon.