Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Door shows as "No Response" in HomeKit until rebooted #94

Closed
lancep opened this issue Jan 16, 2024 · 77 comments
Closed

Door shows as "No Response" in HomeKit until rebooted #94

lancep opened this issue Jan 16, 2024 · 77 comments

Comments

@lancep
Copy link

lancep commented Jan 16, 2024

I can successfully add the ratgdo to HomeKit, and everything works great, but after a day or so, it shows as "No Response" until I reboot the device.

@pattont
Copy link

pattont commented Jan 16, 2024

0.9 I am having a similar issue where at some point it goes “no response” and I have to manually unplug and re-plug.

@drewskix86
Copy link

drewskix86 commented Jan 17, 2024

Me three, it seems to hang and will occasionally come back on its own after a couple of minutes, other times it requires restart. Running version 0.9 with a Chamberlain CS105MYQ with an obstruction sensor installed.

@egrim
Copy link

egrim commented Jan 17, 2024

I thought it was just me or maybe my RATGDO's hardware had a problem. I'm planning to try to debug at some point, but not sure where to begin with that yet so any tips are appreciated.

@OleSam85
Copy link

OleSam85 commented Jan 17, 2024

@egrim I'm in no way qualified to offer much technical guidance here, but I documented how I was able to log from the device in #24 (comment)
There might be other ways and other tricks needed to get the necessary data to fully debug the issue, but it might be a good place to start. Of course I had to keep my computer awake and connected to the ratgdo in the garage the entire time, and as soon as I started logging, I was unable to reproduce the problem! Anyway, hopefully that helps.

@dkerr64
Copy link
Collaborator

dkerr64 commented Jan 17, 2024

@thenewwazoo is working on replacing the Arduino-based HomeKit stack with the ESP-HomeKit stack. We are hopeful that this will bring more stablity to the connections.

@pattont
Copy link

pattont commented Jan 17, 2024 via email

@egrim
Copy link

egrim commented Jan 17, 2024

Oh! I'll give that a try as I am indeed on UniFi.

@sttonev
Copy link

sttonev commented Jan 17, 2024

I'm on UniFi too and have been fortunate in having really good uptime. My configuration is to have a separate (2.4Ghz) SSID broadcasting only from the closest AP. Not sure if there's a material difference between that and forcing a client to a given AP, but I wanted to offer another (working) solution.

@lancep
Copy link
Author

lancep commented Jan 17, 2024

Ah, I'm also on UniFi, I'll try the roaming lock

@pattont
Copy link

pattont commented Jan 18, 2024 via email

@jgstroud
Copy link
Collaborator

I'm on unifi and I locked my device to the closest AP. I have twice hit this issue where I had to power cycle the ratgdo to get it back alive. And it was several weeks apart. So, this helps, but don't expect it to be a magic bullet. I've also seen where if I power cycle my AP, it may not reconnect. Which seems like it might be similar to moving between APs. I think once it's connected it doesn't like getting disconnected.

@dmsinger
Copy link

Also Unifi, have also run into this issue twice. However, I can see the ratgdo has not connected to another AP within Unifi's console, so that wasn't issue. I've had no AP resets either. ratdgo was also still connected to the AP when it was reset according to the connection log.

@lancep
Copy link
Author

lancep commented Jan 18, 2024

I did the AP lock yesterday, woke up this morning to "no response" :(

@dmwilson1990
Copy link

Same issue here - UniFi and locked to AP. Still no response.

@infamous-pattern
Copy link

Been having the same issue with my RATGDO on 0.9 and also running UniFi. Going to lock to a single AP just to be safe. Looking forward to the next firmware update. Thanks.

@sttonev
Copy link

sttonev commented Jan 19, 2024

I'm on UniFi too and have been fortunate in having really good uptime. My configuration is to have a separate (2.4Ghz) SSID broadcasting only from the closest AP. Not sure if there's a material difference between that and forcing a client to a given AP, but I wanted to offer another (working) solution.

In case it matters, I'm on 0.6.0. It doesn't seem like anything was changed in the interim that would affect connectivity, though.

@jim-wong
Copy link

jim-wong commented Jan 20, 2024

I'm on 0.8 and am seeing this problem, as well. In the most recent occurrence, it happened after just a few hours: HomeKit couldn't see the garage door, the web interface wouldn't load, and the device wasn't reachable via ping. I'm also using UniFi network gear, and the RatGDO has line of sight to the nearest AP, which is about 15 feet away.

If there's anything I can do provide data to help troubleshoot this issue, I'd love to know.

@thenewwazoo
Copy link
Collaborator

I appreciate everyone continuing to submit bug reports. Thank you! And I'm not kidding - it helps me get a sense of the scope of the problem.

I do have good news, though! I have made significant progress in re-implementing the firmware to avoid the problem, which is instability in the Arduino core. I've replaced Arduino with the HomeKit SDK written by Espressif (who makes the ESP8266 chips used in the ratgdo), and my prototype is working well.

Stay tuned, I expect to be able to publish a version for intrepid users to test in the next few days.

@tadaschi
Copy link

Thank thenewazzo!
I am another one with unifi and HomeKit 0.9.0

same problem

we gonna be waiting for a new firmware!

keep the great work!

excluding this bug the HomeKit is by far the easiest and most responsive of all three firmwares for me

@Guy559
Copy link

Guy559 commented Jan 22, 2024

I'll also confirm issue with Unifi and 0.9.0 firmware. Three other devices using ESPHome work fine via HASS. Only this device fails to update, even when connecting to known good AP.

Appreciate acknowledgement and I, too, am looking forward to f/w update.

@blakej
Copy link

blakej commented Jan 24, 2024

Unifi user here who did a good bit of troubleshooting before. Three APs, three ratgdos.

v0.8 has definitely corrected the "No Response" issue I was very very frequently getting caused by hopping between APs.

What v0.8 did not (and could not) correct was WiFI signal issues that only my ESPs were seeming to have. Despite "No Response" being corrected (when WiFi was working), I would still randomly and frequently get drop offs from WiFi entirely - some ratgdos more than others. At least two of them would not reliably stay connected for even an hour.

About two weeks ago I made two changes in Unifi, and since then only one device has reconnected one time (and only a 5 sec disconnect) - the other two devices haven't lost connection a single time.

Change 1: Stopped using "Minimum Data Rate Control" on my 2.4 GHz network. It was previously set to "5.5 Mbps" as a minimum, and I think this was forcing hopping as well as disconnects from not meeting the minimum.

Change 2 (and probably more impactful): I increased AP 2.4 GHz signal strength. Yes, I know just increasing AP signal strength doesn't mean the devices can respond stronger, but it has helped - massively. Before this, each ratgdo was typically under 10 Mbps on rx & tx. After this, right now, they are all 52-54 Mbps on rx & tx. "AP/Client Signal Balance" are all "Good". "WiFi Experience" are all "Excellent".

Note: Locking to an AP did not help me. Prior to these changes I tried a few different combinations of locking with no reliable improvement. I currently do not lock to an AP for any of my ratgdos.

For everyone else on Unifi, if you're experience high latency / low data rates / full wifi disconnects - check the settings I listed above. Looking forward to @thenewwazoo's HomeKit rewrite, but I suspect it will have no impact if you can't get WiFi stable first like I was experiencing.

I'm using v0.9, and since making the listed Unifi changes above, all three ratgdo's have been flawless - instant responses and never showing "No Response".

@Guy559
Copy link

Guy559 commented Jan 24, 2024

Thanks @blakej. I’ll try #1 to see if that makes a difference.

I have four devices: one is using HomeKit-RatGDO and the other three are using ESPHome firmware. All four devices are on UniFi APs. All four maintain strong WiFi connections with no drops, so that doesn’t seem to be the problem, but perhaps bandwidth sifting could be causing some issues.

I can successfully add the HomeKit-RatGDO firmware device, control door, and all looks good for a day or two. Then, despite strong and consistent WiFi connection, Apple Home app shows “No Response” only for the HomeKit device. The non-responsive device is still connected to same WiFi AP and I can access the web page for it. Rebooting also doesn’t help - device will reestablish WiFi connection successfully but Home still shows “No response.”

Using 0.9.0 firmware. Next step will be to check the recommendation you offer above, remove device from Home, power cycle, reinstall firmware, and re-add to Home.

@DMBlakeley
Copy link

I do not have a UniFi system but am using a Velop system. One setting on the Velop which I have found made a difference for HomeKit device connectivity was to turn off "Device Steering". This lets the device connect to its preferred node rather than the Velop system steering which can result in a disconnect. Not sure if similar setting on UniFi but one to also try if available.

@Guy559
Copy link

Guy559 commented Jan 24, 2024

Went ahead and reloaded f/w, reconnected to closest UniFi WiFi and all is good in Home at the moment. Will keep an eye on it in days to come.

Looking forward to next release. Impressive to see how quickly this project went from concept to reality, so understand growing pains will be a part of this process.

@Minnow1986
Copy link

I had the same problem with Unifi (UAP-AC-HD). I downgraded WPA2/WPA3 (combination) to WPA2 (only) and it seems to be working well. I do use Band Steering and Fast Roaming. I don't use QOS. It doesn't seem to be causing an issue at the moment.

@DMBlakeley
Copy link

Also recommend having your 2.4GHz network configured for WPA2 and not WPA2/WPA3 (combination).

@thetvguru
Copy link

Are you guys resolving this issue by rebooting from the web interface or removing power? Are you needing to re-pair to HomeKit?
I have been unable to fix this issue my rebooting both ways, but haven't tried to re-pair to HomeKit.

@egrim
Copy link

egrim commented Jan 24, 2024

I've been able to resolve by rebooting in the web interface, but it sounds like others weren't able to reach it so were forced to power-cycle it.

@jim-wong
Copy link

@thetvguru For me, power cycling the box has done the trick.

@thetvguru
Copy link

Mine is working again. For myself, I had to remove and reinstall to HomeKit.

@jgstroud
Copy link
Collaborator

I've been having similar issues and I'm begin to suspect it's because of my eero network. I'm also getting other symptoms including false notifications (details: #105)

eero networks seem to be a bit of a common thread with regard to these issues. I don't think the Eero's are to blame, but there is something about these networks that the ratgdo doesn't seem to like.

@atmendez
Copy link

atmendez commented Feb 12, 2024 via email

@dkerr64
Copy link
Collaborator

dkerr64 commented Feb 12, 2024

I have a Unifi Dream Machine and had connectivity troubles on 5G. Switching to a (non-Unifi) access point with a stronger 2.4 Ghz signal seems to have fixed the problem for me.

THe ESP8266 does not support 5GHz, it can only connect on 2.4GHz.

@dmsinger
Copy link

I used to have an Eero setup and the switching was very frequent. If there's any way to lock to one Eero, that would probably make it a lot more stable for you.

I haven't had to do anything with the Unifi setup outside of lock it to an AP (which it was doing outside of resets).

One item I do not see anybody mentioning is what they are using for their Apple hub?

My hub has switched a few times, and I think it could have been one of my drops, but it was only once (the drop). I've done multiple upgrades. Luckily I'm on a wired Apple TV right now, but you never know when that'll flip. Hopefully we get control of that one day.

@jgstroud
Copy link
Collaborator

I use a wireless HomePod mini for my hub and mine has been very solid. At this point I wouldn't rule anything out though. I am also on a Unifi AP and I locked my device to the nearest AP.

I have a spare ratgdo with the serial console being logged. I might try forcing it to roam and see if I can trigger this issue. Based on mrthiti/Arduino-HomeKit-ESP8266@3825ef4 I'm wondering if there is still something bad happening in the mdns stack when the ratgdo roams.

@billy27607
Copy link

Like many here, I have a couple of ratgdo devices and a Unifi Network. The devices go non-responsive all the time. Nothing I do with the network helps at all. Even when they are non-resonsive in homekit, I can reach the web management page and use it responsively. I have written my own homekit appliances a few times and have always had trouble when tying to use the Arduino-Homekit-Esp8266 library. Some kind of behavior. So I am convinced that this is something in the timing of the protocol between homekit and the device.

While I hate to do it, I have a home bridge instance running already for some other stuff so I loaded the mqtt-ratgo firmware, used the homebridge-ratgdo plugin and everything runs well and is responsive.

I have had much better luck with the Espressif Homekit SDK in terms of performance and stability, but it is a lot harder to get going with it. So I hope that @thenewwazoo has better luck with that attempt. I hearby volunteer for any testing you want to run when you are ready.

@iyerusad
Copy link

iyerusad commented Feb 29, 2024

I've been running v0.11.0 for 22+ days uptime without any of these issues. I use it just about everyday and working consistently (haven't noticed a failure/issue).

Some speculation:

  • HomePod Mini related strength? I use a wireless HomePod mini for my hub and mine has been very solid. I have 3-4 as hubs, as well as wired ATV4k (isn't chosen as primary hub as often as you'd like)

    • I have additional Eve Water plug that includes a Threads radio as part of my HomeKit. I don't run any of the bridges etc, just vanilla Homekit, and just RatGDO, Eve Water sensor, ATV, and the handful of homepod minis.
  • RatGDO hardware issue? Mine is a v2.5i, ~2023-Dec batch.

  • Wiring issue? I used recommended solid core wire (not stranded).

    • [Super unlikely but a thought] Maybe wiring is a vector for intentional/unintentional glitching. Silly but a thought. My GDO was never registered to MyQ or whatever their cloud thing is.
  • Environmental issue causing CPU issue (heat? interference?)? Mine is wrapped in a Etsy case, maybe that helps. Unit is in garage, temps are 40-60 degrees Fahrenheit (maybe summer heat will introduce failure condition)

  • Network issue? This one increasingly seems less likely to me, particularly if getting unpaired from Homekit. FWIW, multiple Unifi 7 Pros, static leased IP, pretty standard network config (no VLANs or tinkering, no pinning to specific AP, multicast DNS enabled). I do run separate SSID for these IOT like devices, but it all feeds into same network.

    • Edit: I do run on a different subnet than typical, 10.0.0.0/24. Shouldn't matter but worth noting.

Not sure if this helps anyone else but some datapoints.

@billy27607
Copy link

For what it is worth, I have a bit of data to add to the discussion. When my ratgdo devices were flashed with ratgdo-homekit, ping times to the devices were all over the place ranging from 10ms-60ms. There would be bursts of long latency times. Now that they are flashed with mqtt-ratgdio the ping times are constantly under 5ms. Same location, same network settings, etc. This suggests that the chip does not have enough juice to run the tcp stack and homekit in the Arduino framework. Note that I have like 50+ accessories in this particular home and there are 5 potential hubs to talk to.

I have a number of devices with 8266s that work well in this environment, but all are running code based in the Homekit SDK rather than the Arduino framework. These are nice little boards and there is no reason they be able to handle this with a good framework.

@thenewwazoo
Copy link
Collaborator

@billy27607 I believe you're basically correct, as evidenced by my work porting this codebase to use the Espressif HomeKit SDK. The ESP8266 is (in my opinion) simply not powerful enough to meet the use cases of this firmware.

I wrote a blog post about it, announcing that I'd be retiring from supporting this firmware, and recommending alternatives:

On Abandoning my RATGDO Native HomeKit Users

Not every maintainer of this repo shares my pessimism! @jgstroud deserves crazy applause for his work to improve reliability, and he's previously stated that he's going to keep working on it. Choosing to stick it out is a reasonable option, so long as it's done after having expectation set accordingly.

@billy27607
Copy link

billy27607 commented Mar 2, 2024

@thenewwazoo

I certainly understand and appreciate the position that you are in. Sometimes when the juice ain't worth the squeeze enough is enough. There is working solution for those of us who have purchased the boards.

I for one, appreciate all of the effort that you put into this. You should not worry about anythingat all.

@dkerr64
Copy link
Collaborator

dkerr64 commented Mar 3, 2024

The no-response error occurs for me "reliably" if the ratgdo is left to run for ~3 days, at which point it spontaneously reboots and on startup believes that no HomeKit device is paired. If I do not pair HomeKit and leave the ratgdo running, it runs for many more days (I don't know how many, I reset it after 6 days).

To me this smells awfully like a memory leak within the HomeKit code.

It will likely take time to find the cause of the problem, or may never be found. But what works reliably for me is to reboot the ratgdo before the error occurs. We could do this automatically.

In the PR #117 that is awaiting merge, I have added an automatic reboot every 24 hours (by default, configurable between 1 and 72 hours, with 0 meaning never). This is of course a nasty hack / work-around... but you would be surprised at the number of devices I have had to set to automatically reboot on a regular interval... not necessarily daily, but maybe weekly... commercial supported devices like cable modems, routers, etc. that would hang for no apparent reason, but work great as soon as rebooted.

@iyerusad
Copy link

iyerusad commented Mar 3, 2024

@billy27607 I believe you're basically correct, as evidenced by my work porting this codebase to use the Espressif HomeKit SDK. The ESP8266 is (in my opinion) simply not powerful enough to meet the use cases of this firmware.

I wrote a blog post about it, announcing that I'd be retiring from supporting this firmware, and recommending alternatives:

On Abandoning my RATGDO Native HomeKit Users

Not every maintainer of this repo shares my pessimism! @jgstroud deserves crazy applause for his work to improve reliability, and he's previously stated that he's going to keep working on it. Choosing to stick it out is a reasonable option, so long as it's done after having expectation set accordingly.

Completely understandable and a big thank you for all the hard work, this clearly is what a lot of people were interested in and you definitely got it quite far.

If there are any more expensive options on market would be helpful to list those in the blog post for those who prefer not to trade one buggy tool for a time-consuming homelab setup. Likewise, a fancier high end RatGDO that has more powerful SOC/memory/etc. would be also nice if it exists.


There is some curiosity/merit into trying to conclusively pin down what produces a "working" unit vs a crashy/buggy unit and/or root cause. It irks my brain to think there are, for a lack of a better word, 'golden' units vs general units, like an math formula that shouldn't work but does, and consistently.

Does the flasher/flashing method matter? This might be a common denominator that would explain larger majority of people having issues vs handful of people without this issue. I don't personally know if the initial flash on this platform matters, but my understanding is initial firmware flash is different than firmware upgrade.

  • To get on the HomeKit-RatGDO, I believe I flashed using a locally cloned version of https://github.com/esphome/esp-web-tools
    • Have to edit a couple of lines that reference the firmware file to get it working, as well as providing actual homekit firmware bin and manifest file
    • To get it to run via windows powershell used ChatGPT to convert the shell script to powershell and it ran, albeit with some benign errors. I then navigated to the dist/web folder and ran launched simple http server serving that directory and was able to flash
    • I think I tried another flasher app at some point but it didn't do wifi provisioning so I moved to the ESP Web Tools option
  • My guess is majority of people are flashing using Paul's hosted ESPHome flasher and then manually upgrading from web interface to homekit-ratgdo. I did run into the wifi bug multiple times using Paul's flasher but I think that is in general a ESPhome bug.
  • My very very limited understanding is the flasher does include some helper components from wifi and those versions may differ from what's hosted on Paul's site vs the latest from the repo

Probably nothing but I think that covers all the possible variations my consistently working build/setup could have from majority of everyone else's builds.

@edrikk
Copy link

edrikk commented Mar 4, 2024

I’ve been reading tonight, and this seems like something to consider given that it’s a further optimization (with a focus on ram) vs what thenewwazoo attempted in the other branch:

https://www.blessuisse.net/?_=%2FJKoss2%2FArduino-HomeKit-ESP8266%23KJWqMdlUlBnsIvkdRR%2BuhIT4

Some interesting reading there as well within the readme…

@billy27607
Copy link

It may be that the appearance of reliability depends on how much HomeKit network traffic, or how many hubs you have. As I mentioned in an earlier post, I have over 50 appliances, and five devices that can act as a hub. My ratgdo devices would consistently go non-responsive in two or three minutes. That is about the same time. It would take a hub to notice that the device had gone away.

It is certainly possible that the flashing method adds enough memory usage to make the problem worse. It also may be that how many hubs it has to talk to and register, taxes memory or makes a memory leak worse.

I have at least 15 devices that run on esp8266 hardware that work fine. The firmware running on them uses the SDK framework rather than the Arduino framework.

@dmsinger
Copy link

dmsinger commented Mar 4, 2024

Following up to some of the earlier messages on this issue:
I'm still on 0.9.0. The moment I was able to make sure my ratgdo didn't move around on the network, it's been stable.

@dkerr64
Copy link
Collaborator

dkerr64 commented Mar 7, 2024

This bug report is worth reading... esp8266/Arduino#8722

@dkerr64
Copy link
Collaborator

dkerr64 commented Mar 9, 2024

I mention above bug report because I have seen the same in ratgdo...

>>> [261837188] HomeKit: [Client 1073692780] Update Characteristics
>>> [261837205] HomeKit: [Client 1073692780] Get Characteristics
>>> [261837221] HomeKit: [Client 1073692780] Get Characteristics
>>> [261837237] HomeKit: [Client 1073692780] Get Characteristics
>>> [261837246] RATGDO: get current door state: 1
>>> [261837251] RATGDO: get target door state: 1
>>> [261837255] RATGDO: get obstruction: 0
>>> [261837259] RATGDO: get light state: Off

--------------- CUT HERE FOR EXCEPTION DECODER ---------------

Soft WDT reset

Exception (4):
epc1=0x40268ed6 epc2=0x00000000 epc3=0x00000000 excvaddr=0x00000000 depc=0x00000000

Level1Interrupt: Level-1 interrupt as indicated by set level-1 bits in the INTERRUPT register
  epc1=0x40268ed6 in pm_send_nullfunc at ??:?

>>>stack>>>

ctx: sys
sp: 3fffeda0 end: 3fffffb0 offset: 0160
3fffef00:  3fff07a0 f790a922 60000600 60000600  
3fffef10:  00000000 3ffeb318 00000000 00000000  
3fffef20:  40268cec 3ffeb318 3fff07a0 60000600  
3fffef30:  4026efe9 3ffeb318 3fff07a0 f790b9b6  
3fffef40:  4026f02e 3fffdab0 00000000 3fffdcb0  
3fffef50:  3fff07b0 3fffdad0 3fff1920 4023243a  
3fffef60:  40000f49 40000f49 ffffff00 40000f49  
3fffef70:  40000e19 0009215c bff00000 0000bfff  
3fffef80:  00000005 aa55aa55 0000007d 40105c25  
3fffef90:  40105c2b bff00000 0000bfff 61007264  
3fffefa0:  4010000d bff00000 0009215c 401000ab  
3fffefb0:  00000000 3fffef4c 00000000 3fffff28  
3fffefc0:  3fffffd0 00000000 00000000 feefeffe  
3fffefd0:  feefeffe feefeffe feefeffe feefeffe  
3fffefe0:  feefeffe feefeffe feefeffe feefeffe  
3fffeff0:  feefeffe feefeffe feefeffe feefeffe  
3ffff000:  feefeffe feefeffe feefeffe feefeffe  
3ffff010:  feefeffe feefeffe feefeffe feefeffe  
3ffff020:  feefeffe feefeffe feefeffe feefeffe  
3ffff030:  feefeffe feefeffe feefeffe feefeffe  
3ffff040:  feefeffe feefeffe feefeffe feefeffe  
3ffff050:  feefeffe feefeffe feefeffe feefeffe  
3ffff060:  feefeffe feefeffe feefeffe feefeffe  
3ffff070:  feefeffe feefeffe feefeffe feefeffe  
3ffff080:  feefeffe feefeffe feefeffe feefeffe  
3ffff090:  feefeffe feefeffe feefeffe feefeffe  
3ffff0a0:  feefeffe feefeffe feefeffe feefeffe  
3ffff0b0:  feefeffe feefeffe feefeffe feefeffe  
3ffff0c0:  feefeffe feefeffe feefeffe feefeffe  
3ffff0d0:  feefeffe feefeffe feefeffe feefeffe  
3ffff0e0:  feefeffe feefeffe feefeffe feefeffe  
3ffff0f0:  feefeffe feefeffe feefeffe feefeffe  
3ffff100:  feefeffe feefeffe feefeffe feefeffe  
3ffff110:  feefeffe feefeffe feefeffe feefeffe  
3ffff120:  feefeffe feefeffe feefeffe feefeffe  
3ffff130:  feefeffe feefeffe feefeffe feefeffe  
3ffff140:  feefeffe feefeffe feefeffe feefeffe  
3ffff150:  feefeffe feefeffe feefeffe feefeffe  
3ffff160:  feefeffe feefeffe feefeffe feefeffe  
3ffff170:  feefeffe feefeffe feefeffe feefeffe  
3ffff180:  feefeffe feefeffe feefeffe feefeffe  
3ffff190:  feefeffe feefeffe feefeffe feefeffe  
3ffff1a0:  feefeffe feefeffe feefeffe feefeffe  
3ffff1b0:  feefeffe feefeffe feefeffe feefeffe  
3ffff1c0:  feefeffe feefeffe feefeffe feefeffe  
3ffff1d0:  feefeffe feefeffe feefeffe feefeffe  
3ffff1e0:  feefeffe feefeffe feefeffe feefeffe  
3ffff1f0:  feefeffe feefeffe feefeffe feefeffe  
3ffff200:  feefeffe feefeffe feefeffe feefeffe  
3ffff210:  feefeffe feefeffe feefeffe feefeffe  
3ffff220:  feefeffe feefeffe feefeffe feefeffe  
3ffff230:  feefeffe feefeffe feefeffe feefeffe  
3ffff240:  feefeffe feefeffe feefeffe feefeffe  
3ffff250:  feefeffe feefeffe feefeffe feefeffe  
3ffff260:  feefeffe feefeffe feefeffe feefeffe  
3ffff270:  feefeffe feefeffe feefeffe feefeffe  
3ffff280:  feefeffe feefeffe feefeffe feefeffe  
3ffff290:  feefeffe feefeffe feefeffe feefeffe  
3ffff2a0:  feefeffe feefeffe feefeffe feefeffe  
3ffff2b0:  feefeffe feefeffe feefeffe feefeffe  
3ffff2c0:  feefeffe feefeffe feefeffe feefeffe  
3ffff2d0:  feefeffe feefeffe feefeffe feefeffe  
3ffff2e0:  feefeffe feefeffe feefeffe feefeffe  
3ffff2f0:  feefeffe feefeffe feefeffe feefeffe  
3ffff300:  feefeffe feefeffe feefeffe feefeffe  
3ffff310:  feefeffe feefeffe feefeffe feefeffe  
3ffff320:  feefeffe feefeffe feefeffe feefeffe  
3ffff330:  feefeffe feefeffe feefeffe feefeffe  
3ffff340:  feefeffe feefeffe feefeffe feefeffe  
3ffff350:  feefeffe feefeffe feefeffe feefeffe  
3ffff360:  feefeffe feefeffe feefeffe feefeffe  
3ffff370:  feefeffe feefeffe feefeffe feefeffe  
3ffff380:  feefeffe feefeffe feefeffe feefeffe  
3ffff390:  feefeffe feefeffe feefeffe feefeffe  
3ffff3a0:  feefeffe feefeffe feefeffe feefeffe  
3ffff3b0:  feefeffe feefeffe feefeffe feefeffe  
3ffff3c0:  feefeffe feefeffe feefeffe feefeffe  
3ffff3d0:  feefeffe feefeffe feefeffe feefeffe  
3ffff3e0:  feefeffe feefeffe feefeffe feefeffe  
3ffff3f0:  feefeffe feefeffe feefeffe feefeffe  
3ffff400:  feefeffe feefeffe feefeffe feefeffe  
3ffff410:  feefeffe feefeffe feefeffe feefeffe  
3ffff420:  feefeffe feefeffe feefeffe feefeffe  
3ffff430:  feefeffe feefeffe feefeffe feefeffe  
3ffff440:  feefeffe feefeffe feefeffe feefeffe  
3ffff450:  feefeffe feefeffe feefeffe feefeffe  
3ffff460:  feefeffe feefeffe feefeffe feefeffe  
3ffff470:  feefeffe feefeffe feefeffe feefeffe  
3ffff480:  feefeffe feefeffe feefeffe feefeffe  
3ffff490:  feefeffe feefeffe feefeffe feefeffe  
3ffff4a0:  feefeffe feefeffe feefeffe feefeffe  
3ffff4b0:  feefeffe feefeffe feefeffe feefeffe  
3ffff4c0:  00000005 00000000 00000020 4010087c  
3ffff4d0:  feefeffe feefeffe 00000005 4010288c  
3ffff4e0:  3ffeb5d5 40105b23 3ffef010 feefeffe  
3ffff4f0:  40103417 3ffef010 feefeffe feefeffe  
3ffff500:  3ffeb5d5 40105b23 3ffef010 401035f4  
3ffff510:  0000000c 00000000 00000020 4010087c  
3ffff520:  ffffffeb 989673b9 0000000c 4010288c  
3ffff530:  3ffeb5dc 401021d6 04000000 00000001  
3ffff540:  00000001 00000004 4021c42b 00000000  
3ffff550:  00000005 00000000 00000020 4010087c  
3ffff560:  00000005 00000000 00000005 4010288c  
3ffff570:  00000005 00000000 00000020 4010087c  
3ffff580:  40103417 3ffef038 00000005 4010288c  
3ffff590:  3ffeb5d5 40105b23 3ffef038 401035f4  
3ffff5a0:  40103417 3ffef038 00000000 4010288c  
3ffff5b0:  ffffffeb 24c0b5c2 3ffef96c 401035f4  
3ffff5c0:  00000005 00000000 00000020 4010087c  
3ffff5d0:  3ffeb5d5 40105b23 3ffef038 4010087c  
3ffff5e0:  40103417 3ffef038 00000005 4010288c  
3ffff5f0:  00000005 00000000 00000020 4010087c  
3ffff600:  00000000 00000000 0000001f 4010087c  
3ffff610:  3ffeb5d5 40105b23 3fffc228 401063bd  
3ffff620:  4000050c 3ffeef98 00000000 00000001  
3ffff630:  4026b557 00000030 00000000 ffffffff  
3ffff640:  3fff5fa8 3ffeec78 0000002c 00000010  
3ffff650:  00000024 16000000 007fffff f6b74ce6  
3ffff660:  3ffef420 00ffffff 000000c1 3ffeec78  
3ffff670:  3fff23c4 00000001 3fff5f8c 00000030  
3ffff680:  3ffebe74 40105b23 3ffef010 e387b14b  
3ffff690:  3ffebe5c 2c9f0300 4000050c 3fffc278  
3ffff6a0:  40100b34 3fffc200 0000000d 40100bc8  
3ffff6b0:  00000020 00000030 0000001a ffffffff  
3ffff6c0:  40210fea 01000000 00000000 00000022  
3ffff6d0:  4000050c 40100afc 3fffc258 4000050c  
3ffff6e0:  4025410e 00000030 00000018 ffffffff  
3ffff6f0:  00000000 3fff4fe4 4026b50b 00000001  
3ffff700:  ffffffff 00000000 3ffeb721 00000008  
3ffff710:  4026b55a 3ffeec78 3fff23c4 00000001  
3ffff720:  00000002 00000000 00000020 4010087c  
3ffff730:  00000005 00000005 00000002 4010288c  
3ffff740:  3ffeb5d2 4026a617 4026b50b 00000001  
3ffff750:  ffffffff 00000000 4026b50b 00000001  
3ffff760:  ffffffff 00000000 3ffeb721 00000008  
3ffff770:  4026b55a 3ffeeca0 3fff23c4 00000001  
3ffff780:  4026b666 3ffeeca0 3fff23c4 3ffeeca0  
3ffff790:  00000005 00000005 00000008 3fff66c0  
3ffff7a0:  3ffeb5d2 4026a617 3ffeeca0 3fff66a4  
3ffff7b0:  00000000 40253c13 3fff00c0 3fff23c4  
3ffff7c0:  00000000 00000002 00000000 3ffeeca0  
3ffff7d0:  3fff66da 40106167 3fff4460 3fff1d14  
3ffff7e0:  3fff266c 40244c65 3fff44b4 3fff1d14  
3ffff7f0:  00000000 40244c65 3fff4460 3fff1d14  
3ffff800:  00000000 00000001 3ffff8d0 00057d5a  
3ffff810:  3ffff91c 00e18f02 02d7c203 3fff1d2c  
3ffff820:  3fff1d2c 3fff66a4 3fff66a4 40244e97  
3ffff830:  3fff1d64 3fff66a4 00000000 402452f6  
3ffff840:  3fff207c 13ce03ac 3ffff90c f80ee3f2  
3ffff850:  f8d208e8 3d8036a4 01000000 3fff207c  
3ffff860:  3fff1d2c 3fff1d64 3fff2070 4024c878  
3ffff870:  000e1358 02806ad4 ee01c138 ff8a63bc  
3ffff880:  3fff66a4 20b5c082 507bb307 f1eb2508  
3ffff890:  b4809464 572c33dd 1716d596 3fff66a4  
3ffff8a0:  3fff66a4 3fff1d38 3fff1d2c 4024c94c  
3ffff8b0:  5fa30000 4455e0a0 b0745d3d 3ffffa12  
3ffff8c0:  3fff66e8 3fff1d2c 3fff4ae0 4024e3f4  
3ffff8d0:  00000000 3fff4adc 00000020 401013bc  
3ffff8e0:  00000014 3fff4adc 000000ff 00000000  
3ffff8f0:  00000006 3fff2158 00000020 40245c04  
3ffff900:  00000000 00000000 0000005a 3fff41dc  
3ffff910:  3fff1d2c 3fff4adc 3fff4194 4024e45c  
3ffff920:  3fff1d2c 00000000 00000000 05000000  
3ffff930:  00000000 57000000 efa43e56 4024e482  
3ffff940:  3fff1d2c 00000000 00000000 4024f3c3  
3ffff950:  00000003 00000017 3ffffa10 40249fde  
3ffff960:  3fff1d2c 3fff66a4 3fff41dc 00000000  
3ffff970:  0000003c 003f28dc e5e353f8 49073f6a  
3ffff980:  2095a1d0 740e0b60 42c4a874 00000002  
3ffff990:  00000001 00000000 3fff52c4 40226ffa  
3ffff9a0:  00000017 00000000 00000d3c 00000000  
3ffff9b0:  00000000 00000000 00000000 00000000  
3ffff9c0:  00000017 3fff5894 3fff5494 40228dc8  
3ffff9d0:  3ffffa10 00000002 00000000 00000000  
3ffff9e0:  3fff5494 3fff5894 000000b8 00000000  
3ffff9f0:  3fff5494 3fff5894 00000005 40228f54  
3ffffa00:  3ffffa12 3ffffa17 dcba29ac ee74dd79  
3ffffa10:  7e7e0005 83aa52a9 78323fcc 3baadc04  
3ffffa20:  100df0ae ccb48a0f 65bb6d89 e1457aac  
3ffffa30:  99a4af9e 9db376cf 9c09ae17 1d538de6  
3ffffa40:  974af6bf d1bc8c97 551a1538 05124995  
3ffffa50:  e5db76c9 50cd8523 29db0b0e 1efc7e4e  
3ffffa60:  ce487bb7 5d393511 5ce8257e 356e48db  
3ffffa70:  3c2162e0 119ef5da e68204c3 96d58516  
3ffffa80:  a6010eb3 8b200fe9 ce77d3c8 b99a7014  
3ffffa90:  0b8b3cfd 5d158f82 e88e90ed e8474655  
3ffffaa0:  10eb5ab5 4e4d2afd b440553e 785b9cb7  
3ffffab0:  e4563692 9c29111b 22adfd8c 9481cf28  
3ffffac0:  d330583b 6abb922f b06fdffc 063ec014  
3ffffad0:  a5c90005 7fd40000 f5f03063 3ffe8368  
3ffffae0:  3fff5f6d 40240000 00000000 40240651  
3ffffaf0:  00000001 00000003 4026b50b 00000001  
3ffffb00:  3fff5f7a 00000001 00000001 3fff5f4c  
3ffffb10:  00000000 40240000 3fff5f6c 00000001  
3ffffb20:  00000001 00000002 0000000f 00000000  
3ffffb30:  00000000 0000000f 00000000 40240000  
3ffffb40:  00000001 00000000 00000000 40240000  
3ffffb50:  3fff5f6d 00000000 00000000 3ffe8368  
3ffffb60:  00000000 00000000 3ffffd00 40241e34  
3ffffb70:  3ffffc6a 3ffffc93 00000000 4023c68d  
3ffffb80:  3ffffbf8 3ffffb94 3ffffb90 3ffe8368 <
3ffffb90:  00000001 3ffffc50 3ffffd00 40241e34  
3ffffba0:  40241d70 3ffffc6a 00000001 402422bc  
3ffffbb0:  40242350 40104ce6 3ffef798 3ffe8368  
3ffffbc0:  00000000 00000000 3ffffd30 40241e34  
3ffffbd0:  3ffe9f06 3ffffcc3 00000000 00000000  
3ffffbe0:  00000002 00000000 0000000a 3ffe8368  
3ffffbf0:  00000005 3ffe8368 3ffffd30 40241e34  
3ffffc00:  0000000b 00000000 00000010 00000000  
3ffffc10:  00000000 00000000 00000010 00000000  
3ffffc20:  00000000 00000000 00000010 00000000  
3ffffc30:  00000000 80000000 00000000 0000001b  
3ffffc40:  00000000 80000000 00000000 0000001b  
3ffffc50:  00000005 00000000 00000020 4010087c  
3ffffc60:  00000005 00000000 00000020 4010087c  
3ffffc70:  3ffeb5d5 40105b23 00000005 4010288c  
3ffffc80:  3ffeb5d5 40105b23 3ffef038 4010087c  
3ffffc90:  40103417 3ffef038 00000005 4010288c  
3ffffca0:  ffffffc5 f78e2c01 3ffef96c 401035f4  
3ffffcb0:  3ffebe68 00000000 00000000 00000100  
3ffffcc0:  ffffffc5 f78e2c01 40103aae 00000100  
3ffffcd0:  3ffebe68 7fffffff 00002200 00000001  
3ffffce0:  00000000 00000000 0000001f 4010087c  
3ffffcf0:  00000000 00000000 3fffc228 401063bd  
3ffffd00:  4000050c 2c9f0300 3fffc228 401063bd  
3ffffd10:  400043e1 00000030 00000010 ffffffff  
3ffffd20:  400044ab 3fffc718 3ffffe10 08000000  
3ffffd30:  60000200 08000000 08000000 00000000  
3ffffd40:  fffdffff 3fffc6fc 00275c40 3ffffe50  
3ffffd50:  00000050 003fb080 00000000 00000030  
3ffffd60:  00000030 003fb0a0 00000000 00000030  
3ffffd70:  3fffff08 00000032 00000005 4010288c  
3ffffd80:  3ffeb5d5 40105b23 3ffef038 3ffffec0  
3ffffd90:  40103417 3ffef038 00000001 00000030  
3ffffda0:  ffffffeb f78f3e76 3ffef96c 401035f4  
3ffffdb0:  3ffebe8c 00000000 00000000 00000001  
3ffffdc0:  ffffffeb f78f3e76 40103aae 00000100  
3ffffdd0:  3ffebe8c 7fffffff 00002200 00000001  
3ffffde0:  00000001 00000080 4000050c 3fffc278  
3ffffdf0:  3ffebe8c 3fffc200 00000022 f78f3e76  
3ffffe00:  3ffebe50 2c9f0300 4000050c 3fffc278  
3ffffe10:  00000000 400042db 00000022 4010087c  
3ffffe20:  40004b31 3ffffe50 00000050 003fb0d0  
3ffffe30:  40106f78 f78f4877 3fffc228 401063bd  
3ffffe40:  402064dc 3ffffec0 3ffffec0 402064fd  
3ffffe50:  00504148 44413301 32353733 30422d42  
3ffffe60:  342d3141 2d354639 37323338 3434302d  
3ffffe70:  38363144 44393336 66564d30 8dacbf06  
3ffffe80:  4283b7df 9dc6e452 b01b7973 4cbdd722  
3ffffe90:  7b74af39 55236a57 00000032 00000000  
3ffffea0:  000000b0 3ffffee7 3fffc228 3fff1920  
3ffffeb0:  3fffdad0 00000000 3fffff50 4022b0cb  
3ffffec0:  00000001 33444133 42323537 4130422d  
3ffffed0:  39342d31 382d3546 2d373233 44343430  
3ffffee0:  36383631 30443933 66564d00 8dacbf06  
3ffffef0:  4283b7df 9dc6e452 b01b7973 4cbdd722  
3fffff00:  7b74af39 55236a57 00000032 00000000  
3fffff10:  00000000 00000000 00000000 00000000  
3fffff20:  00000000 00000000 00000000 00000000  
3fffff30:  00000000 00000000 00000000 402276e2  
3fffff40:  00000000 00000000 00000000 00000001  
3fffff50:  00000002 3fffff70 00000001 4023a180  
3fffff60:  0000003c 003f6077 99db22d1 49478a4a  
3fffff70:  4023ae58 00000000 00001388 3fff1920  
3fffff80:  00000000 003f6077 9ae147ae 49478a4b  
3fffff90:  3fffdad0 3fff429c 3fff5894 3fff1920  
3fffffa0:  3fffdad0 00000000 3fff18f4 3fff1920  
<<<stack<<<

0x40268cec in pm_keep_active_enable at ??:?
0x4026efe9 in ets_timer_handler_isr at ??:?
0x4026f02e in ets_timer_handler_isr at ??:?
0x4023243a in loop_task(ETSEventTag*) at core_esp8266_main.cpp:?
0x40105c25 in call_user_start_local at ??:?
0x40105c2b in call_user_start_local at ??:?
0x4010000d in call_user_start at ??:?
0x401000ab in app_entry_redefinable at ??:?
0x4010087c in ets_post at ??:?
0x4010288c in pp_post at ??:?
0x40105b23 in lmacRxDone at ??:?
0x40103417 in rcReachRetryLimit at ??:?
0x40105b23 in lmacRxDone at ??:?
0x401035f4 in rcReachRetryLimit at ??:?
0x4010087c in ets_post at ??:?
0x4010288c in pp_post at ??:?
0x401021d6 in pp_soft_wdt_feed_local at ??:?
0x4021c42b in fe_mul__distinct at ??:?
0x4010087c in ets_post at ??:?
0x4010288c in pp_post at ??:?
0x4010087c in ets_post at ??:?
0x40103417 in rcReachRetryLimit at ??:?
0x4010288c in pp_post at ??:?
0x40105b23 in lmacRxDone at ??:?
0x401035f4 in rcReachRetryLimit at ??:?
0x40103417 in rcReachRetryLimit at ??:?
0x4010288c in pp_post at ??:?
0x401035f4 in rcReachRetryLimit at ??:?
0x4010087c in ets_post at ??:?
0x40105b23 in lmacRxDone at ??:?
0x4010087c in ets_post at ??:?
0x40103417 in rcReachRetryLimit at ??:?
0x4010288c in pp_post at ??:?
0x4010087c in ets_post at ??:?
0x4010087c in ets_post at ??:?
0x40105b23 in lmacRxDone at ??:?
0x401063bd in ets_timer_disarm at ??:?
0x4026b557 in pp_attach at ??:?
0x40105b23 in lmacRxDone at ??:?
0x40100b34 in interrupt_handler at ??:?
0x40100bc8 in interrupt_handler at ??:?
0x40210fea in wc_Poly1305Update at ??:?
0x40100afc in interrupt_handler at ??:?
0x4025410e in ieee80211_send_nulldata at ??:?
0x4026b50b in pp_attach at ??:?
0x4026b55a in pp_attach at ??:?
0x4010087c in ets_post at ??:?
0x4010288c in pp_post at ??:?
0x4026a617 in ppTxPkt at ??:?
0x4026b50b in pp_attach at ??:?
0x4026b50b in pp_attach at ??:?
0x4026b55a in pp_attach at ??:?
0x4026b666 in pp_attach at ??:?
0x4026a617 in ppTxPkt at ??:?
0x40253c13 in ieee80211_output_pbuf at ??:?
0x40106167 in wdt_feed at ??:?
0x40244c65 in glue2esp_linkoutput at /local/users/gauchard/arduino/arduino_esp8266/origin/tools/sdk/lwip2/builder/glue-esp/lwip-esp.c:301
0x40244c65 in glue2esp_linkoutput at /local/users/gauchard/arduino/arduino_esp8266/origin/tools/sdk/lwip2/builder/glue-esp/lwip-esp.c:301
0x40244e97 in new_linkoutput at /local/users/gauchard/arduino/arduino_esp8266/origin/tools/sdk/lwip2/builder/glue-lwip/lwip-git.c:272
0x402452f6 in ethernet_output at /local/users/gauchard/arduino/arduino_esp8266/origin/tools/sdk/lwip2/builder/lwip2-src/src/netif/ethernet.c:312
0x4024c878 in etharp_output_to_arp_index at /local/users/gauchard/arduino/arduino_esp8266/origin/tools/sdk/lwip2/builder/lwip2-src/src/core/ipv4/etharp.c:770
0x4024c94c in etharp_output_LWIP2 at /local/users/gauchard/arduino/arduino_esp8266/origin/tools/sdk/lwip2/builder/lwip2-src/src/core/ipv4/etharp.c:885
0x4024e3f4 in ip4_output_if_opt_src at /local/users/gauchard/arduino/arduino_esp8266/origin/tools/sdk/lwip2/builder/lwip2-src/src/core/ipv4/ip4.c:1765
0x401013bc in malloc at ??:?
0x40245c04 in do_memp_malloc_pool at /local/users/gauchard/arduino/arduino_esp8266/origin/tools/sdk/lwip2/builder/lwip2-src/src/core/memp.c:255
0x4024e45c in ip4_output_if_opt at /local/users/gauchard/arduino/arduino_esp8266/origin/tools/sdk/lwip2/builder/lwip2-src/src/core/ipv4/ip4.c:1577
0x4024e482 in ip4_output_if at /local/users/gauchard/arduino/arduino_esp8266/origin/tools/sdk/lwip2/builder/lwip2-src/src/core/ipv4/ip4.c:1550
0x4024f3c3 in ip_chksum_pseudo at /local/users/gauchard/arduino/arduino_esp8266/origin/tools/sdk/lwip2/builder/lwip2-src/src/core/inet_chksum.c:395
0x40249fde in tcp_output at /local/users/gauchard/arduino/arduino_esp8266/origin/tools/sdk/lwip2/builder/lwip2-src/src/core/tcp_out.c:1362
0x40226ffa in ClientContext::_write_from_source(char const*, unsigned int) at ??:?
0x40228dc8 in write(_client_context_t*, unsigned char*, int) at ??:?
0x40228f54 in client_send_encrypted_(_client_context_t*, unsigned char*, unsigned int) at ??:?
0x40240000 in _dtoa_r at /workdir/repo/newlib/newlib/libc/stdlib/dtoa.c:522
0x40240651 in _dtoa_r at /workdir/repo/newlib/newlib/libc/stdlib/dtoa.c:854
0x4026b50b in pp_attach at ??:?
0x40240000 in _dtoa_r at /workdir/repo/newlib/newlib/libc/stdlib/dtoa.c:522
0x40240000 in _dtoa_r at /workdir/repo/newlib/newlib/libc/stdlib/dtoa.c:522
0x40240000 in _dtoa_r at /workdir/repo/newlib/newlib/libc/stdlib/dtoa.c:522
0x40241e34 in __ssputs_r at /workdir/repo/newlib/newlib/libc/stdio/nano-vfprintf.c:232
0x4023c68d in __cvt at /workdir/repo/newlib/newlib/libc/stdio/nano-vfprintf_float.c:102
0x40241e34 in __ssputs_r at /workdir/repo/newlib/newlib/libc/stdio/nano-vfprintf.c:232
0x40241d70 in __ssputs_r at /workdir/repo/newlib/newlib/libc/stdio/nano-vfprintf.c:179
0x402422bc in _sungetc_r at /workdir/repo/newlib/newlib/libc/stdio/nano-vfscanf.c:183
0x40242350 in __ssrefill_r at /workdir/repo/newlib/newlib/libc/stdio/nano-vfscanf.c:230
0x40104ce6 in lmacRecycleMPDU at ??:?
0x40241e34 in __ssputs_r at /workdir/repo/newlib/newlib/libc/stdio/nano-vfprintf.c:232
0x40241e34 in __ssputs_r at /workdir/repo/newlib/newlib/libc/stdio/nano-vfprintf.c:232
0x4010087c in ets_post at ??:?
0x4010087c in ets_post at ??:?
0x40105b23 in lmacRxDone at ??:?
0x4010288c in pp_post at ??:?
0x40105b23 in lmacRxDone at ??:?
0x4010087c in ets_post at ??:?
0x40103417 in rcReachRetryLimit at ??:?
0x4010288c in pp_post at ??:?
0x401035f4 in rcReachRetryLimit at ??:?
0x40103aae in wDev_ProcessFiq at ??:?
0x4010087c in ets_post at ??:?
0x401063bd in ets_timer_disarm at ??:?
0x401063bd in ets_timer_disarm at ??:?
0x4010288c in pp_post at ??:?
0x40105b23 in lmacRxDone at ??:?
0x40103417 in rcReachRetryLimit at ??:?
0x401035f4 in rcReachRetryLimit at ??:?
0x40103aae in wDev_ProcessFiq at ??:?
0x4010087c in ets_post at ??:?
0x40106f78 in spi_flash_read at ??:?
0x401063bd in ets_timer_disarm at ??:?
0x402064dc in homekit_storage_next_pairing at ??:?
0x402064fd in homekit_storage_next_pairing at ??:?
0x4022b0cb in homekit_is_paired at ??:?
0x402276e2 in WiFiServer::accept() at ??:?
0x4023a180 in WiFiClient::operator bool() at ??:?
0x4023ae58 in DhcpServer::isRunning() const at ??:?


--------------- CUT HERE FOR EXCEPTION DECODER ---------------

@iyerusad
Copy link

iyerusad commented Mar 9, 2024

Some interesting tidbits from referenced discussion:

Issue Opened: 2022-Nov

I have about 70 devices online now running the almost the same firmware. I have tried everything i can think off.
I even went back to a old version from before the issue was noticed and the issue persisted.
It seems like the memory of the units get corrupted somehow, and then it starts.
I have tried clearing many of the nodes but it does not resolve the issue.

Same firmware with small modifications runs without issue on ESP32 boards.

if free RAM is less than 22KB, then ESP has many problems with stability. I'm using ESP mail client and AsynsWebSrv/TCP(5 webpagegs use Strings taken from 4 sensors, RTC and EEPROM). Those libraries use a lot of RAM and RAM defragmentation is more than 30-40%. I moved few voids to ICACHE_FLASH_ATTR. And now ESP can work for even 5 days. But after that SWDT and pm_send_nullfunc error will come. My sketch uses 707KB of flash and almost 62KB of RAM. After 2-3 days free RAM is about 10-11KB.

Some thoughts:

  • Adding a memory counter to webpage/debug output in case running out of memory?
  • Pinging Paul about considering new h/w with more memory board?

@dkerr64
Copy link
Collaborator

dkerr64 commented Mar 9, 2024

@iyerusad new h/w isn't going to solve for existing boards in the field. But I do like idea of reporting available memory in the web page. What is the right function to use to find that? I see the HomeKit code uses system_get_free_heap_size() to report. Is that the right function?

Thanks.

@dkerr64
Copy link
Collaborator

dkerr64 commented Mar 9, 2024

@iyerusad I have added reporting of free heap size to the user intterface in #127 so if that gets merged then it will display. In a few hours of observation I am consistently seeing around 25KB of free heap, which feels okay. It displays at the bottom of the web page.

I looked at also tracking stack usage, the ESP8266 apparently has a 4KB stack. When I did this however the stack usage (which remember, is at the point I measure, and not the maximum any other function could have used) was in the 100's of bytes not KB. It never changed. I don't know any way to find out what the highest stack usage has been at any point of time. Given that the info was not useful, I did not add it to the user interface.

@dkerr64
Copy link
Collaborator

dkerr64 commented Mar 9, 2024

Now, having said what I said above about monitoring stack size... looking at the crash dump I posted...

0x3fffffa0 - 0x3fffef00 = 0x10A0 which is 4256 which exceeds 4KB

So deep in homekit_storage_next_pairing() we ARE exceeding available stack. Humm.

@iyerusad
Copy link

@iyerusad I have added reporting of free heap size to the user intterface in #127 so if that gets merged then it will display.

Can you spin off heap size/memory stat into a separate pull request? Most likely to get faster merge.

@dkerr64
Copy link
Collaborator

dkerr64 commented Mar 10, 2024

@iyerusad I have added reporting of free heap size to the user intterface in #127 so if that gets merged then it will display.

Can you spin off heap size/memory stat into a separate pull request? Most likely to get faster merge.

Anything is possible, but I am not touching the code right now while I run a long-term test of #127 to see if I can last more than a few days without it crashing.

@jgstroud
Copy link
Collaborator

Fixed in #129

@plank3ffect
Copy link

plank3ffect commented Mar 29, 2024

I thought I had this same issue, but it was because I had three HomeKit hubs (two HomePod minis and an AppleTV). The one closest to my ratgdo was on a different network. If you have trouble with "no response" be sure to check all your hubs and ensure they are on the same network!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests