Missing sleep state, long wake times and errors in log with MCU2 updated cars #3084
Comments
markusdd
changed the title
|
fyi HTTP 408's are just a way for the api to communicate the verhicle IS in standby. After which you'd send a 'wake_up' api-call. But, I'm also seeing increased drain caused by verhicle standby. Will try to rollback a version and see if that solves it. |
|
I'm seeing the same pattern with my Model S 2017 upgraded to MCU2 (version 2023.2.12) When teslamate is running I'm getting 54 minutes of offline and 19 minutes of online according to grafana. If I turn off teslamate and poll the API (using the command from this comment: #1288 (reply in thread) ) It seems that teslamate is keeping it awake for longer than necessary.
So the question might be: @adriankumpf do you remember off the top of your head? :) Is it getting vehicle_data or starting the streaming API ? |
|
I confirm. I have run similar experiments and is definetly teslamate keeping the car awake and creating this pattern. The phantom drain (and hence cost) of this is pretty substantial. But I would hate having to turn of teslamte. It would be good to get some support on debugging this. |
|
Still having teslamate off I did a poll on vehicle_data when the car came online. This extended the time :( And the normal state response (https://owner-api.teslamotors.com/api/1/vehicles/:id) doesn't contain any info if its a real online or just a i'm-just-checking-in online Hopefully the streaming API can connect for more info and not extend the wake time... I'm not really into that yet, so hopefully someone else can enlighten us :) |
|
I have done some tests using https://github.com/tdorssers/TeslaPy My code is here if somebody would like to try it out: #!/usr/bin/env python
import teslapy
from datetime import datetime
import time
import logging
state=None
code_verifier=None
#logging.basicConfig(level=logging.DEBUG)
# First stage
tesla = teslapy.Tesla('mymail@mail.com, retry=2, timeout=15)
if not tesla.authorized:
state = tesla.new_state()
code_verifier = tesla.new_code_verifier()
print('Use browser to login. Page Not Found will be shown at success.')
print('Open: ' + tesla.authorization_url(state=state, code_verifier=code_verifier))
tesla.close()
# Second stage
tesla = teslapy.Tesla('[email protected]', state=state, code_verifier=code_verifier)
if not tesla.authorized:
tesla.fetch_token(authorization_response=input('Enter URL after authentication: '))
vehicles = tesla.vehicle_list()
vehicle = vehicles[0]
current_state = vehicle.get_vehicle_summary()['state']
t_start = datetime.now()
print(str(datetime.now()) + ': ' + current_state)
last_state = current_state
#stream callback
def stream_cb(mydict):
print(mydict)
time.sleep(10)
exCount = 0
try:
while True:
try:
current_state = vehicle.get_vehicle_summary()['state']
exCount = 0
except Exception:
exCount = exCount + 1
print(str(datetime.now()) + ": Exception "+ str(exCount) +", sleep 90")
time.sleep(90)
if last_state != current_state:
t_end = datetime.now()
print(str(datetime.now()) + ': ' + '{:>7}'.format(last_state) + ' for {:6.2f}'.format((t_end - t_start).total_seconds() / 60) + ' min, new state: ' + current_state)
t_start = datetime.now()
# start the stream
vehicle.stream(stream_cb) # retry=12 seems to get more samples, but also locks up and no new states are detected
# print(vehicle.get_latest_vehicle_data())
# requests.exceptions.HTTPError: 404 Client Error: Not Found for url:
# #https://owner-api.teslamotors.com/api/1/vehicles/{id}/latest_vehicle_data
# even in online state
last_state = current_state
time.sleep(10)
except KeyboardInterrupt:
print('closing')
tesla.close()First test is with Teslamate off: I start a stream every time there is a change in state: At this point, when the car is in offline, I open the Tesla app and thereby wake the car for real: Notice the power says 0 instead of None. I then tried with teslamate on, teslapy poll 10 sec and stream retry of 12 (hope to catch a None before teslamate poll vehicle_data) It seems that power is None when the online is the 'fake' online. Maybe this could be used in Teslamate for handling this issue? |
|
That is a VERY interesting observvation and maybe the key to finally get rid of this phantom drain. @adriankumpf what do you say? I know very little about Elixir, might this be useful? |
|
I have the same issues awake/sleep issues with my Model S with upgraded MCU and I think it applies to more people indeed (#2556). |
|
As a call to action here: I would be willing to look into this but my experience with Elixir is absolutely zero, as well as with this code base. @adriankumpf do you have any time to address this based on what has been posted above or can you give at least a hint to the part of the code base where changes would likely need to be made? I do run a manual install so doing local changes and recompile would be quite easy to do for me. |
|
@micves and @sb1089 as Adrian is unfortunately not responding at all for weeks now I tried to have a look myself It then proceeds to poll the vehicle_data API So at this point we need to somehow recognize it is a fake online state and let the vehicle sleep. |
|
I also started looking into it myself last week. Never got around to write about it, but I'll try to describe my findings so far. I also started with the log 'Start / :online' but couldn't really figure out how everything was working. Elixir is quite different to my other known languages :) I got a development environment set up and added some more Log.info around the code. Old school debug, but it works. At some point I'll write my steps for that here: #1903 At 'Start / :online' it is too late to check if it is a fake online. The api call to vehicle_data has already been executed and woken the car up. When the car is offline this regularly gets the state: schedule_fetch ends up calling: which in case of online (also fake online) calls: that end up in here: I'm guessing 'Start / :online' is an event based on the fact that the state is online. The stream is started in the same place as 'Start / :online'. This is a issue, that it is started so late, if we want to see if the power is nil or 0 to determine if its a fake or a real online. This is kind of where I got stuck for now. And I'm a little scared of Elixir :D |
|
I can relate. I would not even say that I am completely devoid of understanding how functional programming works but Elixir is just somethign else. The syntax is...special. To be kind. I can follow what you are saying and agree with it. we probably somehow need to pick apart that last statement and make sure whenever power shows something similar to a NULL value we just act as if the car is asleep. |
|
and I just try it again: @adriankumpf if you don't have the time that is ok, but can you at least give us SOME hint where we need to look to apply a fix for this? I think we are pretty close here but Elixir makes it really hard. I have real issues understanding the control flow of this code :( |
|
Ok I now also put in some logging in my installation and it is a hard one to catch: Teslamate correctly suspends logging after some time after a drive and goes into stream only mode. As you can see there are no detailed logs, so I still have not found the location in the code where this call happens and first and foremost: WHY. This is frustrating... |
|
I think the errors are not handled gracefully because its a rare behavior that only MCU2 upgraded cars do. I have talked to a Tesla service center that says it normal behavior for a MCU2 upgraded car to be offline ~1hour and online for 2-3 minutes. Also with no 3rd party access. So that is something we cannot control. But we can fix the extension of online and going from subsystem online to real online. I think your API call starts at the line i also wrote in my last post. This however happens every 30 seconds so it will flood the log. So every 30 seconds you get into the fetch_with_unreachable_assumption: I'm guessing your log is just before the 'call(deps.api, :get_vehicle_with_state, [id])' But look at the line above: 'with {:ok, %Vehicle{state: "online"}} <- call(deps.api, :get_vehicle, [id]) do' As I see that line it calls the get_vehicle function that gets the state of the car without waking the car. |
|
Good point. Still, I am unsure how to distinguish between 'real online' and 'fake online'. |
|
Ok, so I modified the function a little and this is interesting: I basically pulled out the call and dumped the data we get State says online, but everything else is nil, nevertheless Teslamate tries to pull the full state. So...would the proper fix be to check if any of the other state are not nil and just plain ignore the state? |
|
What does it print when its a real online (open the app)? My experiments with TeslaPy did not show any difference in the equivalent to get_vehicle when it was a fake online compared to real online. The only thing I found different was the power in the message from streaming. I had to start the stream to get those. |
|
I tried with the app and it is no different. The car just went offline now, and I do not understand why and how this sequence happens.
|
|
I am still at a loss to understand how Teslamate even decides a car is now offline. That is all so confusing. I have now 'mined' the fetch functions with logging and I cannot understand it. Is it taking the 408 HTML response as a sleep indicator? What is triggering this 'fetching vehicle state...'? |
|
I think I slowly understand what is going on. I think when the car wakes up it is absolutely no problem to consider it online, even if it is the 'fake' online, as teslamate relatively quickly goes into suspended mode where it only polls the streaming API but not the state API. I think the issue is this: there are regulary polls to the non streaming API (not the waking one though) but when these fail (for whatever reason this happens regulary on MCU2 cars, probably when the car just wanted to go to sleep) then it triggers an uncoditional fetch_with_unreachable_assumption, the streaming API still says online, but it interrupts the 'suspended mode', hence hindering the car from truly sleeping: I think when we are in suspended mode, and there is no reason to leave that mode (and an API call fail isn't one of them, especially if the stream is still running), there must not be a fetch of vehicle_data. That is the whole point of suspend mode. So essentially my theory would be: if car in suspend mode, disallow those fetches. What do you say? |
|
I agree that it is no problem to be in online state as long as the get_vehicle_with_state is not called. Havent looked into suspended. The what, when and how. It does make sense not to call get_vehicle_with_state in suspended. But there are still cases that are not covered. I have testet a little with replaced the get_vehicle_with_state with the non-waking get_vehicle. Doing this replace of course makes everything weird, but I found a place to start the stream. At the moment it never goes to real online in teslamate, as this is apperantly based on get_vehicle_with_state. Something else to think about (robbing the sleep... :D ) is what about cars that originally have the MCU2. They might act differently :| |
|
suspended is not a state of the car. It is a strategy by Teslamate if nothing is going on (no_charge, no sentry, no service, no drive, no climate etc.) to not poll the API and hence let the car go to sleep. I have not yet understood though why this wakelock breaks. At some point the car just seems to 'force-sleep' and then we enter this mysterious sequence here: I have no idea why that happens, because my surrounding print outs all still say state' online'. The next wakeup then occurs presumably when we once again have the 'fake-online' going on which teslamate picks up on and then teh wakelock start anew... |
|
Here is another picture perfect example just now: Teslamate is in suspend mode. Just Streaming API running.
|
|
what is surprising is the case where it actually goes into offline. It uses a sifferent one of the fetch functions and the api response is not socket closed but nil. It then fetches again against non-vehicle_data URL and detmines the car is offline. What does the streaming API say in those circumstances? |
|
Another interesting observation: I changed the fetch schedule in suspend mode from 30 to 60 minutes. This way you will never see it go offline. |
|
As I understand it Teslamate state is based on what comes in as response from the non-vehicle_data URL. And I dont think the streaming API can keep the car awake, and also not wake it up. In the code I have running now, I try to start the stream whenever there is a result in from the fetch with non-vehicle_data And I think you have some double trouble going on. Is your car on stable internet? |
|
The car is on normal LTE mobile. Normal street parker...so no permanent Wifi for me. I might still be misunderstanding you: If the streaming API does not tell us anything about the state, why is it even being polled in suspend mode? This would seem non-sensical to me then. The more I dig into this the less sense it makes. |
|
Mine is also on LTE. And I agree, it has to work. Stream tells us indirectly if its a real or a fake online based on whether power is 0 or nil. It will detect if you start charging or driving (and I guess faster than if you have to wait 30 second for it to poll vehicle_data) |
|
ok that's a fair point. But still, based on my log above: Teslmate is not polling vehicle data, yet streaming API never puts power to nil. The car is not going to sleep as per this api. But as soon as teslmate, as per schedule, tries to pull the non vehicle_data URL, it gets those weird errors like socket closed. |
|
Here's some activity |
|
Hello we have a Tesla Model S100D, 2017, upgraded to MCU2 since 2022. the car never goes to sleep, as long as we have it. Other cars on the same account (MY and MX 2016 McU1) do go sleeping properly. can it be an issue with Teslamate? Now we have 1.28.5, but it was the same on al versions. the issue is with the car in the attachment in the bottom. The upoer one‘s issue was the caised by eevee app (now is the working ok). no other apps do connect to the car now |
|
@micves, can you merge the current teslamate to your fork? |
|
@USAFPride, just did a sync fork on github, so the images should be ready soon. I haven't updated and tested my own system yet. @wodnik7, I don't think its a Teslamate issue, but maybe start a new issues with logs. There are some other issues and discussions you can look through. It could be Sentry mode is always on. It could be the 12V battery needing replacement. |
|
@micves, all is going well on the updated image. |
|
I've been enjoying this fork for a few months now, not updated it to the latest version yet, but just noticed that for the first time my car is now reporting itself as being asleep rather than offline. I guess Tesla have changed something in their API? |
|
I have got this situation since this morning update to 2024.14.8, so it could be that Tesla has changed something in the software rather than the API, but I'm not an expert on this.
|
Since 2024.8.3 Tesla do report only offline for Intel MCUs since Spring Update for all ECUs |
I'm MCU2 (hence using this fork), but still on 2024.8.9 since 19th April. I started using TeslaMate on 19th February when I was on 2023.44.30.9. Ever since I've used TeslaMate my car has only reported "offline" never "asleep". It started reporting "asleep" instead of offline last night - and it's still reporting "asleep" - without any software changes to the car. |
|
2024.14.8
|
|
That's interesting. It also switched from offline to asleep at the same time as mine, but mine hasn't switched backed to offline since. |
|
@micves, can you merge the current teslamate to your fork? Have you had any luck sorting out the errors? |
|
@USAFPride, I just clicked the 'sync fork' on github.com (I haven't tested on own system) I have given the errors/pending-adjustments-in-tests a thought once in a while, but nothing more. I know it would make sense to get it into master at some point. Maybe I'll find some time after the summer holidays ;) |
|
This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions. |
JakobLichterfeld
added
area:tesla api
Now we just need @leandreeberhard to update his docker image of @micves 's code (pretty please!) |
Why are you not using the official docker images from the PR? To do so follow: https://docs.teslamate.org/docs/development#testing-with-our-ci-which-builds-the-docker-images-automatically-per-pr with |
I just tried and there is some permission error:
|
Ok, strange, perhaps the build got deleted after retention days. |
|
@micves, is your fork currently synched? |
|
@USAFPride, yes a few days ago. If you subscribe to the PR here #3262 you should get e-mails whenever there are some changes. |
Mine has switched back to showing Offline instead of Asleep, since 25th October. Car firmware hasn't changed. The Tesla App still shows the car as "Asleep" so it seems Tesla have just changed the API response again? 
It was quite nice having distinct offline and asleep states, as offline would mean the car has lost its LTE connection, which happened intermittently though relatively briefly. (Or it was rebooting) |
And after finally updating to 2024.38.6 it's now gone back to "asleep" state instead of "offline". Very odd. |
|
I still occasionally get "Asleep" instead of "Offline" as of software 2024.38.7. |
|
Interesting indeed. My car seems to have gone back to offline after its brief return to asleep following the last update!
I had a nice 5 months of the asleep condition, but that seems to be my lot! |
and others
Is there an existing issue for this?
What happened?
The car only switches between online and offline state, sleep does never occur.
Also, the log is littered with weird API errors and the wakeup times of the car are far too frequent and long, leading to bad phantom drain.
Also raised this in the discussion area: #3083
Expected Behavior
the car goes to sleep (not offline), there are no API query errors in the log and the car actually sleeps when it is not being used or charged
Steps To Reproduce
if only I knew, but I can give my config:
In the current example the below log output belongs to the last offline/online phase, error prints start happeneing at around the red arrow marker:
Relevant log output
Screenshots
ok....and now after a long wake period we are suddenly offline....that is weird.
Additional data
No response
Type of installation
Manual
Version
v1.27.2
The text was updated successfully, but these errors were encountered: