Throttling on youtube [kodi leia only]

[pitsi]

The following applies to systems that have python2 only and thus can only run youtube-dl through the addon (youtube-dl is python2 and python3 compatible, yt-dlp is python3 compatible only). For the same reason, it happens only on kodi versions which use python2, i.e. on kodi 18 and older versions.

So, one of the major issues youtube-dl has been facing since early October is youtube throttling any sort of streaming/downloading that is started through it. The issue was resolved in yt-dlp but is still present on the latest youtube-dl (2021.12.17), and although there is an available patch for it, it has not been merged yet.

"Visual" proof of the throttling. After sending a video to kodi (18.9) via sendtokodi (0.9.31 as of today), this is what bwm-ng captures for one second

  bwm-ng v0.6.1 (probing every 0.500s), press 'h' for help
  input: /proc/net/dev type: rate
  |         iface                   Rx                   Tx                Total
  ==============================================================================
             eth0:           0.00 KB/s            0.00 KB/s            0.00 KB/s
               lo:           0.00 KB/s            0.00 KB/s            0.00 KB/s
            wlan0:         132.74 KB/s            4.85 KB/s          137.59 KB/s
  ------------------------------------------------------------------------------
            total:         132.74 KB/s            4.85 KB/s          137.59 KB/s

and on the next second that drops to 0

  bwm-ng v0.6.1 (probing every 0.500s), press 'h' for help
  input: /proc/net/dev type: rate
  \         iface                   Rx                   Tx                Total
  ==============================================================================
             eth0:           0.00 KB/s            0.00 KB/s            0.00 KB/s
               lo:           0.00 KB/s            0.00 KB/s            0.00 KB/s
            wlan0:           0.00 KB/s            0.00 KB/s            0.00 KB/s
  ------------------------------------------------------------------------------
            total:           0.00 KB/s            0.00 KB/s            0.00 KB/s

This behavior alterates every second and makes every video unwatchable, even on low resolutions like 360p, because of the constant buffering and dropped frames.

As mentioned in a previous issue report (= the one for adding yt-dlp support) there is this fork of youtube-dl
https://github.com/ytdl-org/youtube-dl/
There has also been a pull request (#49), but I don't know if it has reached the addon yet.

[nullket]

Thanks for the recap. But have you seen #36 (comment)? I have submitted an addon version which should include a patched version of youtube-dl.
Edit:
Check if you have:

• Version 0.9.31 installed
• Restarted kodi (better be safe than sorry)
• The throttling persists

[pitsi]

I saw your comment on the previous issue, that is why I waited for ~1 day before opening this issue here, because I wanted to be sure I got the latest update of the addon.
I am definitely on 0.9.31+ as you can see here. The update to 0.9.32 was probably installed earlier today that I booted that installation and it was obviously restarted since then.

# cat .kodi/addons/plugin.video.sendtokodi/addon.xml | head -2
<?xml version='1.0' encoding='UTF-8'?>
<addon id="plugin.video.sendtokodi" name="sendtokodi" version="0.9.32" provider-name="firsttris">

If anyone is interested in testing it under kodi 19, he can switch the resolver to youtube-dl from settings and try. The result is the same and here is the output of bwm-ng as proof

  bwm-ng v0.6.3 (probing every 0.500s), press 'h' for help
  input: /proc/net/dev type: rate
  -         iface                   Rx                   Tx                Total
  ==============================================================================
               lo:           0.00 KB/s            0.00 KB/s            0.00 KB/s
             eth0:          52.95 KB/s            1.16 KB/s           54.11 KB/s
  ------------------------------------------------------------------------------
            total:          52.95 KB/s            1.16 KB/s           54.11 KB/s

Is there a parameter of youtube-dl and yt-dlp that can show me the extracted url which is sent for playback so as to compare them?

p.s. Please name yt-dlp as "yt-dlp" under settings and not as "youtube-dlp"! It is not my ocd, it is just easier to check what I have set it to :P

---edit
I found the parameter, it's --get-url. I compared the outputs of youtube-dl and yt-dlp and they are actually different. Stock youtube-dl 2021.12.17 vs stock yt-dlp 2021.12.27.

[pitsi]

The change in the title should become "youtube-dl only" because as I said above the issue can be reproduced on kodi 19 too, if you change the resolver to youtube-dl. And it is still there after the upgrade to 0.9.34 that reached my installation just now.

[nullket]

The change in the title should become "youtube-dl only" because as I said above the issue can be reproduced on kodi 19 too, if you change the resolver to youtube-dl.

kodi 19 defaults to youtube-dlp so it is a solved problem there. No need to use youtube-dl anymore on kodi matrix.

And it is still there after the upgrade to 0.9.34 that reached my installation just now.

Yes there was a bug with the auto updater. The updater actually checked out the master branch and not the one with the patch. I just fixed that. I got your comment the first time please be a bit patient: There are always new addon versions due to youtube-dlp becoming so much updates. Those new versions do not necessarily mean that anyone has fixed something else. If you want to check what has been changed on the addon simply have a look a the commit history.

Anyways, force kodi to update the addon in order to get 0.9.38+ where youtube-dl should be finally patched. Please report back if the issue is solved.

[pitsi]

Closing as fixed in 0.9.39 as mentioned. It reached my systems a few days ago and I tested it on both kodi 19 and 18.
However, despite the update, it was not working on the s805 tvbox running libreelec 9.2.6. I had to remove it completely and reinstall it to make it work.
The forementioned tvbox is not in my possession. I donated it to a family of 3 kids a ~6 months ago, so I had to visit the family in order to do any maintenance to it, thus the 2 day delay in answering. Sorry.

No need to use youtube-dl anymore on kodi matrix.

About that. It is easier for me to have kodi and the browser running on the same system while troubleshooting than having to look at 2 screens from 2 different systems. I just put the windows side by side and check what happens.
So, if some issue is related to youtube-dl related and not to kodi, I will use my kodi 19 installation and simply switch sendtokodi to youtube-dl for the issue only.

[pitsi]

Sorry for bringing this back to the surface but I have no way to test it otherwise and I need help.

I noticed earlier that videos from invidious and piped instances (I tested only the top 3 instances ) throttle as badly as youtube's once did when they are resolved via youtube-dl.
I know that sendtokodi pulls all commits made to youtube-dl, so its "version" of youtube-dl is definitely newer than 2021.12.17. And since that "version" is different than the one on my system (I am on 2021.12.17) by many commits, I can not test it on my system only. However, on all 3 systems I have and run kodi, the behavior is the same.
The weird thing is that videos from invidious instances used to play with no issues at all until last week that I last tried sending one to kodi (yes, I hate youtube's page, so I use an invidious instance).

To reproduce
a) make sure sendtokodi is installed (and configured) on both your browser and kodi
b) make sure sendtokodi is set to use youtube-dl (if you are on kodi 19 it will default to yt-dlp)
c) find a video on an invidious instance, e.g this one, and use sendtokodi on it
https://yewtu.be/watch?v=wWltASCJO-U

If it does happen on anyone else too, I will file an issue report on youtube-dl's github.

p.s. Playing from youtube directly has no issues.

[nullket]

If you encounter something like this please always test youtube-dl or yt-dlp as a standalone external program. Install the newest version of it and use the cli with the -f best option. As you can see youtube-dl is painfully slow. I additionally ran the test with yt-dlp and there does not seem to be such an issue.

# test with youtube-dl
youtube-dl -f best https://yewtu.be/watch?v=wWltASCJO-U
[youtube] wWltASCJO-U: Downloading webpage
[download] Destination: We’re running out of internet-wWltASCJO-U.mp4
[download]   0.7% of 88.07MiB at 65.87KiB/s ETA 22:39^C
ERROR: Interrupted by user

# test with y-dlp
yt-dlp -f best https://yewtu.be/watch?v=wWltASCJO-U
WARNING: "-f best" selects the best pre-merged format which is often not the best option.
         To let yt-dlp download and merge the best available formats, simply do not pass any format selection.
         If you know what you are doing and want only the best pre-merged format, use "-f b" instead to suppress this warning
[youtube] wWltASCJO-U: Downloading webpage
[youtube] wWltASCJO-U: Downloading android player API JSON
[info] wWltASCJO-U: Downloading 1 format(s): 22
[download] Destination: We’re running out of internet [wWltASCJO-U].mp4
[download]  45.4% of 88.07MiB at   10.95MiB/s ETA 00:04^C
ERROR: Interrupted by user

As this is not our issue, I will close it again. If you get youtube-dl fixed it will land here automatically.

[pitsi]

I know all that and that it is not your fault. I have both youtube-dl and yt-dlp installed for that reason, each on its latest available version (2021.12.17 and 2022.08.14 respectively).

However, the addon is different. For every single commit on any of the 2 projects, the addon gets automatically updated. This is, more or less, what I asked for at #55 and I am happy the way it was implemented. And, regarding youtube, this is the reason the addon's "version" does not throttle, but its latest stable version from upstream does.
Right now, youtube-dl of the addon is on 2021.12.17 + anything up to commit e6a836d54ca1d3cd02f3ee45ef707a46f23e829 which was merged 5 days ago. And in that term, youtube-dl is already fixed regarding youtube's throttling, as it plays youtube videos with no issues

Let's say that there are 100 commits from the release of 2021.12.17 up to this commit. The youtube-dl executable on my system does not have any of them, but I can run any sort of test and get the result you mentioned above because it is not patched.
The addon's youtube-dl, which is patched for youtube's throttling, does not have an executable that I can run and do any sort of testing. In fact, there is not a single thing in there that has the executable flag on it.
So, the only way of "testing" it is to ask if anyone else has the same issue. If one does, I will definitely report it as an issue on its github page.

On top of that, the issue I mention is not about youtube but about invidious. The wierd thing here is that youtube-dl (and yt-dlp) does not support invidious directly (= there is no invidious extractor) but redirects any url for it to youtube's extractor, which does practically the same.

[nullket]

For every single commit on any of the 2 projects, the addon gets automatically updated.

Our addons tries to update the both once a day. So every new commit at the time of checking will be merged

Let's say that there are 100 commits from the release of 2021.12.17 up to this commit. The youtube-dl executable on my system does not have any of them, but I can run any sort of test and get the result you mentioned above because it is not patched.

Then you should find a way to test the latest development version or get the author of youtube-dl to release the actual version. Let me help you with the first:

# install git if you do not have it
# lets clone it to a temp folder
git clone https://github.com/ytdl-org/youtube-dl.git /tmp/youtube-dl
# start the main (which is practically the same as the "executable" as you call it)
python /tmp/youtube-dl/youtube_dl/__main__.py -f best https://yewtu.be/watch?v=wWltASCJO-U
# seems like the issue is now fixed, maybe by the commit from an hour ago

to test the specific version of the lib you either checkout a specific commit in the above snippet or you can even use the very same file provided in the addon:

# change path to your kodi user folder
python /home/kodi/.kodi/addons/plugin.video.sendtokodi/lib/youtube_dl/__main__.py -f best https://yewtu.be/watch?v=wWltASCJO-U
# btw. slow and a lots of warnings for me, lets wait and see how this issue persists tomorrow after the newest youtube specific commits get automatically merged here 

So, the only way of "testing" it is to ask if anyone else has the same issue.

see above :)

if anyone else has the same issue. If one does, I will definitely report

in case somebody else would have the issue as well, you could still not tell if it is a bug within the addon or the resolver. For that you should need to test outside addons (and your point is valid in terms that of course other people confirming an issue is always helpful)

[pitsi]

I just tried what you suggested, cloning the main youtube-dl repo and running the python script as described.

Well, either youtube broke it again or the branch with the fixes is not the default one, because the same happens with youtube
Known invidious instance

$ python __main__.py -f best https://yewtu.be/watch?v=wWltASCJO-U
(lots of useless output)
[download] Destination: We_re_running_out_of_internet-wWltASCJO-U.mp4
[download]   0.8% of 88.03MiB at 55.26KiB/s ETA 26:59^C
ERROR: Interrupted by user

Youtube

$ python __main__.py -f best https://youtube.com/watch?v=wWltASCJO-U
(lots of useless output)
[download] Destination: We_re_running_out_of_internet-wWltASCJO-U.mp4
[download]   0.8% of 88.03MiB at 55.53KiB/s ETA 26:49^C
ERROR: Interrupted by user

On the other hand, the addon now works properly with invidious urls too, as you said above. So, what branch that contains the fixes does the addon follow?

[nullket]

Master branch 😅 so might do not have the the correct branch/commit checked out or a newer commit broke it again ;)

Anyway, I happy that you now got the tools to report „upstream issues“ in the YouTube-dl repo.