Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix youtube CSV playlist importer #4787

Open
wants to merge 5 commits into
base: master
Choose a base branch
from

Conversation

ThatMatrix
Copy link

@ThatMatrix ThatMatrix commented Jul 11, 2024

This PR is a follow up of this issue #3954 .

tldr: This PR aims to fix the Youtube CSV Importer due to Google Takeout changing their export format

I've recently used a invidious account, exported all my playlists data from Google Takeout. However, it seems that they changed their format again by putting most of the datas about a playlist in playlists.csv. Only to leave the bare minimum in the playlist specific files.

This PR changes this by putting default values where info could be extracted before and fixing the out of bound error that could be happening depending on the playlist format.

Here is the error encountered:

Title: `Index out of bounds (IndexError)`
Date: `2024-07-11T01:36:38Z`
Route: `/data_control?referer=%2Ffeed%2Fplaylists`
Version: `2024.04.27-eda7444 @ master`

<details>
<summary>Backtrace</summary>
<p>
   
Index out of bounds (IndexError)
  from /usr/share/crystal/src/json/any.cr:237:3 in 'update_data_control'
  from lib/kemal/src/kemal/route.cr:12:9 in '->'
  from src/invidious/helpers/handlers.cr:30:37 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:30:7 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:30:7 in 'call_next'
  from lib/kemal/src/kemal/filter_handler.cr:21:7 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:30:7 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:30:7 in 'call_next'
  from src/invidious/helpers/handlers.cr:94:12 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:30:7 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:30:7 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:30:7 in 'call_next'
  from src/ext/kemal_static_file_handler.cr:112:11 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:30:7 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:30:7 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:30:7 in 'call'
  from /usr/share/crystal/src/http/server/request_processor.cr:51:11 in 'handle_client'
  from /usr/share/crystal/src/fiber.cr:146:11 in 'run'
  from ???

</p>
</details>

I also give you an extract of what a Google Takeout playlist file looks like

Video ID,Playlist video creation timestamp
dQw4w9WgXcQ,2024-06-24T23:49:51+00:00
-tJYN-eG1zk,2024-06-23T20:30:24+00:00

@ThatMatrix
Copy link
Author

ThatMatrix commented Jul 11, 2024

I have tested importing a playlist with the final commit version and the provided docker-compose.yml file. It works with one of my GoogleTakeout playlist file.

@ThatMatrix
Copy link
Author

I took the liberty to add a default value/placeholders for the description as well as privacy fields in the meantime but I would like to receive advice on what to do with them if possible.
Thanks a lot

@ThatMatrix ThatMatrix marked this pull request as ready for review July 11, 2024 01:35
@ThatMatrix ThatMatrix requested a review from a team as a code owner July 11, 2024 01:35
@ThatMatrix ThatMatrix requested review from unixfox and removed request for a team July 11, 2024 01:35
@ThatMatrix ThatMatrix changed the title (DRAFT) Fix youtube CSV playlist importer Fix youtube CSV playlist importer Jul 11, 2024
@SamantazFox
Copy link
Member

Can you make a separate function for the new format?
Users who have exported when v1 was around should still be able to import their data!

@SamantazFox SamantazFox added the unfinished More work is needed on this PR, or on something this PR uses. label Jul 21, 2024
@PassiveLemon
Copy link

Importing works for me with this PR

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
unfinished More work is needed on this PR, or on something this PR uses.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants