use git clone instead of github api

[neo773]

This PR removes usage of GitHub API which ran into rate limiting issues and instead uses the powerful git cli to clone only the specified path so it's highly efficient and avoids all rate limits.

From my testing it seems to be working.

b.mp4

/claim #905
/closes #905

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
screenpipe	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Dec 28, 2024 9:56pm

[louis030195]

does this work with local folders?

this won't work if user does not have git

[neo773]

Added local folder support.
git is installed by default on Mac and most Linux distros
will look into a solution for Windows.

[louis030195]

well

im sure this code will break in prod

using git to just download a zip seems overkill

is there a way to download the repo using request without rate limit?

also to take into account, in the future, we will have to download repo from other users, not monorepo

idk how obsidian does it https://github.com/obsidianmd/obsidian-releases (they have a store)

[neo773]

is there a way to download the repo using request without rate limit?

git exists solely for that purpose alone and it's the most efficient way, the code uses sparse checkout to only download the specified path.

also to take into account, in the future, we will have to download repo from other users, not monorepo

It will work fine with any repo it's literally just git.

using git to just download a zip seems overkill

problem isn't downloading a zip, it's the way git protocol works it's pretty complex and requires the spec file for it to know which files to download, that's why git cli is a good approach for this.

any other approach will have lot of complexity, maintenance and cost.

[neo773]

Looked into Obsidian approach they use GitHub releases to distribute the plugins, this requires lot of extra steps from developers

It fetches version from manifest.json
https://raw.githubusercontent.com/argenos/nldates-obsidian/refs/heads/master/manifest.json

Then it downloads from release object

[louis030195]

so its a bit like the ffmpeg-sidecar now, installing git at runtime? why not

we could also package git at build time in the files? maybe better - should have to change the install.sh, install.ps1 (CLI only) and pre_build.js (app+cli)

i think atm the pswh script have to be packaged as file so it's annoying

let's use git then

[neo773]

That's a good idea, I forgot we have installation scripts, will move the logic there.

[louis030195]

also alternatively https://github.com/rust-lang/git2-rs

@neo773 but might need some lib file too idk

[neo773]

I looked into libgit2 unfortunately it doesn't support sparse checkout there's an open issue on the GitHub page

[neo773]

Pushed a new commit
tested on windows virtual machine, its installing git as expected.

[louis030195]

seems you can remove the runtime installation of git in the code?

also keep in mind we use pre_build.js for the app build

[louis030195]

#1061

whats the status?

can you remove runtime installation of git? will cause issues