This repo has both the scraping and matching utilities to find mangas which are close in content to others. The idea is to create a recommendation system outside MangaDex since there isn't one and thus allow for users to discover other content. Right now it is pretty rudimentary, but seems to give ok results most of the time. If you want to use it, please download Neko and enable recommendations inside the app or use the Chrome Extension.
Client was generated using swagger. You will need to setup a golang workspace, and then run the following commands. Only manga need to be downloaded / scraped from mangadex to be able to perform similar manga identification.
go get golang.org/x/oauth2
go get github.com/antihax/optional
go get github.com/james-bowman/nlp
go get github.com/caneroj1/stemmer
go get github.com/PuerkitoBio/goquery
- Run
./1_cache_mangasto get local list of all mangas on the site (28 hours for 73k manga) - Run
./2_calc_similarto generate similarity index between manga (6 hours) - Run
./3_calc_mappingsto export external ID to mangadex UUID mapping files (2 minutes) - Run
./4_calc_mu_idsconvert MangaUpdate ids into uniform "new" ids for their new api (27 hours) - Run
python utils/create_sqlite.pyto generate mapping file for Neko
| Key | Related site | URL | URL details |
|---|---|---|---|
| al | anilist | https://anilist.co/manga/`{id}` | Stored as id |
| ap | animeplanet | https://www.anime-planet.com/manga/`{slug}` | Stored as slug |
| bw | bookwalker.jp | https://bookwalker.jp/`{slug}` | Stored has "series/{id}" |
| mu | mangaupdates | https://www.mangaupdates.com/series.html?id=`{id}` | Stored has id |
| nu | novelupdates | https://www.novelupdates.com/series/`{slug}` | Stored has slug |
| kt | kitsu.io | https://kitsu.io/api/edge/manga/`{id}` or https://kitsu.io/api/edge/manga?filter[slug]={slug} | If integer, use id version of the URL, otherwise use slug one |
| amz | amazon | N/A | Stored as full URL |
| ebj | ebookjapan | N/A | Stored as full URL |
| mal | myanimelist | https://myanimelist.net/manga/{id} | Store as id |
| raw | N/A | N/A | Stored as full URL, untranslated stuff URL (original language) |
| engtl | N/A | N/A | Stored as full URL, official english licenced URL |