This project aims to develop a command line interface program to download all the relevant data from joined Telegram's groups and chats.
In order to do that, we heavily rely on Telethon library to interface with Telegram's official APIs. You can read more about its usage in their docs.
Before installing, you must first edit docker-compose.yml and config/config.yaml to provide the necessary credentials for the application. In order for everything to work, you should supply
- Pgadmin credentials (optional)
- Postgres database credentials
- Telegram API credentials
- Twitter API credentials (optional)
Then, you must initialize the necessary infrastructure and virtual environment.
docker-compose up -d
poetry install
poetry run alembic upgrade headThis program implements several functionalities. You can check the most up to date ones by typing poetry run python main.py --help.
By default, it runs on every joined group and chat. If you want to change that behavior, specify in the config/config.yaml file a whitelist or a blacklist.
To list all joined groups and channels, along with their ID, type
poetry run python main.py --list-dialogsTo download channels, users, messages and media from joined groups and channels, type
poetry run python main.pyTo download only messages from joined groups and channels, in order to speed up the process, type
poetry run python main.py --without-mediaTo download only users from joined groups and channels, type
poetry run python main.py --get-participantsTo download only media from already seen messages, type
poetry run python main.py --download-past-mediaTO BE IMPLEMENTED
In order to enumerate channels and groups to join, there are three strategies available:
- Manual search: manually search in the web for available invite links. Any channel/group joined through the official app will be shown next time you run this code.
- Twitter search: use twarc2 to automate the search for invite links in old tweets
- Telegram/Messages search: use the acquired Telegram messages to search for invite links
For now, every valid invite link found with the automated methods will be joined, without the option of manual approval by the user.
You can change the queries used to search twitter by modifying the search_queries.txt file.
poetry run python main.py --search-twitterpoetry run python main.py --search-messagesYou can export the collected data stored in the database as a pg_dump file.
poetry run python scripts.py export [--dest-file DEST_FILE] [--compress]These are two related repositories that heavily inspired the developing of telegram-bot.
- pushshift/telegram
- expectocode/telegram-export (archived)