Setup Feluda Locally for Development (v0.0.8 and before)

This documentation will help you setup an older version of feluda. It covers prerequisites, environment configuration, Docker setup , and steps on how to use operators, store, queue and feluda class. Detailed information on Workers, Testing, and Operators for v0.0.8 and earlier is available through the embedded hyperlinks.

Prerequisites

• Python
• Docker
• git

Step 1 - Cloning the Repository

First step is to clone the repo.

git clone https://github.com/tattle-made/feluda.git
cd feluda/

Step 2 - Setting up Environment Variables

• Create a new file called development.env in the src folder.
• Copy the contents from /src/.env-template into the development.env file
• Replace the value of MQ_USERNAME with the value of RABBITMQ_DEFAULT_USER and MQ_PASSWORD with the value of RABBITMQ_DEFAULT_PASS from docker-compose.yml.
• The development.env file should look like this now.

MQ_USERNAME=admin
MQ_PASSWORD=Admin123
MQ_HOST=rabbitmq
AWS_ACCESS_KEY_ID=XXXXXX
AWS_SECRET_ACCESS_KEY=XXXXXX
AWS_BUCKET=XXXXX
S3_CREDENTIALS_PATH = XXXXX/XXXXX
GOOGLE_APPLICATION_CREDENTIALS="credentials.json"
ES_HOST=es
ES_USERNAME=XXXXX
ES_PASSWORD=XXXXX
ES_IMG_INDEX=imgsearch
ES_TXT_INDEX=txtsearch
ES_VID_INDEX=vidsearch
WSGI_HOST=localhost
WSGI_DEBUG=True
PG_HOST=postgres
PG_DB=postgres
PG_USER=tattle
PG_PASS=tattle_pw

Step 3 - Building the Docker Containers

• The next step is to build the Docker Containers. Run the following command to bring up the docker containers.

docker-compose up

Tip

To just bring up the relevant docker containers, you can also run docker-compose up queue store api

• To verify if every service is up, visit the following URLs:

elasticsearch: http://localhost:9200/

rabbitmq UI: http://localhost:15672/

• Now to install the operators and see Feluda in action, we have to exec into the feluda_api docker container.

docker exec --user python -it feluda_api /bin/sh

Note

You can verify that you are running as the python user by running $ whoami

Step 4 - Installing Packages for Operators.

• After entering the feluda_api container, the next step is to install python libraries for the operators you want to see in action.
• Each operator has to be installed separately.
• Let's say you wish to install the image vec and video vec operator.

Note

The --no-deps flag is required to prevent hash mismatch due to auto-update issues

. ./venv/bin/activate
cd core/operators
pip install --require-hashes --no-deps -r image_vec_rep_resnet_requirements.txt 
pip install --require-hashes --no-deps -r vid_vec_rep_resnet_requirements.txt
pip install --require-hashes --no-deps -r audio_vec_embedding_requirements.txt
# cd out again for running tests
cd ..
cd ..
deactivate

• This will install all the required python libraries for each needed operator.
• To update the packages, see detailed documentation for it below.

Note

You can verify that the packages are installed by running $ pip list

Running the Test for the Operator

• To properly check if the operator has been installed and working properly, you can run the test for the operator.

cd core/operators
python -m unittest test_<operator>.py

• A detailed test documentation can be found here.

Step 5 - Seeing Feluda in Action

• The best way to see the working of Feluda is by running tests.
• Lets run a test that indexes (stores) and searches for media files.
• Make sure you are inside the feluda_api docker container, and in the src folder i.e. the /app folder in docker.

# start the server
python server.py
# keep the server running and in a new terminal
python -m unittest tests.endpoint.test_index_api_as_client

• You can verify that the server is running by opening: http://localhost:7000/
• This test will store (index) media files into Elasticsearch. Now we run the test to search for these files

# start the server (if not already done)
python server.py
# next in a new terminal
python -m unittest tests.endpoint.test_search_api_as_client

---

💻 Feluda is now successfully running and setup locally for development.

---

Updating Pacakges

1. Update packages in src/requirements.in or operator specific requirements file: src/core/operators/<operator>_requirements.in.
2. Use pip-compile to generate requirements.txt.
3. Follow the steps below to generate requirements.txt

• Use a custom tmp directory to avoid memory issues.
• If an operator defaults to a higher version than allowed by feluda core requirements.txt, manually edit the <operator>_requirements.txt to the compatible version. Then run pip install. If it runs without errors, the package version is valid for the operator.

cd src/
pip install --upgrade pip-tools
TMPDIR=<temp_dir> pip-compile --verbose --allow-unsafe --generate-hashes --emit-index-url --emit-find-links requirements.in

# Updating operators
cd src/core/operators/
TMPDIR=<temp_dir> pip-compile --verbose --allow-unsafe --generate-hashes --emit-index-url --emit-find-links <OPERATOR>_requirements.in

# Example
# The link for torch is required since PyPi only hosts the GPU version of torch packages.
TMPDIR=<temp_dir> pip-compile --verbose --allow-unsafe --generate-hashes --emit-index-url --emit-find-links --find-links https://download.pytorch.org/whl/torch_stable.html vid_vec_rep_resnet_requirements.in
TMPDIR=<temp_dir> pip-compile --verbose --allow-unsafe --generate-hashes --emit-index-url --emit-find-links --find-links https://download.pytorch.org/whl/torch_stable.html audio_vec_embedding_requirements.in

Modify generated requirements.txt for platform specific torch packages

Note

Update the command to match python docker image version

# Download package to find hash - you will get an error message if the package has been previously downloaded without the hash. The hash value will be printed in the message. Use that hash

$ pip download --no-deps --require-hashes --python-version 311 --implementation cp --abi cp311 --platform linux_x86_64 --find-links https://download.pytorch.org/whl/torch_stable.html torch==2.2.0+cpu
$ pip download --no-deps --require-hashes --python-version 311 --implementation cp --abi cp311 --platform linux_x86_64 --find-links https://download.pytorch.org/whl/torch_stable.html torchvision==0.17.0+cpu
$ pip download --no-deps --require-hashes --python-version 311 --implementation cp --abi cp311 --platform manylinux2014_aarch64 --find-links https://download.pytorch.org/whl/cpu torch==2.2.0
$ pip download --no-deps --require-hashes --python-version 311 --implementation cp --abi cp311 --platform manylinux2014_aarch64 --find-links https://download.pytorch.org/whl/cpu torchvision==0.17.0

Replace the torch package lines from requirement.txt with the following (depending upon the generated hash values above)

# For arm64 architecture
--find-links https://download.pytorch.org/whl/cpu
torch==2.2.0; platform_machine=='aarch64' \
    --hash=sha256:9328e3c1ce628a281d2707526b4d1080eae7c4afab4f81cea75bde1f9441dc78
    # via
    #   -r vid_vec_rep_resnet_requirements.in
    #   torchvision
torchvision==0.17.0; platform_machine=='aarch64' \
    --hash=sha256:3d2e9552d72e4037f2db6f7d97989a2e2f95763aa1861963a3faf521bb1610c4 \
    # via -r vid_vec_rep_resnet_requirements.in

# For amd64 architecture
--find-links https://download.pytorch.org/whl/torch_stable.html
torch==2.2.0+cpu; platform_machine=='x86_64' \
    --hash=sha256:15a657038eea92ac5db6ab97b30bd4b5345741b49553b2a7e552e80001297124 \
    --hash=sha256:15e05748815545b6eb99196c0219822b210a5eff0dc194997a283534b8c98d7c \
    --hash=sha256:2a8ff4440c1f024ad7982018c378470d2ae0a72f2bc269a22b1a677e09bdd3b1 \
    --hash=sha256:4ddaf3393f5123da4a83a53f98fb9c9c64c53d0061da3c7243f982cdfe9eb888 \
    --hash=sha256:58194066e594cd8aff27ddb746399d040900cc0e8a331d67ea98499777fa4d31 \
    --hash=sha256:5b40dc66914c02d564365f991ec9a6b18cbaa586610e3b160ef559b2ce18c6c8 \
    --hash=sha256:5f907293f5a58619c1c520380f17641f76400a136474a4b1a66c363d2563fe5e \
    --hash=sha256:8258824bec0181e01a086aef5809016116a97626af2dcbf932d4e0b192d9c1b8 \
    --hash=sha256:d053976a4f9ca3bace6e4191e0b6e0bcffbc29f70d419b14d01228b371335467 \
    --hash=sha256:f72e7ce8010aa8797665ff6c4c1d259c28f3a51f332762d9de77f8a20277817f
    # via
    #   -r vid_vec_rep_resnet_requirements.in
    #   torchvision
torchvision==0.17.0+cpu; platform_machine=='x86_64' \
    --hash=sha256:00e88e9483e52f99fc61a73941b6ef0b59d031930276fc220ee8973170f305ff \
    --hash=sha256:04e72249add0e5a0fc3d06a876833651e77eb6c3c3f9276e70d9bd67804c8549 \
    --hash=sha256:39d3b3a80c63d18594e81829fdbd6108512dff98fa17156c7bec59133a0c1173 \
    --hash=sha256:55660c67bd8d5b777984655116b75070c73d37ce64175a8120cb59010039fd7f \
    --hash=sha256:569ebc5f47bb765ae73cd380ace01ddcb074c67df05d7f15f5ddd0fa3062881a \
    --hash=sha256:701d7fcfdd8ed206dcb71774190152f0a2d6c999ad7cee277fc5a71a943ae64d \
    --hash=sha256:b683d52753c5579a5b0250d7976deada17deab646071da289bd598d1af4877e0 \
    --hash=sha256:bb787aab6daf2d72600c14cd7c3c11459701dc5fac07e790e0335777e20b39df \
    --hash=sha256:da83b8a14d1b0579b1119e24272b0c7bf3e9ad14297bca87184d02c12d210501 \
    --hash=sha256:eb1e9d061c528c8bb40436d445599ca05fa997701ac395db3aaec5cb7660b6ee
    # via -r vid_vec_rep_resnet_requirements.in

Updating specific packages in requirements.txt

This is useful to update dependencies e.g. when using pip-audit

TMPDIR=<temp_dir> pip-compile --verbose --allow-unsafe --generate-hashes --find-links https://download.pytorch.org/whl/torch_stable.html --upgrade-package <package>==<version> --upgrade-package <package>