Skip to content

Commit

Permalink
Modified download_deps.py (#2747)
Browse files Browse the repository at this point in the history
### What problem does this PR solve?

Modified download_deps.py

### Type of change

- [ ] Bug Fix (non-breaking change which fixes an issue)
- [ ] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [x] Other (please describe): CI
  • Loading branch information
yuzhichang authored Oct 8, 2024
1 parent 04ff9cd commit 2e33ed3
Show file tree
Hide file tree
Showing 3 changed files with 15 additions and 3 deletions.
2 changes: 1 addition & 1 deletion .github/workflows/tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ jobs:
- name: Build ragflow:dev-slim
run: |
RUNNER_WORKSPACE_PREFIX=${RUNNER_WORKSPACE_PREFIX:-$HOME}
cp -r ${RUNNER_WORKSPACE_PREFIX}/huggingface.co ${RUNNER_WORKSPACE_PREFIX}/nltk_data .
cp -r ${RUNNER_WORKSPACE_PREFIX}/huggingface.co ${RUNNER_WORKSPACE_PREFIX}/nltk_data ${RUNNER_WORKSPACE_PREFIX}/libssl*.deb .
sudo docker pull ubuntu:24.04
sudo docker build -f Dockerfile.slim -t infiniflow/ragflow:dev-slim .
Expand Down
5 changes: 3 additions & 2 deletions Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -24,8 +24,9 @@ RUN --mount=type=cache,id=ragflow_base_apt,target=/var/cache/apt,sharing=locked

# https://forum.aspose.com/t/aspose-slides-for-net-no-usable-version-of-libssl-found-with-linux-server/271344/13
# aspose-slides on linux/arm64 is unavailable
RUN if [ "${ARCH}" = "amd64" ]; then \
curl -o libssl1.deb http://archive.ubuntu.com/ubuntu/pool/main/o/openssl/libssl1.1_1.1.1f-1ubuntu2_amd64.deb && dpkg -i libssl1.deb && rm -f libssl1.deb; \
RUN --mount=type=bind,source=libssl1.1_1.1.1f-1ubuntu2_amd64.deb,target=/root/libssl1.1_1.1.1f-1ubuntu2_amd64.deb \
if [ "${ARCH}" = "amd64" ]; then \
dpkg -i /root/libssl1.1_1.1.1f-1ubuntu2_amd64.deb; \
fi

ENV PYTHONDONTWRITEBYTECODE=1 DOTNET_SYSTEM_GLOBALIZATION_INVARIANT=1
Expand Down
11 changes: 11 additions & 0 deletions download_deps.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,11 @@
from huggingface_hub import snapshot_download
import nltk
import os
import urllib.request

urls = [
"http://archive.ubuntu.com/ubuntu/pool/main/o/openssl/libssl1.1_1.1.1f-1ubuntu2_amd64.deb",
]

repos = [
"InfiniFlow/text_concat_xgb_v1.0",
Expand All @@ -20,6 +25,12 @@ def download_model(repo_id):


if __name__ == "__main__":
for url in urls:
filename = url.split("/")[-1]
print(f"Downloading {url}...")
if not os.path.exists(filename):
urllib.request.urlretrieve(url, filename)

local_dir = os.path.abspath('nltk_data')
for data in ['wordnet', 'punkt', 'punkt_tab']:
print(f"Downloading nltk {data}...")
Expand Down

0 comments on commit 2e33ed3

Please sign in to comment.