Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

switch pyenv to uv #253

Closed
wants to merge 43 commits into from
Closed
Show file tree
Hide file tree
Changes from 10 commits
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
b415c16
switch pyenv to uv
coder351 Jul 17, 2024
63b8458
resolve issues
coder351 Jul 18, 2024
c60e49f
remove uv upgrade
coder351 Jul 18, 2024
5e5b8ef
switch to uv
coder351 Jul 22, 2024
d178ab8
remove comment
coder351 Jul 22, 2024
a3de1a2
remove python version
coder351 Jul 22, 2024
f7e55c0
remove python3.12-dev
coder351 Jul 22, 2024
56067dd
remove empty line
coder351 Jul 22, 2024
fd5f9db
use python3.pc
coder351 Jul 22, 2024
2232fe4
test github workflow
coder351 Jul 22, 2024
1bd9278
switch to pyqt-5 + use uv python for pkgconfig
coder351 Jul 24, 2024
7bafbe4
remove references to pyenv
coder351 Jul 24, 2024
8bd41f1
remove references to pyenv
coder351 Jul 24, 2024
db09a10
add python2
coder351 Jul 24, 2024
58ad598
merge master
coder351 Jul 26, 2024
d434034
merge master
coder351 Jul 29, 2024
04d7a05
[upload]
coder351 Jul 31, 2024
4ec3a41
resolve docker warnings
coder351 Jul 31, 2024
c5f49df
[upload]
coder351 Jul 31, 2024
6571d23
setup uv to use /usr/local
coder351 Jul 31, 2024
d91713b
use uv to install pyqt5
coder351 Jul 31, 2024
8a36dae
use uv to install pip
coder351 Aug 1, 2024
5b4a0c9
use --seed to setup pip
coder351 Aug 1, 2024
f609d15
test build using --no-deps
coder351 Aug 1, 2024
920962a
test build pyqt5 using pyproject.toml
coder351 Aug 1, 2024
5b80d73
revert pyqt5 from pyproject.toml
coder351 Aug 1, 2024
ed2e2f1
[upload]
coder351 Aug 1, 2024
85ef729
use PyQt5 wheel
coder351 Aug 3, 2024
87dd173
revert duplicate openpilot_dependencies.sh installed in agnos-base
coder351 Aug 3, 2024
71f46b5
add sudo
coder351 Aug 3, 2024
7763a31
revert dev deps purge
coder351 Aug 3, 2024
f40e27d
remove openpilot_dependencies.sh from agnos-base
coder351 Aug 3, 2024
6b22035
add -y to apt-get
coder351 Aug 3, 2024
8cdd398
add pyproject opencl dep
coder351 Aug 3, 2024
753f242
add gcc-arm-none-eabi dep
coder351 Aug 3, 2024
11ed4cb
test removing dev dependencies
coder351 Aug 3, 2024
f75ba91
test removing dev dependencies
coder351 Aug 3, 2024
ff5ae89
test purge using noninteractive
coder351 Aug 3, 2024
10ea20a
test autoremove dependencies
coder351 Aug 3, 2024
10e8f81
revert autoremove
coder351 Aug 3, 2024
d7db0cf
add pip to handle pip install
coder351 Aug 3, 2024
3fbda24
add new line
coder351 Aug 5, 2024
2e1f21d
rebuild
coder351 Aug 5, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 17 additions & 13 deletions Dockerfile.agnos
Original file line number Diff line number Diff line change
Expand Up @@ -78,24 +78,28 @@ RUN /tmp/agnos/openpilot_python_dependencies.sh
COPY --from=agnos-compiler-ffmpeg /tmp/ffmpeg.deb /tmp/ffmpeg.deb
RUN cd /tmp && apt-get -o Dpkg::Options::="--force-overwrite" install -yq ./ffmpeg.deb

RUN export PATH="/usr/local/pyenv/bin:/usr/local/pyenv/shims:$PATH" && \
export PYENV_ROOT="/usr/local/pyenv" && \
eval "$(pyenv init -)" && \
eval "$(pyenv virtualenv-init -)" && \
RUN export PATH="$HOME/.cargo/bin:$PATH" && \
export UV_BIN="$HOME/.cargo/bin" && \
export MAKEFLAGS="-j$(nproc)" && \
pip install pyqt5-sip==12.12.1 && \
pip install pyqt5==5.15.9 --verbose --config-settings --confirm-license= && \
pyenv rehash
source .venv/bin/activate && \
.venv/bin/python --version && \
uv pip install pyqt6-sip==13.8.0 && \
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
uv pip install pyqt6-sip==13.8.0 && \
uv pip install pyqt5-sip==12.12.1 && \

Copy link
Contributor Author

@coder351 coder351 Jul 24, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have reverted to pyqt5-sip.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did the installation with uv like suggested above not work?

uv pip install pyqt6==6.7.1 --verbose --config-settings="--confirm-license="
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
uv pip install pyqt6==6.7.1 --verbose --config-settings="--confirm-license="
uv pip install pyqt5==5.15.9 --verbose --config-settings="--confirm-license=" --no-deps

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pyqt5 can be installed from sources. I guess we should stay with pyqt5 for now.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have reverted to pyqt5.


# Install openpilot python packages
COPY ./userspace/pyproject.toml ./userspace/uv.lock /tmp/agnos/
RUN export PATH="/usr/local/pyenv/bin:/usr/local/pyenv/shims:$PATH" && \
export PYENV_ROOT="/usr/local/pyenv" && \
eval "$(pyenv init -)" && \
eval "$(pyenv virtualenv-init -)" && \
RUN export PATH="$HOME/.cargo/bin:$PATH" && \
export UV_BIN="$HOME/.cargo/bin" && \
source .venv/bin/activate && \
cd /tmp/agnos && \
export PYOPENCL_CL_PRETEND_VERSION="2.0" && \
MAKEFLAGS="-j$(nproc)" UV_NO_CACHE=1 uv pip install --no-cache-dir --system .

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove empty line to prevent the following docker warning:
WARN: NoEmptyContinuation: Empty continuation line

pc="$(find /usr/lib/ -name python3.pc) " && \
pcpath=${pc%/*} && \
export PKG_CONFIG_PATH=$pcpath && \
cp $pcpath"/python3.pc" $pcpath"/python-3.11.pc" && \
cp $pcpath"/python3-embed.pc" $pcpath"/python-3.11-embed.pc" && \
MAKEFLAGS="-j$(nproc)" UV_NO_CACHE=1 uv pip install --no-cache-dir .

# Install nice to haves
COPY ./userspace/install_extras.sh /tmp/agnos/
Expand Down Expand Up @@ -215,4 +219,4 @@ RUN rm -rf /usr/share/icons/* && \
rm -rf /var/lib/apt/lists/* && \
rm -rf /home/$USERNAME/.cache && \
rm -rf /root/.cache && \
apt-get clean
apt-get clean
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also the newline should probably be preserved. (I guess its a setting of your IDE/text editor). Same for the other files where this happened.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done.

3 changes: 3 additions & 0 deletions userspace/openpilot_dependencies.sh
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,9 @@ apt-get install --no-install-recommends -yq \
ocl-icd-opencl-dev \
opencl-headers \
pkg-config \
libdbus-glib-1-dev \
libgirepository1.0-dev \
Comment on lines +59 to +60
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

these are just build time dependencies of PyQt5? if so, these shouldn't end up in the final image. see how agnos-compiler is used for some other stuff

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@adeebshihadeh not for PyQt5 but for dbus-python in pyproject.toml. In total about 71 build time dependencies are getting installed besides the dbus-python libraries. The build time dependencies can be removed by calling purge in DockerFile but I recommend to handle build-time optimizations in another PR.

python3-dev \
portaudio19-dev \
texinfo \
vnstat \
Expand Down
25 changes: 14 additions & 11 deletions userspace/openpilot_python_dependencies.sh
Original file line number Diff line number Diff line change
Expand Up @@ -2,20 +2,23 @@

echo "Installing python for openpilot"

# Install pyenv
export PYENV_ROOT="/usr/local/pyenv"
curl https://pyenv.run | bash
export PATH="$PYENV_ROOT/bin:$PATH"
eval "$(pyenv init -)"
echo "installing uv..."
curl -LsSf https://astral.sh/uv/install.sh | sh
UV_BIN='$HOME/.cargo/env'
ADD_PATH_CMD=". \"$UV_BIN\""
eval $ADD_PATH_CMD

PYTHON_VERSION="3.11.4"
if [ "$(uname -p)" == "aarch64" ]; then
pyenv install --verbose $PYTHON_VERSION
uv python install $PYTHON_VERSION
else
MAKEFLAGS="-j1" MAKE_OPTS="-j1" taskset --cpu-list 0 pyenv install --verbose $PYTHON_VERSION
MAKEFLAGS="-j1" MAKE_OPTS="-j1" taskset --cpu-list 0 uv python install --verbose $PYTHON_VERSION
fi

echo "Setting global python version"
pyenv global $PYTHON_VERSION

pip3 install --no-cache-dir --upgrade pip uv
# uv requires virtual env either managed or system before installing dependencies
uv venv --python-preference only-managed
# need to activate virtual env otherwise call to uv pip install throws error,
# error: No virtual or system environment found for path ...
source .venv/bin/activate
# install dependencies using managed python
uv pip install --python=$(which python) --no-cache-dir --upgrade pip
Loading