switch pyenv to uv #253

coder351 · 2024-07-22T07:19:10Z

The PR is follow up of #248 and resolves #240

The DockerFile also needed to be switched to uv. The pyqt5 library uses pyqt5-qt5 which only has wheels for macos. pyqt6 has wheels for linux (arm64 and x86-64). so switched to pyqt6 and pyqt6-sip.
The dbus-python library also had missing dependencies libdbus-glib-1-dev , libgirepository1.0-dev. The pkg-config also was failing due to missing pc files for python 311. Package python3-dev and python3.12-dev installs pc files python-3.12.pc and python-3.11-embed.pc. The python-3.12.pc files can also be used for building python 3.11. Full build is passing without errors.

userspace/openpilot_dependencies.sh

robin-reckmann · 2024-07-23T15:08:05Z

With your changes python and all packages from the pyproject.toml are installed to either /tmp/agnos or the home folder of the root user. These places are not accessible when you boot the image. Also userspace/files/profiles should be adapted accordingly.

robin-reckmann · 2024-07-23T15:02:10Z

Dockerfile.agnos

-    pyenv rehash
+    source .venv/bin/activate && \
+    .venv/bin/python --version && \
+    uv pip install pyqt6-sip==13.8.0 && \


Suggested change

uv pip install pyqt6-sip==13.8.0 && \

uv pip install pyqt5-sip==12.12.1 && \

I have reverted to pyqt5-sip.

Did the installation with uv like suggested above not work?

robin-reckmann · 2024-07-23T15:02:43Z

Dockerfile.agnos

+    source .venv/bin/activate && \
+    .venv/bin/python --version && \
+    uv pip install pyqt6-sip==13.8.0 && \
+    uv pip install pyqt6==6.7.1 --verbose --config-settings="--confirm-license="


Suggested change

uv pip install pyqt6==6.7.1 --verbose --config-settings="--confirm-license="

uv pip install pyqt5==5.15.9 --verbose --config-settings="--confirm-license=" --no-deps

pyqt5 can be installed from sources. I guess we should stay with pyqt5 for now.

I have reverted to pyqt5.

adeebshihadeh · 2024-07-23T16:26:52Z

Also needs to be PyQt5 since openpilot is on Qt5.

coder351 · 2024-07-24T18:00:39Z

With your changes python and all packages from the pyproject.toml are installed to either /tmp/agnos or the home folder of the root user. These places are not accessible when you boot the image. Also userspace/files/profiles should be adapted accordingly.

@robin-reckmann I checked using ./scripts/check-space.sh and anything installed using uv pip including packages from pyproject.toml are stored in /.venv/lib/. I have also adapted the various services and userspace/files/profile to use uv python installation (/.venv/bin/python).

I also noticed gabrielfalcao/pyenv-action@2f49ca7 is used by build_kernel.sh and is installing python2. Removing the github action causes the github build to fail. uv python does not have python version < 3.7, perhaps the necessary thing to do would be to also switch from python2 to python3. Also if possible, the generated system.img should be booted to test and confirm if the services are able to load correctly using uv based python.

andiradulescu · 2024-07-25T08:28:08Z

@coder351 - python2 in CI for build_kernel.sh has no connection with what happens inside Dockerfile.agnos. Just ignore it, the current kernel needs python2 to build.

Also, with #247 python2 is removed from CI.

I might be able to try out what you did here, after I get my PR merged.

coder351 · 2024-07-25T13:17:43Z

@adeebshihadeh The PR also resolves the #30 (build python with optimizations). After installing UV, I can check the build flags and uv python is installed using --enable-optimizations

CONFIG_ARGS = "'--build=x86_64-unknown-linux-gnu' '--host=x86_64-unknown-linux-gnu' '--prefix=/install' '--with-openssl=/tools/deps' '--with-system-expat' '--with-system-libmpdec' '--without-ensurepip' '--with-readline=editline' '--enable-shared' '--enable-optimizations' '--with-lto`

For claiming the bounty I am also posting before and after benchmark for python, before took 5:14 and after took 2:51.

Before using --enable-optimizations

===================================================================== slowest 10 durations ===================================================================== 17.09s call selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_025_FORD_EXPLORER_MK6 16.01s call selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_026_FORD_FOCUS_MK4 15.92s call selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_027_FORD_F_150_LIGHTNING_MK1 15.88s call selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_028_FORD_F_150_MK14 15.44s call selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_024_FORD_ESCAPE_MK4 15.43s call selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_029_FORD_MAVERICK_MK1 15.38s call selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_023_FORD_BRONCO_SPORT_MK1 15.14s call selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_030_FORD_MUSTANG_MACH_E_MK1 14.91s call selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_034_GENESIS_G80 14.59s call selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_099_KIA_FORTE =============================================================== 208 passed in 314.95s (0:05:14) ================================================================

After using --enable-optimizations

using uv python '--enable-optimizations' '--with-lto' ===================================================================== slowest 10 durations ===================================================================== 8.26s call selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_192_VOLKSWAGEN_ATLAS_MK1 8.20s call selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_150_SKODA_OCTAVIA_MK3 8.15s call selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_112_KIA_OPTIMA_H_G4_FL 8.14s call selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_198_VOLKSWAGEN_PASSAT_NMS 8.13s call selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_191_VOLKSWAGEN_ARTEON_MK1 8.11s call selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_195_VOLKSWAGEN_GOLF_MK7 8.06s call selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_149_SKODA_KODIAQ_MK1 7.97s call selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_199_VOLKSWAGEN_POLO_MK6 7.89s call selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_197_VOLKSWAGEN_PASSAT_MK8 7.88s call selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_151_SKODA_SUPERB_MK3 =============================================================== 208 passed in 171.89s (0:02:51) ================================================================

coder351 · 2024-07-25T14:09:56Z

@andiradulescu Thanks for helping test in the mean time, I suppose if anybody has instructions if possible on how to test booting the image without the actual hardware would be helpful.

adeebshihadeh · 2024-07-28T20:13:07Z

@andiradulescu Thanks for helping test in the mean time, I suppose if anybody has instructions if possible on how to test booting the image without the actual hardware would be helpful.

I'd recommend preparing an openpilot branch and reaching out on Discord for someone to give it a quick test.

coder351 · 2024-07-30T18:45:43Z

@andiradulescu since PR (#247) is merged, please do help test booting the image on physical device. Thanks.

andiradulescu · 2024-07-30T18:57:00Z

The PR looks good, will boot it and let you know how it goes.

andiradulescu · 2024-07-31T09:11:33Z

@coder351 can you push another commit that has [upload] inside the commit message - the commit can be empty - so it uploads the artifact for me to download and flash?

if you don't want to overthink it, just do:

git commit --allow-empty -m "[upload]"
git push

coder351 · 2024-08-03T16:09:35Z

@robin-reckmann I wanted to request to put PR #278 on hold while the pyenvtouv PR is merged because PRs like #277 are duplicating the effort. The pyenv PR already handled the FromAsCasing warning. Thanks.

coder351 · 2024-08-03T18:21:14Z

@adeebshihadeh The script openpilot_dependencies.sh, is duplicated and run on agnos-compiler and agnos-base.

agnos-builder/Dockerfile.agnos

Line 35 in 360675b

RUN /tmp/agnos/openpilot_dependencies.sh

agnos-builder/Dockerfile.agnos

Line 73 in 360675b

RUN /tmp/agnos/openpilot_dependencies.sh

The duplication is basically installing 71 libraries in agnos-base. The openpilot_dependencies.sh should only run on agnos-compiler and only necessary packages should be installed in agnos-base. After some testing I found out of 71 packages only 3 packages are necessary to build pyproject.toml and the rest of the DockerFile. I have placed the 3 packages namely portaudio19-dev, opencl-headers, gcc-arm-none-eabi in openpilot_python_dependencies.sh.
By removing the duplication, the build size is also reduced. In regards to removing build time dependencies i.e packages ending with -dev, I ran some tests and found the command apt-get purge lib*-dev is able to remove -dev packages and conserve about 0.5 gb of disk space from the final image. The command fails when run in github runner due to frontend-lock. I have also put back shims/pip, pointing PIP_PATH to uv based pip in venv. The PR can be merged and the removal of dev packages would reduce the agnos system image size which can be discussed in #225.

robin-reckmann · 2024-08-05T00:50:10Z

userspace/usr/comma/shims/pip

Permission are not the same anymore. Should be 755

robin-reckmann · 2024-08-05T00:52:54Z

Dockerfile.agnos

 RUN rm -rf /usr/share/icons/* && \
    rm -rf /var/lib/apt/lists/* && \
    rm -rf /home/$USERNAME/.cache && \
    rm -rf /root/.cache && \
-    apt-get clean
+    apt-get clean


Also the newline should probably be preserved. (I guess its a setting of your IDE/text editor). Same for the other files where this happened.

Dockerfile.agnos

robin-reckmann · 2024-08-05T00:57:04Z

Dockerfile.agnos

+ENV XDG_DATA_HOME="/usr/local"
+ENV PATH="$XDG_DATA_HOME/.cargo/bin:$PATH"
+
+COPY ./userspace/files/PyQt5-5.15.9-cp38-abi3-manylinux_2_17_aarch64.whl /tmp/agnos/


From this comment the way to go is to still build PyQt5 for now but make sure its somewhere in the build where other changes to the dockerfile dont break the cache. Also its unrelated to the uv introduction, so could we do this in another PR and remove it here?

according to docker build cache invalidation general rules,
"During cache lookup, cache is invalidated if the file metadata has changed for any of the files involved."
the docker cache should not bust as long as the wheel file metadata is not modified. I have already built the wheel and setup in dockerfile and using the pre-built PyQt5 wheel also provides the much needed build time optimization. Unless updating to pyqt6, I suggest to keep using the pre-built wheel.

robin-reckmann · 2024-08-05T00:59:05Z

userspace/files/profile


+export PATH="/usr/local/.cargo/bin:$PATH"
+export XDG_DATA_HOME="/usr/local"
+source $XDG_DATA_HOME/venv/bin/activate


@adeebshihadeh This will activate the venv for all sessions on the device. It is a different behavior compared to now, where you have to use pyenv exec. Is this ok?

@robin-reckmann pyenv exec would still use python "3.11.4", when run using pyenv exec

agnos-builder/userspace/files/profile

Line 36 in 360675b

export PYENV_VERSION="3.11.4"

unless the idea is to use different python versions across sessions, then activating venv ensures the python version 3.11.4 is consistent across sessions.

adeebshihadeh · 2024-08-07T23:22:09Z

With 24.04, we will no longer need to install a custom python (#262).

Gonna close this since 24.04 will be merged soon, but you're good to claim the bounty since the work was done.

coder351 · 2024-08-08T09:09:22Z

@adeebshihadeh Thanks, regarding the bounty, please highlight exactly how much I should be claiming, also please note regarding issue #30, I had posted before and after benchmarks, the uv based python is already using optimized python. So if the issue is still valid, perhaps the python portion of the bounty can also be factored in.

adeebshihadeh · 2024-08-09T17:21:01Z

@coder351 bounty is locked to you, but let's open a new PR since this one has gotten pretty messy. Looking over this, the remaining TODOs are:

I see uv has the optimizations enabled by default, but can we explicitly enable them so we're not relying on defaults?
PyQt5: either need a script to build/pull the wheel or build/install in the Docker build again
final test on device
- openpilot runs
- python, python3, pip, pip3, scons all work in a shell logged in as comma

coder351 added 4 commits July 17, 2024 11:39

switch pyenv to uv

b415c16

resolve issues

63b8458

remove uv upgrade

c60e49f

switch to uv

5e5b8ef

coder351 mentioned this pull request Jul 22, 2024

switch pyenv to uv #248

Merged

coder351 added 2 commits July 22, 2024 04:11

remove comment

d178ab8

remove python version

a3de1a2

adeebshihadeh reviewed Jul 22, 2024

View reviewed changes

userspace/openpilot_dependencies.sh Outdated Show resolved Hide resolved

coder351 added 4 commits July 22, 2024 14:41

remove python3.12-dev

f7e55c0

remove empty line

56067dd

use python3.pc

fd5f9db

test github workflow

2232fe4

robin-reckmann reviewed Jul 23, 2024

View reviewed changes

coder351 added 4 commits July 24, 2024 03:18

switch to pyqt-5 + use uv python for pkgconfig

1bd9278

remove references to pyenv

7bafbe4

remove references to pyenv

8bd41f1

add python2

db09a10

merge master

58ad598

merge master

d434034

coder351 force-pushed the pyenvtouv branch from a3c52b4 to d434034 Compare July 29, 2024 13:44

coder351 added 6 commits August 3, 2024 10:29

use PyQt5 wheel

85ef729

revert duplicate openpilot_dependencies.sh installed in agnos-base

87dd173

add sudo

71f46b5

revert dev deps purge

7763a31

remove openpilot_dependencies.sh from agnos-base

f40e27d

add -y to apt-get

6b22035

coder351 added 8 commits August 3, 2024 12:12

add pyproject opencl dep

8cdd398

add gcc-arm-none-eabi dep

753f242

test removing dev dependencies

11ed4cb

test removing dev dependencies

f75ba91

test purge using noninteractive

ff5ae89

test autoremove dependencies

10ea20a

revert autoremove

10e8f81

add pip to handle pip install

d7db0cf

robin-reckmann reviewed Aug 5, 2024

View reviewed changes

Dockerfile.agnos Show resolved Hide resolved

robin-reckmann reviewed Aug 5, 2024

View reviewed changes

coder351 added 2 commits August 5, 2024 06:25

add new line

3fbda24

rebuild

2e1f21d

adeebshihadeh closed this Aug 7, 2024

robin-reckmann mentioned this pull request Aug 9, 2024

[no merge yet] System python 3.12 and uv venv #295

Closed

adeebshihadeh reopened this Aug 9, 2024

adeebshihadeh closed this Aug 9, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

switch pyenv to uv #253

switch pyenv to uv #253

coder351 commented Jul 22, 2024

robin-reckmann commented Jul 23, 2024

robin-reckmann Jul 23, 2024

coder351 Jul 24, 2024 •

edited

Loading

robin-reckmann Jul 31, 2024

robin-reckmann Jul 23, 2024

robin-reckmann Jul 23, 2024

coder351 Jul 24, 2024

adeebshihadeh commented Jul 23, 2024

coder351 commented Jul 24, 2024 •

edited

Loading

andiradulescu commented Jul 25, 2024

coder351 commented Jul 25, 2024

coder351 commented Jul 25, 2024 •

edited

Loading

adeebshihadeh commented Jul 28, 2024

coder351 commented Jul 30, 2024

andiradulescu commented Jul 30, 2024

andiradulescu commented Jul 31, 2024 •

edited

Loading

coder351 commented Aug 3, 2024

coder351 commented Aug 3, 2024

robin-reckmann Aug 5, 2024

coder351 Aug 5, 2024

robin-reckmann Aug 5, 2024

coder351 Aug 5, 2024

robin-reckmann Aug 5, 2024

coder351 Aug 5, 2024

robin-reckmann Aug 5, 2024

coder351 Aug 5, 2024

adeebshihadeh commented Aug 7, 2024

coder351 commented Aug 8, 2024

adeebshihadeh commented Aug 9, 2024

	uv pip install pyqt6-sip==13.8.0 && \
	uv pip install pyqt5-sip==12.12.1 && \

	uv pip install pyqt6==6.7.1 --verbose --config-settings="--confirm-license="
	uv pip install pyqt5==5.15.9 --verbose --config-settings="--confirm-license=" --no-deps

switch pyenv to uv #253

switch pyenv to uv #253

Conversation

coder351 commented Jul 22, 2024

robin-reckmann commented Jul 23, 2024

Choose a reason for hiding this comment

coder351 Jul 24, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

adeebshihadeh commented Jul 23, 2024

coder351 commented Jul 24, 2024 • edited Loading

andiradulescu commented Jul 25, 2024

coder351 commented Jul 25, 2024

coder351 commented Jul 25, 2024 • edited Loading

adeebshihadeh commented Jul 28, 2024

coder351 commented Jul 30, 2024

andiradulescu commented Jul 30, 2024

andiradulescu commented Jul 31, 2024 • edited Loading

coder351 commented Aug 3, 2024

coder351 commented Aug 3, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

adeebshihadeh commented Aug 7, 2024

coder351 commented Aug 8, 2024

adeebshihadeh commented Aug 9, 2024

coder351 Jul 24, 2024 •

edited

Loading

coder351 commented Jul 24, 2024 •

edited

Loading

coder351 commented Jul 25, 2024 •

edited

Loading

andiradulescu commented Jul 31, 2024 •

edited

Loading