Building wheel for aarch platform

[MatsMoll]

Releasing version for aarch in order to support Docker on M1 - related to #535

[wangxiaoying]

Hi @MatsMoll , have you tried to run this release script? I think we had some issues with this, so we comment it out (mentioned here but I kind of forgot the reason of failure).

[mkhalid12]

Hi @MatsMoll any update regarding this issue when it will merge? Are any temporary fixes available?

[MatsMoll]

I got focused on some other stuff and forgot to answer.

However, I have not tried to run the release script @wangxiaoying. As I am unsure how to test it locally. So I guess I mainly wanted to start a new discussion about this, and hopefully look into how to fix it. But I never got to the last part

[duvenagep]

We currently build from source using a multi-stage docker build and has been running on arch now for +- 3months without problem! Here we describe that multi-stage build. Perhaps a multi-stage github action could solve this problem?

[Ed1123]

Hi, is there something preventing from merging this?

[evbo]

@wangxiaoying how are you feeling about releasing this? Seems it would close quite a few issues:
#186
#214
#240
#386
#535
#545

[j0bekt01]

@wangxiaoying how are you feeling about releasing this? Seems it would close quite a few issues: #186 #214 #240 #386 #535 #545

Any update on this?

[wangxiaoying]

I got focused on some other stuff and forgot to answer.

However, I have not tried to run the release script @wangxiaoying. As I am unsure how to test it locally. So I guess I mainly wanted to start a new discussion about this, and hopefully look into how to fix it. But I never got to the last part

IC, I tried to test it in prerelease ci: https://github.com/sfu-db/connector-x/actions/runs/8014425468,
but it seems like the error still exists: https://github.com/sfu-db/connector-x/actions/runs/8014425468/job/21892997456#step:13:17

Any idea how to fix this issue?

[evbo]

@wangxiaoying can you please try upgrading maturin-action version?:
https://github.com/sfu-db/connector-x/actions/runs/8014425468/workflow#L48

Currently the latest version is 1.41.0 but you are using 0.14.15. Maybe that issue has long been fixed?

[duvenagep]

@wangxiaoying seeing that the CI tests now pass, is there anything preventing this PR from being merged?

[wangxiaoying]

@wangxiaoying can you please try upgrading maturin-action version?: https://github.com/sfu-db/connector-x/actions/runs/8014425468/workflow#L48

Currently the latest version is 1.41.0 but you are using 0.14.15. Maybe that issue has long been fixed?

I tried to upgrade maturin-action to the latest version, but it seems like the error still exists: https://github.com/sfu-db/connector-x/actions/runs/8076356913/job/22064690961#step:13:20

[wangxiaoying]

@wangxiaoying seeing that the CI tests now pass, is there anything preventing this PR from being merged?

Since this PR is not for convention CI (test cases) but related to the release process, we cannot merge it until we can successfully run the release script on github action. Currently the error still exists: https://github.com/sfu-db/connector-x/actions/runs/8076356913/job/22064690961#step:13:20

[j0bekt01]

@wangxiaoying seeing that the CI tests now pass, is there anything preventing this PR from being merged?

Since this PR is not for convention CI (test cases) but related to the release process, we cannot merge it until we can successfully run the release script on github action. Currently the error still exists: https://github.com/sfu-db/connector-x/actions/runs/8076356913/job/22064690961#step:13:20

It looks like Docker is not installed on the machine on which you're running this workflow.

[wangxiaoying]

@wangxiaoying seeing that the CI tests now pass, is there anything preventing this PR from being merged?

Since this PR is not for convention CI (test cases) but related to the release process, we cannot merge it until we can successfully run the release script on github action. Currently the error still exists: https://github.com/sfu-db/connector-x/actions/runs/8076356913/job/22064690961#step:13:20

It looks like Docker is not installed on the machine on which you're running this workflow.

We are running our release workflow using github-action and on github-hosted runners, which is why we need to compile it by specifying --target aarch64-unknown-linux-gnu.

The error here is still the same with the one I encountered back in this issue: #240.

[evbo]

is it necessary using this container?: https://github.com/sfu-db/connector-x/blob/main/.github/workflows/release.yml#L12C16-L12C50

The default container for ubuntu-latest supposedly has docker installed

[wangxiaoying]

is it necessary using this container?: https://github.com/sfu-db/connector-x/blob/main/.github/workflows/release.yml#L12C16-L12C50

The default container for ubuntu-latest supposedly has docker installed

That's a good point. The previous container may be wrong. According to maturin's doc, we may need to use a different docker mentioned for aarch in the table. However, I tried https://ghcr.io/rust-cross/manylinux_2_28-cross:aarch64 with no luck. It got error on setup-poetry before reaching to maturin: https://github.com/sfu-db/connector-x/actions/runs/8091126113/job/22109709826#step:7:7

[j0bekt01]

is it necessary using this container?: https://github.com/sfu-db/connector-x/blob/main/.github/workflows/release.yml#L12C16-L12C50
The default container for ubuntu-latest supposedly has docker installed

That's a good point. The previous container may be wrong. According to maturin's doc, we may need to use a different docker mentioned for aarch in the table. However, I tried https://ghcr.io/rust-cross/manylinux_2_28-cross:aarch64 with no luck. It got error on setup-poetry before reaching to maturin: https://github.com/sfu-db/connector-x/actions/runs/8091126113/job/22109709826#step:7:7

what about this one: quay.io/pypa/manylinux_2_28_aarch64:latest?

[wangxiaoying]

quay.io/pypa/manylinux_2_28_aarch64:latest

It is already pulling the latest one: https://github.com/sfu-db/connector-x/actions/runs/8076356913/job/22064690961#step:2:19

[j0bekt01]

quay.io/pypa/manylinux_2_28_aarch64:latest

It is already pulling the latest one: https://github.com/sfu-db/connector-x/actions/runs/8076356913/job/22064690961#step:2:19

but that's x86 not aarch64

[j0bekt01]

Update: I was able to build it locally as instructed here on r7g.4xlarge instance running 4.14.334-252.552.amzn2.aarch64. It works, however, it's not feasible to do a build every time I spin up the instance.

[wangxiaoying]

quay.io/pypa/manylinux_2_28_aarch64:latest

It is already pulling the latest one: https://github.com/sfu-db/connector-x/actions/runs/8076356913/job/22064690961#step:2:19

but that's x86 not aarch64

Seems like this one cannot work on the github-hosted runner:
https://github.com/sfu-db/connector-x/actions/runs/8105184340/job/22153117379#step:2:115

[wangxiaoying]

Update: I was able to build it locally as instructed here on r7g.4xlarge instance running 4.14.334-252.552.amzn2.aarch64. It works, however, it's not feasible to do a build every time I spin up the instance.

I think you don't need to build every time. Can you store the built wheel file to a place and fetch the wheel file to install everytime you start an instance?

[j0bekt01]

Update: I was able to build it locally as instructed here on r7g.4xlarge instance running 4.14.334-252.552.amzn2.aarch64. It works, however, it's not feasible to do a build every time I spin up the instance.

I think you don't need to build every time. Can you store the built wheel file to a place and fetch the wheel file to install everytime you start an instance?

I'll store it in s3 and go from there. Thanks!

[wangxiaoying]

Seems like we still cannot find a way to do this in our current release workflow.

I am closing this PR for now but feel free to reopen it anytime.

[evbo]

@wangxiaoying there's roughly half a dozen issues in this repo requesting support for Arm Linux, so why close this issue if it's the root cause preventing that support?

What do you see as the work around? I currently am forced to use something besides connectorx when connecting to a db on Arm Linux, writing to a text file, then having polars read that text file.

[wangxiaoying]

there's roughly half a dozen issues in this repo requesting support for Arm Linux, so why close this issue if it's the root cause preventing that support?

This is why the issues remain open, looking for help. But the change made by this PR does not really solve the problem and we currently cannot find a way to make it work. Therefore I closed this PR. But again, please feel free to open a PR if you find a way to making the release workflow support Arm Linux. I think you can fork the repo and test the gihub actions locally.

What do you see as the work around?

For work around, an easy way is to build connectorx manually on your arm linux machine and use the wheel file to install. You can check this above conversation for a reference: #545 (comment) .

[martijnvdwoude]

For work around, an easy way is to build connectorx manually on your arm linux machine and use the wheel file to install. You can check this above conversation for a reference: #545 (comment) .

I put together a simple configurable Dockerfile build script for this, for anyone interested in easily building a wheel file for Linux ARM64:

https://gist.github.com/martijnvdwoude/39dcb7f9d8be2b2f3c2e224f32b800b7

[j0bekt01]

For work around, an easy way is to build connectorx manually on your arm linux machine and use the wheel file to install. You can check this above conversation for a reference: #545 (comment) .

I put together a simple configurable Dockerfile build script for this, for anyone interested in easily building a wheel file for Linux ARM64:

https://gist.github.com/martijnvdwoude/39dcb7f9d8be2b2f3c2e224f32b800b7

Thanks for putting this together!
Rather than spending hours building Python with a makefile, consider checking Docker for a prebuilt image with the required Python version. I opted for the image arm64v8/python:3.9.19-bookworm instead of the one you're using. Here's the Dockerfile with the change I made.

[bfgdigital]

Can we get some files uploaded to Pypi using this approach?

[evbo]

@bfgdigital I think that's a good point. Probably within the next 3 years Microsoft will have better support for ARM containers, so just as a stop gap why not release the latest connector-x arm linux builds? The chances of this library changing significantly are pretty low, and if needed one could build it themselves.

Would help a lot of people just releasing something to Pypi

[mike-luabase]

this worked for me on Mac M1 with Docker without needing to build from source

FROM --platform=linux/amd64 python:3.9-slim-buster

# Install system dependencies
RUN apt-get update && apt-get install -y \
    gnupg2 \
    curl \
    unixodbc-dev \
    && curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add - \
    && curl https://packages.microsoft.com/config/debian/10/prod.list > /etc/apt/sources.list.d/mssql-release.list \
    && apt-get update \
    && ACCEPT_EULA=Y apt-get install -y msodbcsql17

# Check Python and pip versions
RUN python --version && pip --version

# Install ConnectorX (this might take a while as it needs to compile from source)
RUN pip install --no-cache-dir connectorx==0.2.3

# Copy your application code
COPY . /app
WORKDIR /app

# Install other dependencies
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt