How to create AWS Graviton (ARM) Python Lambda on Intel machine with Poetry

Question

I'm trying to build a Python Lambda with Poetry. My function depends on psycopg2. This library, in turn, depends on a platform binary: libpq. So, I need to bundle it in my distro (a ZIP file). There is a psycopg2-binary package on PyPi and I believe it has the wheel I need: psycopg2_binary-2.9.6-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl. Python version 3.9, just like my runtime, ARM 64 architecture. It contains the libpq I need:

$ file psycopg2_binary.libs/libpq-33589b1f.so.5.15

psycopg2_binary.libs/libpq-33589b1f.so.5.15: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, BuildID[sha1]=0874810fb70766ff96a80897579633a2ef7af60e, stripped

Definitely it is what I need.

I build my function with Poetry in three steps:

poetry build
poetry run pip install --upgrade --target package dist/*.whl
cd package; zip -r ../distro.zip . -x '*.pyc'

The problem is that when I install the psycopg2-binary (poetry add psycopg2-binary) it installs the Intel wheel (my host OS).

I even tried downloading the wheel I need and installing it with

poetry add ./psycopg2_binary-2.9.6-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl

But it it gave me an error:

ERROR: psycopg2_binary-2.9.6-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl is not a supported wheel on this platform.

I tried adding a platform:

poetry add --platform=manylinux2014_aarch64 ./psycopg2_binary-2.9.6-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl

But this just does nothing. It doesn't fail and it doesn't install anything.

So, how do I cross-compile for ARM with Poetry on Intel?

P.S. Please, don't suggest using aws-psycopg2, its libpq version is too low (9) and I need 14+.

I think the easiest solution would be an ARM CI runner. The next best thing would be to use an [ARM docker image](https://www.docker.com/blog/multi-arch-images/). — DeepSpace, Apr 18 '23 at 19:56
There is no such thing as cross-compilation, forget. Use ARM virtual machine. — phd, Apr 18 '23 at 20:01
But all I need is just to assemble a ZIP file! I don't need to run anything locally, I just need to get a file for a different platform… — madhead, Apr 18 '23 at 20:04
I just wrote up my solution to doing this with `pip` [on Medium](https://therightstuff.medium.com/building-binary-python-dependencies-for-amd64-and-arm64-lambda-layers-efc6f17e4bf6), I'm not sure how it translates for `poetry` but I'm confident the process is similar... — therightstuff, Jul 27 '23 at 05:49

score 0 · Accepted Answer · answered Apr 19 '23 at 20:50

It turns out that @DeepSpace and @phd from the comments are right. The most straightforward way is to use Docker. You don't even need a docker buildx. Here are the two simple steps to achieve the goal:

Create a Dockerfile for the builder:

FROM --platform=arm64 python:3.9

RUN apt update && apt install -y zip
RUN curl -sSL https://install.python-poetry.org | python -
ENV PATH="/root/.local/bin:$PATH"

Create a docker-compose.yml using this builder:

services:
  builder:
    platform: linux/arm64
    build: .
    volumes:
      - ./:/lambda
    working_dir: /lambda

I also had a Makefile, but it's optional. A Bash script would do as well.

clean:
    rm -rf dist package neonbranch.zip

dist: lambda
    poetry build

package: dist
    poetry run pip install --upgrade --target package dist/*.whl

distro.zip: package
    cd package; zip -r ../distro.zip . -x '*.pyc'

And that's simple.

Now, run docker-compose up to initiate the builder. It may take a while.

After it's done, I am able to build my package with docker-compose run builder make distro.zip!

How to create AWS Graviton (ARM) Python Lambda on Intel machine with Poetry

1 Answers1