workflows: add build-pyarrow.yml for riscv64 manywheel builds - #292
Closed
luhenry wants to merge 2 commits into
Closed
workflows: add build-pyarrow.yml for riscv64 manywheel builds#292luhenry wants to merge 2 commits into
luhenry wants to merge 2 commits into
Conversation
pyarrow is Cython bindings over the Apache Arrow C++ libraries. Build the Arrow C++ libs once from source with bundled third-party deps (ARROW_DEPENDENCY_SOURCE=BUNDLED — the manylinux riscv64 image has no vcpkg binary cache, unlike upstream's x86/arm images), then build a wheel per Python (cp312/313/314/314t) against that shared C++ build. Mirrors apache/arrow's ci/scripts/python_wheel_xlinux_build.sh, minus vcpkg, and tests like ci/scripts/python_wheel_unix_test.sh (pytest --pyargs pyarrow, feature-gated via PYARROW_TEST_*). Feature set (workflow env, single source of truth): Parquet, Dataset, Acero, Compute, CSV, JSON, Filesystem, HDFS, ORC, Substrait, Flight, Parquet-encryption + all compression. S3/GCS/Azure/Gandiva off for now — their dep trees (aws-sdk/google-cloud-cpp/azure-sdk/LLVM) are unverified on riscv64 and will be enabled incrementally. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
luhenry
changed the title
The Arrow C++ build failed at CMake generate with "OpenSSL::SSL / OpenSSL::Crypto ... target was not found", cascading into the bundled gRPC, arrow_util and parquet link targets. Root cause: -DARROW_DEPENDENCY_USE_SHARED=OFF (all bundled deps static, as upstream's vcpkg build does) cascades into ARROW_OPENSSL_USE_SHARED=OFF, which makes FindOpenSSLAlt set OPENSSL_USE_STATIC_LIBS=ON. OpenSSL is the one dep Arrow can't bundle — it comes from the OS — and the manylinux_2_39_riscv64 image (Rocky 10) ships only shared libssl/libcrypto, no static .a. So the static lookup returned OPENSSL_CRYPTO_LIBRARY-NOTFOUND and the imported targets were never created. Upstream's images don't hit this because their vcpkg cache provides static OpenSSL. Pin -DARROW_OPENSSL_USE_SHARED=ON so Arrow links the shared OpenSSL that exists in the image; everything else stays static/bundled. Verified under riscv64 qemu: CMake now reports "Found OpenSSL Crypto Library: /usr/lib64/libcrypto.so" and configure proceeds past the previous generate-step failure. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Member
Author
|
Superseded by #294 — branch renamed from |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds
.github/workflows/build-pyarrow.ymlto build riscv64 pyarrow wheels (25.0.1) and publish them topypi.riseproject.dev.pyarrow is Cython bindings over the Apache Arrow C++ libraries, so this is a heavy build: the Arrow C++ libs are built once from source with bundled third-party deps (
ARROW_DEPENDENCY_SOURCE=BUNDLED— the manylinux riscv64 image has no vcpkg binary cache, unlike upstream's x86/arm images), then a wheel is built per Python against that shared C++ build. This mirrors apache/arrow's ownci/scripts/python_wheel_xlinux_build.sh(minus vcpkg) and tests likeci/scripts/python_wheel_unix_test.sh.Shape
docker run(not cibuildwheel): pyarrow's C++ sources are a sibling of the Python package, which cibuildwheel's copy-package-dir model can't handle, and the manylinux riscv64 image ships no Node (so acontainer:job can't run JS actions). Checkout/upload run on the riscv host; the heavy build runs indocker runinsidequay.io/pypa/manylinux_2_39_riscv64on the native riscv64 runner.cp312 / cp313 / cp314 / cp314tbuilding wheels (scikit-build-core,PYARROW_BUNDLE_ARROW_CPP=ON), thenauditwheel repair --strip.pytest -r s --pyargs pyarrow, feature-gated viaPYARROW_TEST_*.Feature set (workflow
env, single source of truth)ON: Parquet, Dataset, Acero, Compute, CSV, JSON, Filesystem, HDFS, ORC, Substrait, Flight, Parquet-encryption + all compression (snappy/zlib/zstd/lz4/brotli/bz2), mimalloc.
OFF (enabled incrementally later): S3, GCS, Azure, Gandiva, jemalloc, OpenTelemetry, TensorFlow. The network-storage stacks (aws-sdk / google-cloud-cpp / azure-sdk) and LLVM-based Gandiva have dep trees unverified on riscv64; enabling them is a one-line flip in the
envblock per feature.Validation done locally
actionlintclean (only the expectedubuntu-24.04-riscvunknown-label warning).dnf install ninja-build openssl-develworks and OpenSSL dev headers land (the image ships neither by default) — the prerequisites for the Flight/encryption features.Notes for review
numpy/libcst/pandasare pulled frompypi.riseproject.dev(no public-PyPI riscv64 wheels);pandasis optional in the test step and gates the pandas-dependent tests (matters for cp314t, which may lack a free-threaded pandas wheel).🤖 Generated with Claude Code