air: add convert-to-dabs (run YAML -> Databricks Asset Bundle) - #6111
Open
vinchenzo-db wants to merge 2 commits into
Open
air: add convert-to-dabs (run YAML -> Databricks Asset Bundle)#6111vinchenzo-db wants to merge 2 commits into
vinchenzo-db wants to merge 2 commits into
Conversation
Contributor
Approval status: pending
|
Collaborator
Integration test reportCommit: fd0ab4a
8 interesting tests: 4 SKIP, 3 KNOWN, 1 RECOVERED
Top 3 slowest tests (at least 2 minutes):
|
vinchenzo-db
force-pushed
the
air-convert-to-dabs-main
branch
2 times, most recently
from
July 31, 2026 13:46
4703d56 to
a81002a
Compare
Adds `air convert-to-dabs`, which translates an AIR CLI run YAML into a
deployable Databricks Asset Bundle so a workload authored for `air run` can be
managed and deployed with the standard DABs workflow (validate/deploy/run).
The emitted bundle is schema-valid: the ai_runtime_task maps to the SDK
jobs.AiRuntimeTask (experiment + deployments[].{command_path,compute} +
code_source_path), with framework fields (retries, timeout, budget policy) on
the surrounding task and the runtime environment in environments[].
Snapshotting is owned by the deploy-time aicode mutator, not by convert:
code_source_path points at a local *directory* staged inside the bundle, and
`bundle deploy` (aicode.PackageAndUpload) packages it into a content-addressed
tarball and uploads it. convert only lays down the source bytes — copying the
working tree (honoring .gitignore) or materializing a pinned git commit into the
directory. requirements.yaml is likewise not emitted: aicode.SynthesizeRequirements
regenerates it from the environments[] spec, so convert folds the whole dependency
set (inline or requirements-file) into that spec instead.
env_variables / secrets / parameters have no native ai_runtime_task field, so they
ride as env_vars.json / secret_env_vars.json / hyperparameters.yaml sidecars (same
as `air run`), and a "Notes:" section tells a migrating user what was transformed
or staged out-of-band.
This is the top of a 2-PR stack: it builds on the aicode deploy-time packaging
mutator so the two compose end-to-end.
Co-authored-by: Isaac
The Windows CI job failed the convert-to-dabs acceptance test with "tar (child): Cannot connect to C: resolve failed": the system tar reads the `C:` in an absolute archive path as a remote host:path. Two fixes: - createPlainTarball now passes the archive as a bare basename with cmd.Dir set to the output directory (and an absolute parent), so no `C:\...` path reaches tar's -f argument. Mirrors how git archive is invoked; safe on GNU tar and bsdtar. (This helper is shared with the `air run` snapshot path.) - extractTarball is rewritten in pure Go (archive/tar + compress/gzip) instead of shelling out to `tar -xzf`, eliminating the same drive-letter hazard on the extract side and dropping the external-tar dependency for extraction. It rejects entries that would escape the destination (path traversal, absolute/escaping symlinks) and bounds each file copy to its header size. Adds unit tests for extractTarball (happy path incl. nested dirs + in-tree symlink; traversal + escaping-symlink rejection). Co-authored-by: Isaac
vinchenzo-db
force-pushed
the
air-convert-to-dabs-main
branch
from
July 31, 2026 14:14
a81002a to
fd0ab4a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
air convert-to-dabs, which translates an AIR CLI run YAML into a deployable Databricks Asset Bundle so a workload authored forair runcan be managed and deployed with the standard DABs workflow (validate/deploy/run).The emitted bundle is schema-valid: the ai_runtime_task maps to the SDK jobs.AiRuntimeTask (experiment + deployments[].{command_path,compute} + code_source_path), with framework fields (retries, timeout, budget policy) on the surrounding task and the runtime environment in environments[].
Snapshotting is owned by the deploy-time aicode mutator, not by convert: code_source_path points at a local directory staged inside the bundle, and
bundle deploy(aicode.PackageAndUpload) packages it into a content-addressed tarball and uploads it. convert only lays down the source bytes — copying the working tree (honoring .gitignore) or materializing a pinned git commit into the directory. requirements.yaml is likewise not emitted: aicode.SynthesizeRequirements regenerates it from the environments[] spec, so convert folds the whole dependency set (inline or requirements-file) into that spec instead.env_variables / secrets / parameters have no native ai_runtime_task field, so they ride as env_vars.json / secret_env_vars.json / hyperparameters.yaml sidecars (same as
air run), and a "Notes:" section tells a migrating user what was transformed or staged out-of-band.This is the top of a 2-PR stack: it builds on the aicode deploy-time packaging mutator so the two compose end-to-end.
Co-authored-by: Isaac
Changes
Why
Tests