Default source: GIT on git_source job tasks - #6751
Merged
Merged
Conversation
A job with a git_source keeps repo-relative task file paths (path translation is skipped for git_source jobs). The backend treats a task as git-sourced only when its source is explicitly GIT, so it rejects the relative path with "Invalid python file reference" when source is unset. The direct engine never set it, so `bundle deploy` failed; terraform worked because tfdyn stamps it. Add an ApplyDefaultTaskSource mutator that runs after PythonMutator and sets source: GIT on the tasks of git_source jobs. This covers both engines from one place and is observable in `bundle validate`. Add testserver validation mirroring the backend and a direct-only acceptance test that reproduces the failure. Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
pietern
requested review from
andrewnester and
denik
and removed request for
denik
September 18, 2026 11:23
denik
approved these changes
Sep 18, 2026
denik
left a comment
Contributor
There was a problem hiding this comment.
Do we need another test for PyDABs? I vaguely remember there was some issue between this mutator and PyDABs resources. cc @andrewnester
andrewnester
approved these changes
Sep 18, 2026
Collaborator
Integration test reportCommit: 94fbafa
Top 3 slowest tests (at least 2 minutes):
|
source: GIT on git_source job tasks so the direct engine can deploy themsource: GIT on git_source job tasks so the direct engine can deploy them
source: GIT on git_source job tasks so the direct engine can deploy themsource: GIT on git_source job tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
Restore
ApplyDefaultTaskSourceas a shared resource mutator that setssource: GITon the tasks of a job with agit_source(for the task types that support the field:dbt_task,gen_ai_compute_task,notebook_task,spark_python_task,sql_task.file, includingfor_each_task, and only when the user did not set it). It runs afterPythonMutator, so it reflects anygit_source/tasks that Python code added or removed, is not exposed to Python, and is still observable inbundle validate/summary. Non-git jobs are left untouched, and an explicitgit_source: nullis treated as absent (matchingTranslatePaths).tfdynis unchanged and becomes an idempotent no-op.Why
Deploying a job with a
git_sourceand aspark_python_task(repo-relativepython_file) failed on the direct engine withInvalid python file reference: <path> (400 INVALID_PARAMETER_VALUE), while Terraform deployed it fine. Path translation is intentionally skipped forgit_sourcejobs, so task file paths stay relative to the Git repo. The backend treats a task as git-sourced only when itssourceis explicitlyGIT; it does not infer that from the job-levelgit_source. Terraform stampedsource: GITintfdyn, but the direct engine sent nosource, so the backend read the relative path as a workspace path and rejected it. This was latent since the defaulting moved from a shared mutator intotfdyn(#3528) and only became user-visible once the direct engine became the default. See #3359 and #3528 for the history.Tests
Adds a direct-only acceptance test that reproduces the failure, plus testserver validation that mirrors the backend's
python_filecheck so the test fails without the fix. Adds unit tests covering git/non-git/null/for_eachand explicit-source cases. The only existing golden that changes istask-source/out.git_job.direct.txt, which now includessource: GIT. Verified end-to-end against e2-dogfood (direct deploy of a git-sourcedspark_python_tasknow succeeds).This pull request and its description were written by Isaac.