Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
bundle:
name: test-bundle-$UNIQUE_NAME

variables:
# Root default: resolvable in every target, so sync may introduce
# references to it anywhere.
region:
default: us-west-1
# No root default, but assigned in every target: also resolvable everywhere.
etl_schema:
description: Schema name, assigned in every target
# No root default and only assigned in dev: other targets cannot resolve it,
# so sync must not introduce ${var.dev_catalog} outside the dev section.
dev_catalog:
description: Catalog name, assigned only in the dev target
# Same shape as dev_catalog; referenced from the dev override section where
# dev-only variables are allowed.
dev_label:
description: Cost center tag, assigned only in the dev target

resources:
jobs:
my_job:
name: test-job-$UNIQUE_NAME
parameters:
- name: region
default: ${var.region}
- name: schema
default: ${var.etl_schema}
- name: catalog
default: ${var.dev_catalog}
tasks:
- task_key: main
notebook_task:
notebook_path: /Users/{{workspace_user_name}}/notebook
new_cluster:
spark_version: $DEFAULT_SPARK_VERSION
node_type_id: $NODE_TYPE_ID
num_workers: 1

targets:
dev:
default: true
variables:
etl_schema: dev_schema
dev_catalog: dev_data
dev_label: dev-team
resources:
jobs:
my_job:
tags:
costcenter: ${var.dev_label}
prod:
variables:
etl_schema: prod_schema

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle-[UNIQUE_NAME]/dev/files...
Deploying resources...
Updating deployment state...
Deployment complete!

=== Modify job remotely

=== Detect and save changes
Detected changes in 1 resource(s):

Resource: resources.jobs.my_job
parameters[name='catalog_copy']: add
parameters[name='data_region']: add
parameters[name='region'].default: replace
parameters[name='schema_copy']: add
tags['costcenter']: replace



=== Configuration changes

>>> diff.py databricks.yml.backup databricks.yml
--- databricks.yml.backup
+++ databricks.yml
@@ -25,9 +25,15 @@
parameters:
- name: region
- default: ${var.region}
+ default: dev_data
- name: schema
default: ${var.etl_schema}
- name: catalog
default: ${var.dev_catalog}
+ - default: dev_data
+ name: catalog_copy
+ - default: ${var.region}
+ name: data_region
+ - default: ${var.etl_schema}
+ name: schema_copy
tasks:
- task_key: main
@@ -50,5 +56,5 @@
my_job:
tags:
- costcenter: ${var.dev_label}
+ costcenter: ${var.dev_catalog}
prod:
variables:

>>> [CLI] bundle destroy --auto-approve
The following resources will be deleted:
delete resources.jobs.my_job

All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/test-bundle-[UNIQUE_NAME]/dev

Deleting files...
Destroy complete!
49 changes: 49 additions & 0 deletions acceptance/bundle/config-remote-sync/cross_target_variables/script
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
#!/bin/bash

envsubst < databricks.yml.tmpl > databricks.yml

cleanup() {
trace $CLI bundle destroy --auto-approve
}
trap cleanup EXIT

$CLI bundle deploy
job_id="$(read_id.py my_job)"

title "Modify job remotely"
echo
edit_resource.py jobs $job_id <<EOF
# --- Add operations (sibling-based restoration) ---
# "us-west-1" matches sibling \${var.region} (root default, resolvable in
# every target) -> restored.
r["parameters"].append({"name": "data_region", "default": "us-west-1"})
# "dev_schema" matches sibling \${var.etl_schema} (assigned in every target)
# -> restored.
r["parameters"].append({"name": "schema_copy", "default": "dev_schema"})
# "dev_data" matches sibling \${var.dev_catalog}, which is only assigned in
# the dev target: restoring it into the shared resources section would break
# every other target -> stays hardcoded.
r["parameters"].append({"name": "catalog_copy", "default": "dev_data"})

# --- Replace fallback (re-targeting to another variable) ---
# "region" was \${var.region}; the new value uniquely matches dev-only
# \${var.dev_catalog} -> the shared definition keeps the hardcoded value.
for p in r["parameters"]:
if p["name"] == "region":
p["default"] = "dev_data"

# --- Replace inside the current target's override section ---
# costcenter is defined under targets.dev; dev-only variables are allowed
# there, so the unique match \${var.dev_catalog} is restored.
r["tags"]["costcenter"] = "dev_data"
EOF

title "Detect and save changes"
echo
cp databricks.yml databricks.yml.backup
$CLI bundle config-remote-sync --save

title "Configuration changes"
echo
trace diff.py databricks.yml.backup databricks.yml
rm databricks.yml.backup
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
Cloud = true

RecordRequests = false
Ignore = [".databricks", "databricks.yml", "databricks.yml.backup"]

[Env]
DATABRICKS_BUNDLE_ENABLE_EXPERIMENTAL_YAML_SYNC = "true"

[EnvMatrix]
DATABRICKS_BUNDLE_ENGINE = ["direct", "terraform"]
Loading
Loading