Migrate terraform state to direct before deploy

.nextchanges/bundles/migrate-before-deploy.md

@@ -0,0 +1 @@
+* Migrate existing Terraform deployment state to the direct engine before deploying (previously done after a Terraform deploy), so the deploy runs on the direct engine. ([#6749](https://github.com/databricks/cli/pull/6749))

acceptance/bundle/deploy/snapshot-comparison/output.txt

@@ -54,6 +54,8 @@ Destroy: 2 deleted
 
 === Cleanup bundle 2
 >>> [CLI] bundle destroy --auto-approve
+Notice: automatically migrating your bundle to direct deployment engine (https://github.com/databricks/cli/issues/6765).
+Migrated 2 resources to direct deployment engine.
 The following resources will be deleted:
   delete resources.jobs.test_job
   delete resources.pipelines.test_pipeline

acceptance/bundle/migrate/auto/artifacts/databricks.yml

@@ -0,0 +1,15 @@
+bundle:
+  name: test-bundle
+
+resources:
+  jobs:
+    test_job:
+      name: "Test Auto-Migrate Artifacts Job"
+      tasks:
+        - task_key: TestTask
+          existing_cluster_id: "0717-132531-5opeqon1"
+          python_wheel_task:
+            package_name: "my_test_code"
+            entry_point: "run"
+          libraries:
+            - whl: ./dist/*.whl

acceptance/bundle/migrate/auto/artifacts/output.txt

@@ -0,0 +1,43 @@
+
+=== Initial deploy on terraform (expands the whl glob and uploads the wheel)
+>>> DATABRICKS_BUNDLE_ENGINE=terraform [CLI] bundle deploy
+Warn: unable to get spark version for cluster 0717-132531-5opeqon1, err: Not Found
+Uploading dist/my_test_code-0.0.1-py3-none-any.whl...
+Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files...
+Created jobs.test_job
+Files: 4 uploaded, 0 deleted
+Resources: 1 created, 0 changed, 0 deleted, 0 unchanged
+
+=== Redeploy with direct default: auto-migration runs after the build, so library refs are resolved
+>>> [CLI] bundle deploy
+Warn: unable to get spark version for cluster 0717-132531-5opeqon1, err: Not Found
+Notice: automatically migrating your bundle to direct deployment engine (https://github.com/databricks/cli/issues/6765).
+Migrated 1 resource to direct deployment engine.
+Uploading dist/my_test_code-0.0.1-py3-none-any.whl...
+Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files...
+Files: 0 uploaded, 0 deleted
+Resources: 0 created, 0 changed, 0 deleted, 1 unchanged
+
+=== Local state is now direct
+
+>>> find .databricks/bundle -name resources.json -type f
+.databricks/bundle/default/resources.json
+
+>>> find .databricks/bundle -name terraform.tfstate* -type f
+.databricks/bundle/default/terraform/terraform.tfstate.backup
+
+=== Migrated state's library path (should be the resolved remote path, not the local glob)
+>>> print_state.py
+[
+  {
+    "whl": "/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/artifacts/.internal/my_test_code-0.0.1-py3-none-any.whl"
+  }
+]
+
+=== Subsequent deploy is a no-op on direct
+>>> [CLI] bundle deploy
+Warn: unable to get spark version for cluster 0717-132531-5opeqon1, err: Not Found
+Uploading dist/my_test_code-0.0.1-py3-none-any.whl...
+Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files...
+Files: 0 uploaded, 0 deleted
+Resources: 0 created, 0 changed, 0 deleted, 1 unchanged

acceptance/bundle/migrate/auto/artifacts/script

@@ -0,0 +1,17 @@
+export DATABRICKS_BUNDLE_ENGINE=
+
+title "Initial deploy on terraform (expands the whl glob and uploads the wheel)"
+trace DATABRICKS_BUNDLE_ENGINE=terraform $CLI bundle deploy
+
+title "Redeploy with direct default: auto-migration runs after the build, so library refs are resolved"
+trace $CLI bundle deploy
+
+title "Local state is now direct\n"
+trace find .databricks/bundle -name "resources.json" -type f
+trace find .databricks/bundle -name "terraform.tfstate*" -type f
+
+title "Migrated state's library path (should be the resolved remote path, not the local glob)"
+trace print_state.py | jq '.. | .libraries? | select(.)'
+
+title "Subsequent deploy is a no-op on direct"
+trace $CLI bundle deploy

acceptance/bundle/migrate/auto-migrate-clean/output.txt → acceptance/bundle/migrate/auto/clean/output.txt

@@ -15,12 +15,10 @@ direct_drymigrate_warnings false
   engine: direct
 
 >>> [CLI] bundle deploy
-Warn: Direct engine selected via bundle.engine setting at [TEST_TMP_DIR]/databricks.yml:3:11 but the existing state uses "terraform". Deploying on "terraform"; will attempt to migrate the state to the direct engine after this deploy.
+Migrated 1 resource to direct deployment engine.
 Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files...
 Files: 1 uploaded, 0 deleted
 Resources: 0 created, 0 changed, 0 deleted, 1 unchanged
-Migrating state to direct deployment engine (selected via bundle.engine setting at [TEST_TMP_DIR]/databricks.yml:3:11)...
-Migrated 1 resource to direct deployment engine.
 
 >>> print_migration_telemetry
 direct_migrated_via_config true

acceptance/bundle/migrate/auto-migrate-default/output.txt → acceptance/bundle/migrate/auto/default/output.txt

@@ -12,17 +12,11 @@ direct_drymigrate_warnings false
 
 === Redeploy with nothing set: direct is the default, so auto-migration triggers
 >>> [CLI] bundle deploy
-Notice: the direct deployment engine is the default as of CLI v1.14.0.
-
-This bundle will be automatically migrated to use the direct deployment engine after this deployment.
-
-Learn more: https://docs.databricks.com/dev-tools/bundles/direct
-
+Notice: automatically migrating your bundle to direct deployment engine (https://github.com/databricks/cli/issues/6765).
+Migrated 1 resource to direct deployment engine.
 Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files...
 Files: 0 uploaded, 0 deleted
 Resources: 0 created, 0 changed, 0 deleted, 1 unchanged
-Migrating state to direct deployment engine (selected via default)...
-Migrated 1 resource to direct deployment engine.
 
 >>> print_migration_telemetry
 direct_migrated_via_default true

acceptance/bundle/migrate/auto/delete-all/databricks.yml

@@ -0,0 +1,7 @@
+bundle:
+  name: test-bundle
+
+resources:
+  jobs:
+    job_a:
+      name: "Job A"

acceptance/bundle/migrate/auto/delete-all/output.txt

@@ -0,0 +1,20 @@
+
+=== Initial deploy on terraform
+>>> DATABRICKS_BUNDLE_ENGINE=terraform [CLI] bundle deploy
+Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files...
+Created jobs.job_a
+Files: 3 uploaded, 0 deleted
+Resources: 1 created, 0 changed, 0 deleted, 0 unchanged
+
+=== Remove all resources, then deploy on direct: migrate and delete
+>>> [CLI] bundle deploy
+Notice: automatically migrating your bundle to direct deployment engine (https://github.com/databricks/cli/issues/6765).
+Migrated 1 resource to direct deployment engine.
+Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files...
+Deleted jobs.job_a
+Files: 1 uploaded, 0 deleted
+Resources: 0 created, 0 changed, 1 deleted, 0 unchanged
+
+=== Follow-up plan is a clean no-op
+>>> [CLI] bundle plan
+Plan: 0 to add, 0 to change, 0 to delete, 0 unchanged

acceptance/bundle/migrate/auto/delete-all/script

@@ -0,0 +1,15 @@
+export DATABRICKS_BUNDLE_ENGINE=
+
+title "Initial deploy on terraform"
+trace DATABRICKS_BUNDLE_ENGINE=terraform $CLI bundle deploy
+
+# Removing every resource leaves the migration with no config-declared resources, so
+# BuildStateFromTF records no WAL entries. The migration must still persist the seeded
+# terraform-state resources so the direct plan can delete them (otherwise Finalize would
+# write no state file and the migration would fail with a missing resources.json).
+title "Remove all resources, then deploy on direct: migrate and delete"
+echo '{"bundle": {"name": "test-bundle"}}' > databricks.yml
+trace $CLI bundle deploy
+
+title "Follow-up plan is a clean no-op"
+trace $CLI bundle plan | contains.py "0 to delete, 0 unchanged"

acceptance/bundle/migrate/auto/delete/databricks.yml

@@ -0,0 +1,9 @@
+bundle:
+  name: test-bundle
+
+resources:
+  jobs:
+    job_keep:
+      name: "Keep Job"
+    job_drop: # TO_REMOVE
+      name: "Drop Job" # TO_REMOVE

acceptance/bundle/migrate/auto/delete/output.txt

@@ -0,0 +1,21 @@
+
+=== Initial deploy on terraform with two jobs
+>>> DATABRICKS_BUNDLE_ENGINE=terraform [CLI] bundle deploy
+Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files...
+Created jobs.job_drop
+Created jobs.job_keep
+Files: 3 uploaded, 0 deleted
+Resources: 2 created, 0 changed, 0 deleted, 0 unchanged
+
+=== Drop one job from config, then deploy on direct: migrate two, delete the dropped one
+>>> [CLI] bundle deploy
+Notice: automatically migrating your bundle to direct deployment engine (https://github.com/databricks/cli/issues/6765).
+Migrated 2 resources to direct deployment engine.
+Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files...
+Deleted jobs.job_drop
+Files: 1 uploaded, 0 deleted
+Resources: 0 created, 0 changed, 1 deleted, 1 unchanged
+
+=== Follow-up plan is a clean no-op
+>>> [CLI] bundle plan
+Plan: 0 to add, 0 to change, 0 to delete, 1 unchanged

acceptance/bundle/migrate/auto/delete/script

@@ -0,0 +1,13 @@
+export DATABRICKS_BUNDLE_ENGINE=
+
+title "Initial deploy on terraform with two jobs"
+trace DATABRICKS_BUNDLE_ENGINE=terraform $CLI bundle deploy
+
+# The migration seeds every terraform-state resource into the direct state, so the
+# first direct plan can delete the ones the config no longer declares.
+title "Drop one job from config, then deploy on direct: migrate two, delete the dropped one"
+grep -v 'TO_REMOVE' databricks.yml > updated.yml && mv updated.yml databricks.yml
+trace $CLI bundle deploy
+
+title "Follow-up plan is a clean no-op"
+trace $CLI bundle plan | contains.py "1 unchanged"

acceptance/bundle/migrate/auto/destroy/databricks.yml

@@ -0,0 +1,7 @@
+bundle:
+  name: test-bundle
+
+resources:
+  jobs:
+    job_a:
+      name: "Job A"

acceptance/bundle/migrate/auto/destroy/output.txt

@@ -0,0 +1,34 @@
+
+=== Initial deploy on terraform
+>>> DATABRICKS_BUNDLE_ENGINE=terraform [CLI] bundle deploy
+Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files...
+Created jobs.job_a
+Files: 3 uploaded, 0 deleted
+Resources: 1 created, 0 changed, 0 deleted, 0 unchanged
+
+=== Destroy without approval: stays on terraform
+>>> musterr [CLI] bundle destroy
+Error: this command will destroy all resources deployed by this bundle, including workspace files in the deployment directory.
+Deleting data assets such as schemas, pipelines, or volumes may cause permanent data loss and should be carefully reviewed.
+To proceed, use --auto-approve.
+
+>>> find .databricks/bundle -name resources.json -type f
+
+>>> find .databricks/bundle -name terraform.tfstate* -type f
+.databricks/bundle/default/terraform/terraform.tfstate
+
+=== Destroy on direct: migrate then destroy
+>>> [CLI] bundle destroy --auto-approve
+Notice: automatically migrating your bundle to direct deployment engine (https://github.com/databricks/cli/issues/6765).
+Migrated 1 resource to direct deployment engine.
+The following resources will be deleted:
+  delete resources.jobs.job_a
+
+All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/test-bundle/default
+
+Destroy: 1 deleted
+
+=== Destroy removes both local state files (direct and the superseded terraform state)
+>>> find .databricks/bundle -name resources.json -type f
+
+>>> find .databricks/bundle -name terraform.tfstate* -type f

acceptance/bundle/migrate/auto/destroy/script

@@ -0,0 +1,21 @@
+export DATABRICKS_BUNDLE_ENGINE=
+
+title "Initial deploy on terraform"
+trace DATABRICKS_BUNDLE_ENGINE=terraform $CLI bundle deploy
+
+title "Destroy without approval: stays on terraform"
+trace musterr $CLI bundle destroy
+trace find .databricks/bundle -name "resources.json" -type f
+trace find .databricks/bundle -name "terraform.tfstate*" -type f
+
+# Destroy behaves like deploy: it migrates the terraform state to direct (writing the local
+# direct state), plan-checks it, and - once approved - destroys on the direct engine. destroyCore
+# then removes the local state files: the superseded terraform state first, then the direct state,
+# so a crash between the two never leaves a live terraform.tfstate with no direct state. A declined
+# destroy would discard the migration and stay on terraform.
+title "Destroy on direct: migrate then destroy"
+trace $CLI bundle destroy --auto-approve
+
+title "Destroy removes both local state files (direct and the superseded terraform state)"
+trace find .databricks/bundle -name "resources.json" -type f
+trace find .databricks/bundle -name "terraform.tfstate*" -type f