gsoc26: Manifest Replay for download, delete, and deploy (#65)

README.md

@@ -19,6 +19,8 @@ Command-line and Python client for downloading and deploying datasets on DBpedia
   - [Deploy](#cli-deploy)
   - [Delete](#cli-delete)
   - [Manifest](#cli-manifest)
+    - [Replay](#cli-manifest-replay)
+    - [Summary](#cli-manifest-summary)
 - [Module Usage](#module-usage)
   - [Deploy](#module-deploy)
 - [Development & Contributing](#development--contributing)
@@ -149,8 +151,10 @@ Options:
   --help  Show this message and exit.
 
 Commands:
+  delete    Delete a dataset from the databus.
   deploy    Flexible deploy to Databus command supporting three modes:
   download  Download datasets from databus, optionally using vault access...
+  manifest  Manifest utilities.
 ```
 
 <a id="cli-download"></a>
@@ -609,6 +613,102 @@ The manifest records input parameters, per-file URLs, checksums, byte sizes, tim
 
 Refer [examples/reproducible-download.md](examples/reproducible-download.md) for a full walkthrough.
 
+<a id="cli-manifest-replay"></a>
+#### Replay
+
+Any manifest written with `--manifest` can be replayed later using `databusclient manifest replay <path>`. Replay re-executes the original operation using the parameters recorded in the manifest — you don't need to remember or retype the original command.
+
+```bash
+# Python
+databusclient manifest replay [OPTIONS] MANIFEST_PATH
+# Docker
+docker run --rm -v $(pwd):/data dbpedia/databus-python-client manifest replay [OPTIONS] MANIFEST_PATH
+```
+
+**Important:** credentials are never stored in the manifest and must always be supplied fresh at replay time — `--vault-token`, `--databus-key`, and `--apikey` behave exactly as they do on the original commands.
+
+```bash
+databusclient manifest replay --help
+
+# Output:
+Usage: databusclient manifest replay [OPTIONS] MANIFEST_PATH
+
+  Replay a previously recorded manifest operation.
+
+  Currently supports replay of download, delete, and deploy manifests.
+  For delete manifests, an interactive confirmation is required by
+  default -- use --force to skip it for scripted/unattended use, or
+  --dry-run to preview without prompting or deleting.
+
+Options:
+  --localdir TEXT     Override local output directory for download replay.
+  --databus TEXT      Override Databus endpoint for replay.
+  --vault-token TEXT  Vault token file path, required if manifest auth
+                      method is vault_token.
+  --databus-key TEXT  Databus API key, required if manifest auth method is
+                      databus_key. Also required for delete replay.
+  --apikey TEXT       Databus API key, required for deploy replay.
+  --force             For delete replay: skip the interactive confirmation
+                      prompt. Required for unattended/scripted replay.
+  --dry-run           For delete replay: force a dry-run preview even if
+                      the original operation wasn't one.
+  --help              Show this message and exit.
+```
+
+**Replaying a download:**
+```bash
+databusclient manifest replay ./manifests/download-run.jsonld --localdir ./replayed-data
+```
+If `--localdir` is omitted, replay falls back to the same auto-computed folder structure a fresh download would use — this is not necessarily the same folder the original download used, since the original folder location itself is never stored in the manifest.
+
+**Replaying a delete:** by default, replay asks for confirmation before deleting, exactly like a normal `delete` call:
+```bash
+databusclient manifest replay ./manifests/delete-run.jsonld --databus-key YOUR_API_KEY
+# About to replay a DELETE operation for the following 1 URI(s):
+#   - https://databus.dbpedia.org/...
+# This is irreversible. Proceed? [y/N]:
+```
+For unattended/scripted use (e.g. CI/CD), skip the prompt with `--force`:
+```bash
+databusclient manifest replay ./manifests/delete-run.jsonld --databus-key YOUR_API_KEY --force
+```
+If the original delete was run with `--dry-run --manifest ...`, replay automatically previews without deleting — no flag needed. You can also force a preview on a manifest that wasn't originally a dry run:
+```bash
+databusclient manifest replay ./manifests/delete-run.jsonld --databus-key YOUR_API_KEY --dry-run
+```
+
+**Replaying a deploy:** supported for classic (distributions-as-arguments) and metadata-file deploys. The manifest stores fully-resolved deployment metadata (checksums, sizes, formats already computed), so replay never re-downloads or re-hashes the original files:
+```bash
+databusclient manifest replay ./manifests/deploy-run.jsonld --apikey YOUR_API_KEY
+```
+Replaying redeploys the same version — if it already exists on Databus, it is updated. WebDAV/Nextcloud deploys cannot be replayed, since the originally uploaded local files may no longer exist at their original paths by the time replay runs.
+
+<a id="cli-manifest-summary"></a>
+#### Summary
+
+Print a readable summary of any recorded manifest without replaying it:
+
+```bash
+# Python
+databusclient manifest summary ./manifests/download-run.jsonld
+# Docker
+docker run --rm -v $(pwd):/data dbpedia/databus-python-client manifest summary ./manifests/download-run.jsonld
+```
+
+Example output:
+
+```
+Command  : download
+Executed : 2024-03-24T10:02:49.500418+00:00
+Endpoint : https://databus.dbpedia.org/sparql
+Auth     : vault_token
+Files    : 1 succeeded · 0 failed
+Total    : 100.0 MB
+Status   : completed
+```
+
+Only existing data already stored in the manifest is read — no new files are downloaded or written, and no network access happens.
+
 ## Module Usage
 
 <a id="module-deploy"></a>

databusclient/api/deploy.py

@@ -249,7 +249,7 @@ def create_distribution(
     return f"{url}|{meta_string}"
 
 
-def _create_distributions_from_metadata(
+def create_distributions_from_metadata(
     metadata: List[Dict[str, Union[str, int]]],
 ) -> List[str]:
     """
@@ -524,7 +524,7 @@ def deploy_from_metadata(
     Parameters
     ----------
     metadata : List[Dict[str, Union[str, int]]]
-        List of file metadata entries (see _create_distributions_from_metadata)
+        List of file metadata entries (see create_distributions_from_metadata)
     version_id : str
         Dataset version ID in the form $DATABUS_BASE/$ACCOUNT/$GROUP/$ARTIFACT/$VERSION
     artifact_version_title : str
@@ -538,7 +538,7 @@ def deploy_from_metadata(
     apikey : str
         API key for authentication
     """
-    distributions = _create_distributions_from_metadata(metadata)
+    distributions = create_distributions_from_metadata(metadata)
 
     dataset = create_dataset(
         version_id=version_id,

databusclient/cli.py

@@ -10,6 +10,8 @@
 from databusclient.api.download import download as api_download, DownloadAuthError
 from databusclient.manifest.context import ManifestContext
 from databusclient.manifest.writer import ManifestWriter
+from databusclient.manifest.replay import ManifestReplayError, replay_manifest, load_manifest
+from databusclient.manifest.summary import format_summary
 from databusclient.extensions import webdav
 
 
@@ -138,6 +140,11 @@ def _write_manifest():
                 license_url=license_url,
                 distributions=distributions,
             )
+            if manifest_context:
+                manifest_context.replay_params["deploy_mode"] = "classic"
+                manifest_context.replay_params["resolved_distributions"] = (
+                    dataid["@graph"][-1].get("distribution", [])
+                )
             api_deploy.deploy(dataid=dataid, api_key=apikey)
             if manifest_context:
                 for dist in distributions:
@@ -146,7 +153,7 @@ def _write_manifest():
         except Exception as exc:
             if manifest_context:
                 manifest_context.record_operation_error(exc)
-            raise
+            raise click.ClickException(str(exc))
         finally:
             _write_manifest()
         return
@@ -155,8 +162,11 @@ def _write_manifest():
     if metadata_file:
         click.echo(f"[MODE] Deploy from metadata file: {metadata_file}")
         try:
-            with open(metadata_file, "r") as f:
+            with open(metadata_file, "r", encoding="utf-8-sig") as f:
                 metadata = json.load(f)
+            if manifest_context:
+                manifest_context.replay_params["deploy_mode"] = "metadata"
+                manifest_context.replay_params["resolved_metadata"] = metadata
             api_deploy.deploy_from_metadata(
                 metadata, version_id, title, abstract, description, license_url, apikey
             )
@@ -171,7 +181,7 @@ def _write_manifest():
         except Exception as exc:
             if manifest_context:
                 manifest_context.record_operation_error(exc)
-            raise
+            raise click.ClickException(str(exc))
         finally:
             _write_manifest()
         return
@@ -189,6 +199,8 @@ def _write_manifest():
             )
         click.echo("[MODE] Upload & Deploy to DBpedia Databus via Nextcloud")
         click.echo(f"→ Uploading to: {remote}:{path}")
+        if manifest_context:
+            manifest_context.replay_params["deploy_mode"] = "webdav"
         try:
             metadata = webdav.upload_to_webdav(distributions, remote, path, webdav_url)
             api_deploy.deploy_from_metadata(
@@ -205,7 +217,7 @@ def _write_manifest():
         except Exception as exc:
             if manifest_context:
                 manifest_context.record_operation_error(exc)
-            raise
+            raise click.ClickException(str(exc))
         finally:
             _write_manifest()
         return
@@ -444,6 +456,117 @@ def delete(databusuris: List[str], databus_key: str, dry_run: bool, force: bool,
                     err=True,
                 )
 
+@app.group()
+def manifest():
+    """
+    Manifest utilities.
+
+    Includes replay of previously recorded operations from JSON-LD manifests.
+    """
+    pass
+
+
+@manifest.command("replay")
+@click.argument("manifest_path", type=click.Path(exists=True, dir_okay=False))
+@click.option(
+    "--localdir",
+    default=None,
+    help="Override local output directory for download replay.",
+)
+@click.option(
+    "--databus",
+    default=None,
+    help="Override Databus endpoint for replay (example: https://databus.dbpedia.org/sparql).",
+)
+@click.option(
+    "--vault-token",
+    default=None,
+    help="Vault token file path required if manifest auth method is vault_token.",
+)
+@click.option(
+    "--databus-key",
+    default=None,
+    help="Databus API key required if manifest auth method is databus_key. "
+         "Also required for delete replay (never stored in the manifest).",
+)
+@click.option(
+    "--apikey",
+    "apikey",
+    default=None,
+    help="Databus API key required for deploy replay (never stored in the manifest).",
+)
+@click.option(
+    "--force",
+    is_flag=True,
+    default=False,
+    help="For delete replay: skip the interactive confirmation prompt. "
+         "Required for unattended/scripted replay of a delete operation.",
+)
+@click.option(
+    "--dry-run",
+    "dry_run",
+    is_flag=True,
+    default=False,
+    help="Force a dry-run preview even if the original operation wasn't "
+         "one. If the original delete WAS a dry run, replay already "
+         "previews automatically -- this flag cannot turn that off.",
+)
+def manifest_replay(manifest_path, localdir, databus, vault_token, databus_key, force, dry_run, apikey):
+    """
+    Replay a previously recorded manifest operation.
+
+    Currently supports replay of download, delete, and deploy manifests.
+    For delete manifests, an interactive confirmation is required by
+    default -- use --force to skip it for scripted/unattended use, or
+    --dry-run to preview without prompting or deleting. Deploy replay
+    supports classic and metadata-file modes only (not WebDAV).
+    """
+    overrides = {
+        "localDir": localdir,
+        "endpoint": databus,
+        "token": vault_token,
+        "databus_key": databus_key,
+        "api_key": apikey,
+    }
+    # Keep only explicitly provided text overrides
+    overrides = {k: v for k, v in overrides.items() if v is not None}
+    # Flags are always explicit values (False is a real, meaningful default)
+    overrides["force"] = force
+    overrides["dry_run"] = dry_run
+
+    try:
+        replay_info = replay_manifest(manifest_path, overrides=overrides)
+        if replay_info["command"] == "delete" and not replay_info.get("executed", True):
+            if replay_info.get("dry_run"):
+                click.echo("Delete replay: dry run only, nothing was deleted.")
+            else:
+                click.echo("Delete replay cancelled — nothing was deleted.")
+        else:
+            click.echo(f"Replayed command: {replay_info['command']}")
+        click.echo(f"Source manifest: {manifest_path}")
+    except ManifestReplayError as e:
+        raise click.ClickException(str(e))
+    except DownloadAuthError as e:
+        raise click.ClickException(str(e))
+    except ValueError as e:
+        raise click.ClickException(str(e))
+    except Exception as e:
+        raise click.ClickException(str(e))
+
+@manifest.command("summary")
+@click.argument("manifest_path", type=click.Path(exists=True, dir_okay=False))
+def manifest_summary(manifest_path):
+    """
+    Print a readable summary of a recorded manifest operation.
+
+    Reads the existing summary data already stored in the manifest --
+    no new data is collected and no new file is written.
+    """
+    try:
+        manifest = load_manifest(manifest_path)
+        click.echo(format_summary(manifest))
+    except ManifestReplayError as e:
+        raise click.ClickException(str(e))
 
 if __name__ == "__main__":
     app()

databusclient/manifest/__init__.py

@@ -2,4 +2,14 @@
 
 Provides ManifestContext (records operation details in memory)
 and ManifestWriter (serializes to JSON-LD on disk).
-"""
+"""
+from databusclient.manifest.context import ManifestContext
+from databusclient.manifest.writer import ManifestWriter
+from databusclient.manifest.replay import ManifestReplayError, replay_manifest
+
+__all__ = [
+    "ManifestContext",
+    "ManifestWriter",
+    "ManifestReplayError",
+    "replay_manifest",
+]