feat(api)!: remove the local-download surface (POST /v3/download, download_cohort)

CHANGELOG.md

@@ -13,6 +13,17 @@ Refactors, CI, and formatting land in the git history, not here.
 
 ## [Unreleased]
 
+### Removed
+
+- **The local-download surface: `POST /v3/download` (REST) and the `download_cohort` MCP tool**
+  (beta contract change). Both only worked when the server ran on the caller's own machine and
+  errored everywhere else — on the hosted deployment (the common case) the tool's mere presence
+  misled agents into calling it. Downloading through the server is also never the right path:
+  every IDC bucket is public, so direct S3/GCS transfer is strictly more efficient. Retrieval is
+  now manifests/URLs only on every surface — use `get_cohort_urls` / `POST /v3/cohort/manifest.txt`
+  or the ready-to-run `idc` CLI commands in the `build_cohort` response. The
+  `IDC_API_ENABLE_LOCAL_DOWNLOAD` config variable is gone with it.
+
 ### Fixed
 
 - `get_cohort_urls` / `POST /v3/cohort/manifest.txt` with `source=gcs` now return `s3://` URLs

README.md

@@ -79,8 +79,8 @@ needed.
 
 ```bash
 uv run idc-api                                       # REST API → http://127.0.0.1:8000 (Swagger at /v3/docs)
-uv run idc-mcp                                       # MCP server, stdio (local) — can also download files
-uv run idc-mcp --http --host 0.0.0.0 --port 8080     # MCP server, hosted/shared (manifests only)
+uv run idc-mcp                                       # MCP server, stdio (local)
+uv run idc-mcp --http --host 0.0.0.0 --port 8080     # MCP server, hosted/shared
 ```
 
 Quick taste — build a breast-MRI cohort over REST:

dev/api_v3_plan.md

@@ -42,6 +42,10 @@ open; **no authentication** is required of callers.
   the MCP server **in local (stdio) mode** can additionally perform the real download via
   `idc-index`. Hosted/remote MCP behaves like REST (manifests only). No server-side
   staging/zipping (collections reach TBs).
+  *Superseded during beta:* the local-download surface (`POST /v3/download` /
+  `download_cohort`) was removed before 3.0.0 — a tool that exists but errors on the
+  dominant hosted deployment misled agents, and direct S3/GCS transfer (all buckets are
+  public) is strictly better. Retrieval is manifests/URLs only in every mode; see CHANGELOG.
 - **MVP scope:** **Core radiology + discovery** and **Viewer URLs + citations + licenses**.
   Specialized indices (ct/mr/pt, seg/ann/rtstruct, sm) and clinical data are later phases.
 
@@ -67,7 +71,6 @@ src/idc_api/
       viewer.py          # OHIF/SLIM viewer URLs
       citations.py       # DOI-based citations (APA/BibTeX/CSL/Turtle)
       licenses.py        # license breakdown for a selection
-      download.py        # LOCAL-ONLY: real file transfer via idc-index/s5cmd
     models/              # Pydantic request/response models = the shared contract
     schema/              # index column metadata, filterable-attribute + filter defs
   rest/                  # FastAPI app: routers are ~10-line wrappers over services
@@ -86,8 +89,7 @@ reuse, rather than rebuild:
   all ~16 indices).
 - `IDCClient` methods to wrap directly: `get_idc_version()`, `indices_overview` /
   `get_index_schema()` (schema discovery), `get_viewer_URL()`, `citations_from_selection()`
-  (+ `CITATION_FORMAT_*`), `download_from_selection()` / `download_dicom_series()` (local
-  download), `fetch_index()` (later phases).
+  (+ `CITATION_FORMAT_*`), `fetch_index()` (later phases).
 - **Do not** reuse `IDCClient`'s single shared DuckDB connection for serving — DuckDB
   connections aren't thread-safe. Instead `duckdb_backend.py` opens its own **read-only**
   connection over the same Parquet and hands out per-request cursors (see Safety).
@@ -122,8 +124,8 @@ No forked code, no SQL string surgery, no webapp dependency, no shared-secret to
 
 **Retrieval**
 - `get_manifest` — s5cmd manifest + public https URLs (AWS+GCS) + `idc download` command.
-- `download_cohort` — **MCP local mode only**; performs the transfer via `idc-index`.
-  Absent/disabled on the hosted REST + remote MCP surface.
+- `download_cohort` — *planned as MCP-local-only, then removed during beta* (see the
+  superseded note under *Decisions locked with the user*).
 
 ## LLM-first details (apply throughout)
 - **Prescriptive tool descriptions** — every MCP tool says *when to call it*, not just what
@@ -235,7 +237,7 @@ general MCP guidance to treat all tool inputs as untrusted and apply defense in
   guarded `sql_query` + schema discovery + viewer/citations/licenses + manifest/URLs.
   Both REST and MCP. Cloud Run deploy + local stdio MCP entrypoint.
 - **Phase 2:** specialized indices (ct/mr/pt/contrast/volume_geometry, seg/ann/rtstruct,
-  sm/sm_instance) as tables + targeted tools; local `download_cohort`; CDN caching
+  sm/sm_instance) as tables + targeted tools; CDN caching
   (see [caching_and_cdn.md](caching_and_cdn.md)); generated client SDK + examples.
 - **Phase 3:** `BigQueryBackend` behind the same interface for full DICOM metadata,
   per-segment anatomy, and SR radiomics/qualitative measurements; clinical index + tables.

dev/architecture.md

@@ -10,7 +10,7 @@ for *how to work in it* see [`dev/developer_guide.md`](developer_guide.md).
    must be exposable as a well-described MCP tool *and* an HTTP endpoint with no duplicated
    logic.
 2. **Lean on `idc-index`.** Reuse its bundled Parquet index + DuckDB engine; do not
-   reimplement queries, citations, viewer URLs, or downloads from scratch. This avoids any
+   reimplement queries, citations, viewer URLs, or manifests from scratch. This avoids any
    BigQuery/webapp coupling and SQL-string surgery.
 3. **Swappable backend.** Storage sits behind a narrow `QueryBackend` interface so an
    alternative engine could be substituted without touching services or adapters — not a
@@ -81,7 +81,6 @@ Pydantic models:
 | `ViewerService` | [viewer.py](../src/idc_api/core/services/viewer.py) | OHIF/SLIM viewer URLs |
 | `CitationsService` | [citations.py](../src/idc_api/core/services/citations.py) | DOI-based citations |
 | `LicenseService` | [licenses.py](../src/idc_api/core/services/licenses.py) | license breakdown |
-| `DownloadService` | [download.py](../src/idc_api/core/services/download.py) | local-only file transfer via idc-index |
 
 ### `core/models.py` — the shared contract
 [models.py](../src/idc_api/core/models.py) holds the Pydantic request/response models returned
@@ -179,13 +178,12 @@ Full threat model and primary-source references (OWASP, DuckDB Securing guide) a
 | | REST API | MCP — local (stdio) | MCP — hosted (http) |
 |---|---|---|---|
 | Runs | hosted (Cloud Run) | on the user's machine | hosted |
-| Filesystem access to caller | no | **yes** | no |
-| `download_cohort` / `POST /download` | 501 (disabled) | performs real transfer via idc-index | disabled |
-| Data retrieval | manifest + URLs + `idc` command | real download **or** manifest | manifest + URLs |
+| Data retrieval | manifest + URLs + `idc` command | manifest + URLs + `idc` command | manifest + URLs + `idc` command |
 
-`main()` in [`mcp/server.py`](../src/idc_api/mcp/server.py) flips
-`settings.enable_local_download` on for stdio. `DownloadService.available()` reads that flag at
-call time.
+Retrieval is deliberately **manifests/URLs only** in every mode: all IDC buckets are public, so
+the caller transfers directly from S3/GCS (`idc` CLI / s5cmd) — strictly faster than proxying
+bytes through the service, and identical behavior local or hosted. (A local-only download
+endpoint/tool existed pre-3.0.0 and was removed in beta; see CHANGELOG.)
 
 The hosted HTTP transport is configured **stateless** (`stateless_http=True`,
 `json_response=True` on the `FastMCP(...)` constructor) so each request is self-contained and

dev/developer_guide.md

@@ -198,7 +198,6 @@ Environment variables (prefix `IDC_API_`), defined in
 | `SQL_TIMEOUT_SECONDS` | 30 | Statement timeout |
 | `DEFAULT_PAGE_SIZE` / `MAX_PAGE_SIZE` | 100 / 5000 | Manifest paging |
 | `MANIFEST_HARD_CAP` | 100000 | Max series a single manifest enumerates |
-| `ENABLE_LOCAL_DOWNLOAD` | false | Allow real downloads (stdio MCP sets this) |
 | `CORS_ALLOW_ORIGINS` / `HOST` / `PORT` | `["*"]` / 127.0.0.1 / 8000 | REST serving |
 | `SQL_LOG_MODE` / `SQL_LOG_CHARS` | snippet / 200 | How `run_sql`/`POST /v3/sql` renders in the audit log: `snippet` (first `SQL_LOG_CHARS` chars) or `hash` (a short digest, no query text) |
 | `BUILD` | (unset) | Deploy stamp appended to the MCP `serverInfo.version` |

docs/user-guide.md

@@ -56,7 +56,7 @@ understanding, because picking the right one makes everything else easy:
 |---|---|---|---|
 | **Discovery** | "What exists? What can I filter on?" | `GET /v3/version`, `/v3/stats`, `/v3/collections`, `/v3/collections/{id}`, `/v3/analysis_results`, `/v3/attributes`, `/v3/attributes/{attr}/values` | `get_idc_version`, `get_stats`, `list_collections`, `get_collection`, `list_analysis_results`, `list_attributes`, `get_attribute_values` |
 | **Cohort** | "How big is *my* selection, and what's in it?" | `POST /v3/cohort/counts`, `POST /v3/cohort/manifest` | `build_cohort` |
-| **Retrieval** | "Give me the download links / files" | `POST /v3/cohort/manifest.txt`, `POST /v3/download` | `get_cohort_urls`, `download_cohort` |
+| **Retrieval** | "Give me the download links" | `POST /v3/cohort/manifest.txt` | `get_cohort_urls` |
 | **SQL** | "Run my custom query" + schema | `GET /v3/tables`, `/v3/tables/{table}`, `POST /v3/sql` | `list_tables`, `get_table_schema`, `run_sql` |
 | **Side tools** | View / cite / license-check a cohort | `GET /v3/viewer-url`, `POST /v3/citations`, `POST /v3/licenses` | `get_viewer_url`, `get_citations`, `get_licenses` |
 
@@ -238,7 +238,6 @@ uv run idc-api          # http://127.0.0.1:8000  — Swagger UI at /v3/docs
 | `GET /v3/viewer-url` | OHIF/SLIM viewer link for a study/series |
 | `POST /v3/citations` | Citations for a cohort |
 | `POST /v3/licenses` | License breakdown for a cohort |
-| `POST /v3/download` | Local download convenience (returns 501 unless enabled) — prefer `/cohort/manifest.txt` |
 
 ### Worked examples
 
@@ -325,22 +324,13 @@ curl -s localhost:8000/v3/citations \
 curl -s 'localhost:8000/v3/viewer-url?study_instance_uid=1.3.6.1.4.1.14519.5.2.1.7009.9004.983700485806071099502442051273'
 ```
 
-**Local download** — only when the server runs in local mode (otherwise returns `501`); use
-`dry_run` to preview the plan without transferring:
-
-```bash
-curl -s localhost:8000/v3/download \
-  -H 'content-type: application/json' \
-  -d '{"download_dir": "/data/idc", "collection_id": ["nlst"], "dry_run": true}'
-```
-
 ---
 
 ## 3. Using the MCP server (LLM agents)
 
 ```bash
-uv run idc-mcp                                       # stdio (local) — can also download files
-uv run idc-mcp --http --host 0.0.0.0 --port 8080     # hosted/shared (manifests only)
+uv run idc-mcp                                       # stdio (local)
+uv run idc-mcp --http --host 0.0.0.0 --port 8080     # hosted/shared
 ```
 
 ### Tools, by surface
@@ -350,7 +340,7 @@ uv run idc-mcp --http --host 0.0.0.0 --port 8080     # hosted/shared (manifests
 - **Schema (for SQL):** `list_tables`, `get_table_schema`
 - **Clinical data:** `list_clinical_tables`, `get_clinical_table_schema`, `get_clinical_table`
 - **Cohort / query:** `build_cohort`, `run_sql`
-- **Retrieval & side tools:** `get_cohort_urls`, `download_cohort`, `get_viewer_url`,
+- **Retrieval & side tools:** `get_cohort_urls`, `get_viewer_url`,
   `get_citations`, `get_licenses`
 - **Resources:** `idc://guide` (data model + recommended workflow), `idc://tables`,
   `idc://schema/{table}`
@@ -411,28 +401,19 @@ configured stateless with plain-JSON responses** — each request is self-contai
 - **Session-bound MCP features are not available** (server→client sampling, elicitation,
   resource subscriptions, streamed progress) — this server exposes only client-initiated tools
   + static resources, so it doesn't use them.
-- **`download_cohort` can't write to your machine** over HTTP — retrieval returns a manifest +
-  URLs instead (see *Local vs hosted* below). Use the stdio config above when you want real
-  downloads.
 
 Operator-side detail (deploy command, host-header / DNS-rebinding settings, the autoscaling
 rationale) is in [deployment.md](../dev/deployment.md).
 
-### Local vs hosted (downloads)
-
-An MCP server can run **locally** (stdio, on your machine — it can write files, so
-`download_cohort` actually fetches DICOM via `idc-index`/s5cmd) or **hosted** (HTTP, shared —
-no filesystem access, so retrieval returns a manifest + public URLs + an `idc download`
-command instead). Same tools, two behaviors.
-
 ---
 
 ## 4. Getting the data
 
-**Recommended: get a manifest, then pull the files directly from S3/GCS.** All series URLs
-point at **public AWS S3 and GCS buckets — no credentials needed** — so the transfer never
-goes through the API server, works the same against the hosted or a local instance, and scales
-to whole collections. Two ways to do it:
+**Get a manifest, then pull the files directly from S3/GCS.** All series URLs point at
+**public AWS S3 and GCS buckets — no credentials needed** — so the transfer never goes through
+the API server (the server never moves bytes; retrieval always means URLs/manifests), works
+the same against the hosted or a local instance, and scales to whole collections. Two ways to
+do it:
 
 1. **Whole collection** — the simplest path; `cohort/manifest`'s download payload emits it for
    you when your filter is a single `collection_id`:
@@ -452,14 +433,6 @@ to whole collections. Two ways to do it:
 
 Install the CLI with `pip install idc-index` (provides the `idc` command).
 
-**Local download mode** — `POST /v3/download` (REST) or `download_cohort` (MCP) drives that
-same `idc-index`/s5cmd transfer for you as a convenience, but it works **only when the server
-itself runs on your machine** (hosted deployments return a clear error instead). It doesn't
-move data any faster than options 1–2 above, so prefer those for bulk transfers or when
-scripting against the hosted service; reach for this only when you're already running the
-server locally and want the tool/endpoint to do the transfer for you. Start with `dry_run` to
-report size, confirm, then run for real.
-
 ---
 
 ## 5. Guarded SQL — why it's safe
@@ -506,7 +479,6 @@ Environment variables (prefix `IDC_API_`):
 | `DEFAULT_PAGE_SIZE` | `100` | Default `cohort/manifest` page size |
 | `MAX_PAGE_SIZE` | `5000` | Upper bound on page size |
 | `MANIFEST_HARD_CAP` | `100000` | Max series enumerated into a manifest |
-| `ENABLE_LOCAL_DOWNLOAD` | `false` | Allow `download` to write files locally |
 | `CORS_ALLOW_ORIGINS` | `["*"]` | Allowed CORS origins (REST). List value — set as JSON, e.g. `["https://app.example.com"]` |
 | `HSTS_MAX_AGE` | `31536000` | `Strict-Transport-Security` max-age (seconds) added to every REST and hosted-MCP response. Default is the production value (1 year); dev/test deploys use `3600` so a bad deploy can't lock browsers out for a year. `0` disables the header |
 | `HOST` / `PORT` | `127.0.0.1` / `8000` | REST bind address |

src/idc_api/core/context.py

@@ -10,7 +10,6 @@
     ClinicalService,
     CohortService,
     DiscoveryService,
-    DownloadService,
     LicenseService,
     ManifestService,
     QueryService,
@@ -30,7 +29,6 @@ def __init__(self, settings: Settings | None = None):
         self.viewer = ViewerService(self.backend)
         self.citations = CitationsService(self.backend)
         self.licenses = LicenseService(self.backend)
-        self.download = DownloadService(self.backend, self.settings)
 
     def close(self) -> None:
         self.backend.close()

src/idc_api/core/errors.py

@@ -44,11 +44,3 @@ class QueryTimeoutError(IDCAPIError):
 class ResultTooLargeError(IDCAPIError):
     code = "result_too_large"
     status = 400
-
-
-class UnsupportedOperationError(IDCAPIError):
-    """The capability exists but is disabled in this deployment mode (e.g. local download
-    on a hosted server)."""
-
-    code = "unsupported_operation"
-    status = 501

src/idc_api/core/services/__init__.py

@@ -8,7 +8,6 @@
 from .clinical import ClinicalService
 from .cohort import CohortService
 from .discovery import DiscoveryService
-from .download import DownloadService
 from .licenses import LicenseService
 from .manifest import ManifestService
 from .query import QueryService
@@ -19,7 +18,6 @@
     "ClinicalService",
     "CohortService",
     "DiscoveryService",
-    "DownloadService",
     "LicenseService",
     "ManifestService",
     "QueryService",

src/idc_api/core/services/manifest.py

@@ -1,6 +1,6 @@
 """Manifest generation: public series URLs (AWS + GCS), s5cmd-style manifest text, and
-ready-to-run ``idc`` CLI commands for a cohort. The service never moves bytes — local
-download lives in ``download.py`` (MCP local mode only)."""
+ready-to-run ``idc`` CLI commands for a cohort. The service never moves bytes — callers
+download directly from the public S3/GCS buckets (``idc`` CLI / s5cmd)."""
 
 from __future__ import annotations