Add export history support for the DurableTask SDK

[bachuv]

Issue describing the changes in this PR

Adds the Export History feature to the Java SDK, achieving full .NET parity with durabletask-dotnet/src/ExportHistory.

This feature enables bulk exporting of orchestration instance history to Azure Blob Storage with support for batch (fixed time window) and continuous (real-time tail) export modes, and JSONL+gzip or JSON output formats.

What's included

New exporthistory Gradle module (durabletask-exporthistory)

Category	Files	Description
Entity	ExportJob	Durable entity with 7-operation state machine (Create, Get, Run, CommitCheckpoint, MarkAsCompleted, MarkAsFailed, Delete)
Orchestrators	ExportJobOrchestrator, ExecuteExportJobOperationOrchestrator	Main batch loop with two-level retry (activity + batch) matching .NET; entity operation dispatcher
Activities	ExportInstanceHistoryActivity, ListTerminalInstancesActivity	Blob upload with SHA-256 naming, JSONL+gzip/JSON serialization; instance ID pagination via ListInstanceIds RPC
Client abstractions	ExportHistoryClient, ExportHistoryJobClient + Default impls	Job lifecycle management (create, describe, list, delete)
Static helpers	ExportHistoryClients, ExportHistoryWorkers	Module-layering-safe composition — no circular dependency between client and exporthistory modules
Models	17 POJOs	All Jackson-serializable with default constructors for entity state round-tripping
Constants	ExportJobOperationNames, ExportHistoryConstants	Operation name constants, deterministic orchestrator instance ID derivation
Exceptions	ExportJobNotFoundException, ExportJobInvalidTransitionException	Custom exception types matching .NET
Options	ExportHistoryStorageOptions with Builder	Azure Blob Storage connection string + container config

Changes to client module

File	Change
DurableTaskClient	Added getOrchestrationHistory() and listInstanceIds() default methods
DurableTaskGrpcClient	gRPC implementations using StreamInstanceHistory and ListInstanceIds RPCs (already in proto)
OrchestrationHistoryEvent	New transport-neutral history event POJO (decouples public API from proto types)
OrchestrationHistoryEventMapper	Proto → SDK model mapper for all 28 event types
InstanceIdPage	Pagination type for listInstanceIds()

Sample (samples/)

Spring Boot web app matching the .NET ExportHistoryWebApp sample:

Endpoint	Description
POST /export-jobs	Create a new export job
GET /export-jobs/{id}	Get export job by ID
GET /export-jobs/list	List jobs with optional filters (status, prefix, date range, pagination)
DELETE /export-jobs/{id}	Delete an export job

Run: ./gradlew runExportHistorySample (requires DTS emulator + Azurite, port 5009)

Tests (105 tests, 0 failures)

Test Class	Tests	What it covers
ExportJobCreationOptionsTest	18	Mode/time validation, batch size limits, status filters, defaults
ExportJobTransitionsTest	32	All 16 state machine combinations exhaustively
ExportJobTest	12	Entity operations via Mockito (Create, Get, CommitCheckpoint, MarkAsCompleted, MarkAsFailed, Delete)
ExportInstanceHistoryActivityTest	10	SHA-256 blob naming, JSONL/JSON serialization, camelCase properties
OrchestrationHistoryEventMapperTest	9	Proto→SDK mapping for 7 event types + null/unknown cases
OrchestrationHistoryEventTest	6	Constructor validation, data immutability
ExportHistoryStorageOptionsTest	6	Builder validation (connection string, container required)
ExportFormatTest	5	Default format, schema version, null checks
ExportDestinationTest	4	Container validation
ExportHistoryConstantsTest	3	Deterministic orchestrator ID derivation

Key design decisions

• Module layering: ExportHistoryClients.create() and ExportHistoryWorkers.register() are static helpers in the exporthistory module (not methods on DurableTaskGrpcClientBuilder/DurableTaskGrpcWorkerBuilder) to avoid a circular dependency between client → exporthistory → client.
• Transport-neutral API: getOrchestrationHistory() returns OrchestrationHistoryEvent (SDK type), not proto HistoryEvent, keeping the public API stable.
• Jackson compatibility: All model classes that participate in entity state round-tripping have no-arg default constructors. jackson-datatype-jsr310 registered for Instant serialization.
• Two-level retry (mirrors the .NET ExportJobOrchestrator):
  • Activity-level: each ExportInstanceHistoryActivity invocation is retried up to 3 times with exponential backoff — 15s before attempt 2, 30s before attempt 3 (maxRetryInterval=60s bounds the formula in case the policy is widened).
  • Batch-level: if any instance in a batch still fails after activity retries, the whole batch is retried up to 3 times — 60s before attempt 2, 120s before attempt 3 (MAX_BACKOFF_SECONDS=300 bounds the formula in case MAX_RETRY_ATTEMPTS is raised). After the third batch attempt fails, the orchestration throws and the job is marked failed.
• DTS-only: Requires the gRPC sidecar protocol (StreamInstanceHistory, ListInstanceIds RPCs). Not compatible with Azure Storage, MSSQL, or Netherite backends.

How customers use this feature

1. Add the dependency

dependencies {
    implementation 'com.microsoft:durabletask-client:1.9.0'
    implementation 'com.microsoft:durabletask-exporthistory:1.9.0'
}

2. Configure and register the worker

// Storage options for export destination
ExportHistoryStorageOptions storageOptions = ExportHistoryStorageOptions.newBuilder()
        .connectionString("DefaultEndpointsProtocol=https;AccountName=...;AccountKey=...;")
        .containerName("export-history")
        .build();

// Build the worker — registers entity, orchestrators, and activities
DurableTaskGrpcWorkerBuilder workerBuilder = /* your existing worker builder */;
ExportHistoryWorkers.register(workerBuilder, client, storageOptions);
DurableTaskGrpcWorker worker = workerBuilder.build();
worker.start();

3. Create and manage export jobs

// Create the export history client
ExportHistoryClient exportClient = ExportHistoryClients.create(durableTaskClient, storageOptions);

// Create a batch export job (exports history for a fixed time window)
ExportJobCreationOptions options = new ExportJobCreationOptions(
        null,                                    // auto-generate job ID
        ExportMode.BATCH,                        // or ExportMode.CONTINUOUS
        Instant.parse("2026-01-01T00:00:00Z"),   // completedTimeFrom
        Instant.parse("2026-02-01T00:00:00Z"),   // completedTimeTo (required for BATCH)
        new ExportDestination("my-container", "exports/january/"),
        ExportFormat.DEFAULT,                    // JSONL + gzip
        Arrays.asList(OrchestrationRuntimeStatus.COMPLETED),  // filter
        100);                                    // max instances per batch

ExportHistoryJobClient jobClient = exportClient.createJob(options);

// Check progress
ExportJobDescription desc = jobClient.describe();
System.out.println("Status: " + desc.getStatus());       // ACTIVE, COMPLETED, FAILED
System.out.println("Exported: " + desc.getExportedInstances());

// List all jobs
for (ExportJobDescription job : exportClient.listJobs(null)) {
    System.out.println(job.getJobId() + " - " + job.getStatus());
}

// Delete a job (also terminates its orchestration)
jobClient.delete();

4. Export output

Exported files are written to Azure Blob Storage:

• Path: <container>/<prefix>/<sha256-hash>.jsonl.gz (or .json)
• Naming: SHA-256 hash of (completedTimestamp|instanceId) ensures idempotent writes
• JSONL format (default): One history event per line, gzip compressed
• JSON format: JSON array of all history events, uncompressed
• Blob metadata: instanceId tag on each blob

Configuration reference

Option	Required	Default	Description
connectionString	Yes	—	Azure Storage connection string for blob export destination
containerName	Yes	—	Blob container name
prefix	No	<mode>-<jobId>/	Path prefix for exported blobs
mode	Yes	—	BATCH (fixed window) or CONTINUOUS (real-time tail)
completedTimeFrom	Yes	—	Inclusive start of the completed time filter
completedTimeTo	BATCH only	—	Inclusive end of the completed time filter
format	No	JSONL+gzip, schema v1.0	Output format (JSONL or JSON)
runtimeStatus	No	All terminal	Filter: COMPLETED, FAILED, TERMINATED
maxInstancesPerBatch	No	100	Instances per batch (1–1000)

Pull request checklist

• My changes do not require documentation changes
  • Otherwise: Documentation issue linked to PR
• My changes are added to the CHANGELOG.md
• I have added all required tests (Unit tests, E2E tests)

Additional information

• Verified end-to-end with DTS emulator (Docker) + Azurite: POST /export-jobs → 201, GET /export-jobs/{id} → 200, DELETE /export-jobs/{id} → 204, GET /export-jobs/nonexistent → 404

[Copilot]

Pull request overview

This PR adds an Export History feature to the DurableTask Java SDK (aiming for .NET parity) by introducing a new exporthistory module that can bulk-export orchestration instance history to Azure Blob Storage (batch and continuous modes, JSONL+gzip or JSON), plus core client APIs to stream instance history and list terminal instances for pagination.

Changes:

• Added new exporthistory Gradle module containing the export job entity, orchestrators, activities, client abstractions, and supporting models/constants/exceptions.
• Extended the core client module with getOrchestrationHistory() and listInstanceIds() APIs, plus transport-neutral history event mapping/types.
• Added a Spring Boot sample app and Gradle run task to demonstrate managing export jobs via REST.

Reviewed changes

Copilot reviewed 64 out of 64 changed files in this pull request and generated 19 comments.

Show a summary per file

File	Description
settings.gradle	Adds :exporthistory to the multi-module Gradle build.
samples/build.gradle	Adds runExportHistorySample task and depends on :exporthistory.
samples/src/main/java/io/durabletask/samples/ExportHistorySample.java	Spring Boot entry point wiring up worker registration + REST app.
samples/src/main/java/io/durabletask/samples/ExportJobController.java	REST endpoints to create/get/list/delete export jobs.
samples/src/main/java/io/durabletask/samples/CreateExportJobRequest.java	Request DTO for export job creation API.
exporthistory/build.gradle	New module build with Azure Blob Storage + Jackson dependencies and tests.
exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/options/ExportHistoryStorageOptions.java	Storage configuration options with builder validation.
exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/constants/ExportHistoryConstants.java	Deterministic export orchestrator instance ID derivation.
exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/constants/ExportJobOperationNames.java	Operation name constants for the export job entity.
exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/exception/ExportJobNotFoundException.java	Custom exception for missing jobs.
exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/exception/ExportJobInvalidTransitionException.java	Custom exception for invalid job state transitions.
exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/entity/ExportJob.java	Durable entity implementing the export job lifecycle and progress tracking.
exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/orchestrations/ExportJobOrchestratorFactory.java	Factory registration wrapper for the main export orchestrator.
exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/orchestrations/ExportJobOrchestrator.java	Main export loop orchestrator with activity + batch retry and checkpointing.
exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/orchestrations/ExecuteExportJobOperationOrchestratorFactory.java	Factory for the entity-operation dispatcher orchestrator.
exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/orchestrations/ExecuteExportJobOperationOrchestrator.java	Orchestrator that routes “operation requests” to the target export-job entity.
exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/activities/ExportInstanceHistoryActivityFactory.java	Activity factory injecting client + storage options.
exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/activities/ExportInstanceHistoryActivity.java	Exports history for one instance to Azure Blob Storage (JSONL+gzip / JSON).
exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/activities/ListTerminalInstancesActivityFactory.java	Activity factory for instance listing.
exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/activities/ListTerminalInstancesActivity.java	Lists terminal instance IDs using the new listInstanceIds client API.
exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/client/ExportHistoryClients.java	Static factory to create an ExportHistoryClient without module layering cycles.
exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/client/ExportHistoryWorkers.java	Static helper to register entity/orchestrations/activities on a worker builder.
exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/client/ExportHistoryClient.java	Public export history client abstraction (create/get/list/delete).
exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/client/ExportHistoryJobClient.java	Per-job client abstraction.
exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/client/DefaultExportHistoryClient.java	Default implementation built on durable entities and entity queries.
exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/client/DefaultExportHistoryJobClient.java	Default per-job client (create/describe/delete) via operation orchestrator.
exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/client/ExportJobQueryPageable.java	Auto-paginating iterable for listing export jobs.
exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/models/ExportMode.java	Export mode enum (BATCH/CONTINUOUS).
exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/models/ExportJobStatus.java	Export job lifecycle status enum.
exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/models/ExportJobTransitions.java	Transition validation logic for export job state changes.
exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/models/ExportJobState.java	Entity-persisted state for export job progress and configuration.
exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/models/ExportJobCreationOptions.java	User-facing creation options with validation and defaults.
exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/models/ExportJobConfiguration.java	Derived internal configuration stored on the entity.
exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/models/ExportFilter.java	Completed-time/runtime-status filter for instance enumeration.
exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/models/ExportDestination.java	Blob container/prefix destination model.
exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/models/ExportFormatKind.java	Output format kind enum (JSONL/JSON).
exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/models/ExportFormat.java	Output format model (kind + schema version).
exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/models/ExportRequest.java	Activity input for exporting a single instance’s history.
exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/models/ExportResult.java	Activity output for a single instance export attempt.
exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/models/ExportFailure.java	Failure detail model for a failed export attempt.
exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/models/ExportCheckpoint.java	Cursor/checkpoint model for key-based pagination.
exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/models/CommitCheckpointRequest.java	Entity operation input for committing checkpoint/progress.
exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/models/ListTerminalInstancesRequest.java	Activity input for listing terminal instances.
exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/models/InstancePage.java	Activity output: page of instance IDs + checkpoint.
exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/models/ExportJobRunRequest.java	Orchestrator input for running an export job.
exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/models/ExportJobQuery.java	Client-side query model for listing jobs with filters.
exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/models/ExportJobOperationRequest.java	Orchestrator input for dispatching an entity operation.
exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/models/ExportJobDescription.java	Job description model returned from client APIs.
exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/models/BatchExportResult.java	Internal orchestrator helper model for batch results.
exporthistory/src/test/java/com/microsoft/durabletask/exporthistory/options/ExportHistoryStorageOptionsTest.java	Unit tests for storage options builder validation.
exporthistory/src/test/java/com/microsoft/durabletask/exporthistory/models/ExportJobTransitionsTest.java	Unit tests for state machine transitions.
exporthistory/src/test/java/com/microsoft/durabletask/exporthistory/models/ExportJobCreationOptionsTest.java	Unit tests for options validation/defaults.
exporthistory/src/test/java/com/microsoft/durabletask/exporthistory/models/ExportFormatTest.java	Unit tests for export format validation/defaults.
exporthistory/src/test/java/com/microsoft/durabletask/exporthistory/models/ExportDestinationTest.java	Unit tests for destination validation.
exporthistory/src/test/java/com/microsoft/durabletask/exporthistory/constants/ExportHistoryConstantsTest.java	Unit tests for deterministic orchestrator ID derivation.
exporthistory/src/test/java/com/microsoft/durabletask/exporthistory/entity/ExportJobTest.java	Unit tests for entity operations and transitions via Mockito.
exporthistory/src/test/java/com/microsoft/durabletask/exporthistory/activities/ExportInstanceHistoryActivityTest.java	Unit tests for blob naming and JSON serialization logic.
client/src/main/java/com/microsoft/durabletask/DurableTaskClient.java	Adds default APIs for history retrieval and instance ID listing.
client/src/main/java/com/microsoft/durabletask/DurableTaskGrpcClient.java	Implements new APIs using StreamInstanceHistory and ListInstanceIds RPCs.
client/src/main/java/com/microsoft/durabletask/OrchestrationHistoryEvent.java	Adds transport-neutral history event POJO.
client/src/main/java/com/microsoft/durabletask/OrchestrationHistoryEventMapper.java	Adds proto→SDK mapping logic for history events.
client/src/main/java/com/microsoft/durabletask/InstanceIdPage.java	Adds pagination container type for listInstanceIds().
client/src/test/java/com/microsoft/durabletask/OrchestrationHistoryEventTest.java	Unit tests for the new history event POJO.
client/src/test/java/com/microsoft/durabletask/OrchestrationHistoryEventMapperTest.java	Unit tests for proto→SDK history mapping behavior.

[Copilot]

Pull request overview

Copilot reviewed 65 out of 65 changed files in this pull request and generated 4 comments.

Comments suppressed due to low confidence (5)

exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/models/ExportJobRunRequest.java:1

• The @Nonnull contract on getJobEntityId() is violated by the default constructor setting jobEntityId to null. This pattern appears in multiple Jackson-constructible models in this PR and can mislead callers/tools. Make the getter @Nullable (and/or enforce non-null initialization) so annotations reflect actual runtime behavior.
  exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/models/InstancePage.java:1
• Nullability annotations are internally inconsistent: the constructor requires nextCheckpoint as @Nonnull, but the default constructor sets it to null and the getter is @Nullable. Align these by either (a) allowing nullable checkpoints in the constructor, or (b) ensuring a non-null checkpoint default and making the getter @Nonnull.
  exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/models/InstancePage.java:1
• Nullability annotations are internally inconsistent: the constructor requires nextCheckpoint as @Nonnull, but the default constructor sets it to null and the getter is @Nullable. Align these by either (a) allowing nullable checkpoints in the constructor, or (b) ensuring a non-null checkpoint default and making the getter @Nonnull.
  exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/orchestrations/ExportJobOrchestrator.java:1
• ExportJobConfiguration defines maxParallelExports, but the orchestrator currently schedules activities for the entire instanceIds batch at once. With maxInstancesPerBatch up to 1000, this can create very high fan-out, stressing the worker, storage, and sidecar. Throttle concurrency based on config.getMaxParallelExports() (e.g., process in chunks of N and await each chunk) so throughput is controlled and predictable.
  samples/src/main/java/io/durabletask/samples/ExportHistorySample.java:1
• The sample prints 'started' before Spring Boot actually starts (since SpringApplication.run blocks until shutdown). Also, the worker is started before the web app is known to be healthy and is not stopped on shutdown. For more reliable sample behavior, set server.port before any startup logging and add a shutdown hook (or Spring lifecycle bean) to stop the Durable Task worker when the app exits.

[YunchuWang]

Code review findings from Copilot CLI.

[YunchuWang]

@bachuv thanks for the export-history PR. I found one approval blocker in the list-jobs auto-pagination path that should be fixed before merge, plus one public API validation issue that I think should be addressed while you are here.

Blocking: listJobs() item iteration can stop after an empty filtered page even when more backend pages exist.

ExportJobQueryPageable.iterator().hasNext() fetches at most one page when the current page is exhausted:

java if (!this.hasMorePages) { return false; } fetchNextPage(); return this.currentPage.hasNext();

But DefaultExportHistoryClient.listJobs() applies status / createdFrom / createdTo filters after the entity query page is returned. That means a backend page can contain entities, filter down to
esult.getJobs().isEmpty(), and still return a non-empty continuation token. In that case currentPage.hasNext() returns alse, an enhanced or loop over client.listJobs(query) stops, and matching jobs on later backend pages are never fetched.

Example: caller filters for COMPLETED with pageSize = 2; backend page 1 has two ACTIVE jobs and a continuation token; backend page 2 has two COMPLETED jobs. The current iterator returns no jobs because page 1 filters to empty and hasNext() stops before following the continuation token.

Suggested fix:

java @Override public boolean hasNext() { while (!this.currentPage.hasNext() && this.hasMorePages) { fetchNextPage(); } return this.currentPage.hasNext(); }

Please also add a regression test where the first backend page filters to zero results but has a continuation token, and the next page contains a matching job.

Should fix: validate DurableTaskGrpcClient.listInstanceIds(..., pageSize, ...).

The export path validates maxInstancesPerBatch to 1..1000, but the new public client API accepts any int pageSize and sends it directly to the sidecar. A direct caller can pass �, negative values, or Integer.MAX_VALUE. Please validate the public API defensively, ideally matching the export limit (1..1000), and consider validating completedTimeTo is after completedTimeFrom when both are supplied.

I checked the other potential concerns I had: the continueAsNew placement matches the .NET implementation and returns before scheduling work in that cycle, and the failed-batch checkpoint path preserves the last non-null checkpoint. Those are not blockers from my side.

[Copilot]

Pull request overview

Copilot reviewed 69 out of 69 changed files in this pull request and generated 16 comments.