retry dictionary registration by leoraba · Pull Request #201 · overture-stack/lyric

leoraba · 2026-05-04T16:46:29Z

Description

Note: This PR is a Part # 3 of a series of a PRs to implement Dictionary migration.

It enables the possibility to retry "failed" Dictionary migration by adding a force flag on the Dictionary Registration endpoint.

Details:

Updated POST /dictionary/register endpoint to accept an optional query param force to Re-register the current active dictionary on the category and retries the data migration.

Related tickets:

[Part 3 - Data migration] Retry migration #177

This PR depends on:

Dictionary Migration - Get migration endpoints #200

…mentation

…on_db

…mentation

…on_db

…mentation

…endpoints

…mentation

…endpoints

…tation

…endpoints

leoraba · 2026-05-04T16:52:51Z

+      409:
+        $ref: '#/components/responses/StatusConflict'


if registering the same Dictionary for the category then it throws a 409 error, unless the "force" flag is true.

joneubank · 2026-05-11T14:29:45Z

+      - name: force
+        description: Re-registers the current active dictionary on the category and retries the data migration.
+        in: query
+        required: false
+        schema:
+          type: boolean
+          default: false


Can you add more detail about this behaviour? I don't know what re-registering a dictionary will do. Is this the same as initiating a migration? If the dictionary version is the same as the current dictionary will it run the migration with the intention of revalidating the data (should have no impact on the current data state)?

description updated to Runs dictionary registration and migration again for this category, even if the same dictionary is already registered. Use this only when a previous migration ended unexpectedly and you must rerun both steps.

joneubank · 2026-05-11T14:30:12Z


+If a category is already using the same dictionary name and version, registration returns `409 Conflict`.
+
+Set the `force` query parameter to `true` to allow re-registration and trigger a migration (or retry) to revalidate existing category data against that dictionary.


Repeating my comment since this is the full documentation about registering dictionaries:

Can you add more detail about this behaviour? I don't know what re-registering a dictionary will do. Is this the same as initiating a migration? If the dictionary version is the same as the current dictionary will it run the migration with the intention of revalidating the data (should have no impact on the current data state)?

doc updated

joneubank · 2026-05-11T14:36:56Z

 				const newMigration: NewDictionaryMigration = {
 					categoryId,
-					fromDictionaryId,
+					fromDictionaryId: fromDictionaryId ?? toDictionaryId,


This is confusing to read - perhaps we should leave it as required and the caller can indicate that fromDictionaryId value should be the same value as toDictionaryId. I would not expect the migration service to assume that if from value is not provided you reuse the to value.

code updated to make fromDictionaryId required

joneubank · 2026-05-11T14:38:52Z

+			if (forceRegistration) {
+				logger.info(
+					LOG_MODULE,
+					`Force flag is true, initiating migration for Category '${foundCategory.name}' 
+					with Dictionary '${savedDictionary.name}' version '${savedDictionary.version}'`,
+				);
+
+				const resultMigration = await initiateMigration({
+					categoryId: foundCategory.id,
+					toDictionaryId: savedDictionary.id,
+					userName: username || '',
+				});
+
+				if (!resultMigration.success) {
+					const errorMessage = `Failed to initiate migration for category '${categoryName}' with error: ${resultMigration.data}`;
+					logger.error(LOG_MODULE, errorMessage);
+					throw new Error(errorMessage);
+				}
+
+				return { dictionary: savedDictionary, category: foundCategory, migrationId: resultMigration.data };
+			}
+
+			throw new StatusConflict(
+				`Category '${categoryName}' with Dictionary '${savedDictionary.name}' version '${savedDictionary.version}' already exists`,
+			);


This is not very DRY, a lot of repeated code from the else block. The only difference is omitting the fromDictionaryId and changed log statements. Is there a way to write this that won't open us up to a future code change that is only made in one branch of this if statement?

refactored this code to make it DRY and reusing the initiateMigration logic.

…tions

leoraba · 2026-05-29T15:45:07Z

PR updated to address feedback received. Ready for code review

…tions

joneubank · 2026-06-09T14:04:25Z

Could you add a bit more description to the PR to explain "what happens if the force flag is used when migration is not in failed state"? For example, does it interrupt a migration thats in progress, does it repeat a completed migration. Any behaviours that are modified by this force flag need to be claerly stated.

leoraba · 2026-06-09T20:41:41Z


 Category groups data that is related and shares the same data structure, for that reason, a category must be associated to a registered dictionary. Over time, if the dictionary requires an update, the category needs to be updates accordingly, See [Dictionary Migration](#dictionary-migration) for more details.

+If a category is already using the same dictionary name and version, registration returns `409 Conflict` by default. If the `force` query parameter is set to `true`, Lyric retries only when the same dictionary is already registered and the latest migration previously failed. If no prior failed migration exists, the `force` flag is ignored.


updated doc to describe force flag behaviour

leoraba · 2026-06-09T20:43:43Z


-			return { dictionary: savedDictionary, category: foundCategory };
+			// check last migration of this category to find if it failed, if it failed and forceRegistration is true,
+			// we will re-initiate the migration with the new dictionary
+			const activeMigration = await getMigrationsByCategoryId(foundCategory.id, { pageSize: 1, page: 1 });
+
+			if (
+				forceRegistration &&
+				activeMigration.result.length === 1 &&
+				activeMigration.result.at(0)?.status === 'FAILED'
+			) {


a migration can be re-run only when last migration has FAILED and force flag is set to true

leoraba · 2026-06-09T20:44:32Z

PR updated, changes on readme and openapi doc, to indicate force flag behaviour: "If the 'force' query parameter is set to 'true', Lyric retries only when the same dictionary is already registered and the latest migration previously failed. If no prior failed migration exists, the 'force' flag is ignored."

* update docs * Dictionary migration - Part 1 - Database changes (#182) * database changes * update dbml * updates migration repository * update docs * migration table index * adding ts config noUncheckedIndexedAccess * rename migration enum status IN_PROGRESS * validate undefined query response * Dictionary migration - Part 2 - Execution migration (#183) * database changes * update dbml * migration services * updates migration repository * update migration service * data validation * refactor migration process * block commit submission * fix sort imports * update docs * migration table index * fix typos and logs * migration audit and logs * migration on worker thread * adding ts config noUncheckedIndexedAccess * rename migration enum status IN_PROGRESS * refactoring migration function with Result * Dictionary Migration - Get migration endpoints (#200) * database changes * update dbml * migration services * updates migration repository * update migration service * data validation * refactor migration process * block commit submission * fix sort imports * update docs * migration table index * fix typos and logs * migration audit and logs * migration on worker thread * GET migration endpoints * fix unit tests * GET migration records * adding ts config noUncheckedIndexedAccess * rename migration enum status IN_PROGRESS * refactoring migration function with Result * default constants pagination * change logs and response NotFound * fix swagger docs * include errors or migration changes * type paginated result * using type paginated result * unit test migration formatter functions * remove unhandled error thrown * return 200 json response * tsdocs updates * swagger docs update * retry dictionary registration (#201) * database changes * update dbml * migration services * updates migration repository * update migration service * data validation * refactor migration process * block commit submission * fix sort imports * update docs * migration table index * fix typos and logs * migration audit and logs * migration on worker thread * GET migration endpoints * fix unit tests * GET migration records * adding ts config noUncheckedIndexedAccess * rename migration enum status IN_PROGRESS * refactoring migration function with Result * default constants pagination * change logs and response NotFound * fix swagger docs * include errors or migration changes * type paginated result * using type paginated result * unit test migration formatter functions * remove unhandled error thrown * retry dictionary registration * updating docs * update documentation * DRY refactor * re-run migration only when. prev has failed * Dictionary Migration - Integration tests (#202) * database changes * update dbml * migration services * updates migration repository * update migration service * data validation * refactor migration process * block commit submission * fix sort imports * update docs * migration table index * fix typos and logs * migration audit and logs * migration on worker thread * GET migration endpoints * fix unit tests * GET migration records * adding ts config noUncheckedIndexedAccess * rename migration enum status IN_PROGRESS * refactoring migration function with Result * default constants pagination * change logs and response NotFound * fix swagger docs * include errors or migration changes * type paginated result * using type paginated result * unit test migration formatter functions * remove unhandled error thrown * retry dictionary registration * updating docs * fix integration tests * update documentation * DRY refactor * fix accumulate submission records test * shutdown test provider * create dictionary registration test * dictionary migration tests * tsdocs * pnpm allow build config * re-run migration only when. prev has failed * remove unused code * use common fixtures and assertion function * missing imports * unit test renaming * optional worker pool configuration * Testing workerpool configuration * change pipeline name case

* Build app using Node 22 (#205) * pnpm allow builds * update Dockerfile with node 22 * fix corepack not being found after deployment without egress * add devctx * #169: Data Dictionary Migration (#210) * update docs * Dictionary migration - Part 1 - Database changes (#182) * database changes * update dbml * updates migration repository * update docs * migration table index * adding ts config noUncheckedIndexedAccess * rename migration enum status IN_PROGRESS * validate undefined query response * Dictionary migration - Part 2 - Execution migration (#183) * database changes * update dbml * migration services * updates migration repository * update migration service * data validation * refactor migration process * block commit submission * fix sort imports * update docs * migration table index * fix typos and logs * migration audit and logs * migration on worker thread * adding ts config noUncheckedIndexedAccess * rename migration enum status IN_PROGRESS * refactoring migration function with Result * Dictionary Migration - Get migration endpoints (#200) * database changes * update dbml * migration services * updates migration repository * update migration service * data validation * refactor migration process * block commit submission * fix sort imports * update docs * migration table index * fix typos and logs * migration audit and logs * migration on worker thread * GET migration endpoints * fix unit tests * GET migration records * adding ts config noUncheckedIndexedAccess * rename migration enum status IN_PROGRESS * refactoring migration function with Result * default constants pagination * change logs and response NotFound * fix swagger docs * include errors or migration changes * type paginated result * using type paginated result * unit test migration formatter functions * remove unhandled error thrown * return 200 json response * tsdocs updates * swagger docs update * retry dictionary registration (#201) * database changes * update dbml * migration services * updates migration repository * update migration service * data validation * refactor migration process * block commit submission * fix sort imports * update docs * migration table index * fix typos and logs * migration audit and logs * migration on worker thread * GET migration endpoints * fix unit tests * GET migration records * adding ts config noUncheckedIndexedAccess * rename migration enum status IN_PROGRESS * refactoring migration function with Result * default constants pagination * change logs and response NotFound * fix swagger docs * include errors or migration changes * type paginated result * using type paginated result * unit test migration formatter functions * remove unhandled error thrown * retry dictionary registration * updating docs * update documentation * DRY refactor * re-run migration only when. prev has failed * Dictionary Migration - Integration tests (#202) * database changes * update dbml * migration services * updates migration repository * update migration service * data validation * refactor migration process * block commit submission * fix sort imports * update docs * migration table index * fix typos and logs * migration audit and logs * migration on worker thread * GET migration endpoints * fix unit tests * GET migration records * adding ts config noUncheckedIndexedAccess * rename migration enum status IN_PROGRESS * refactoring migration function with Result * default constants pagination * change logs and response NotFound * fix swagger docs * include errors or migration changes * type paginated result * using type paginated result * unit test migration formatter functions * remove unhandled error thrown * retry dictionary registration * updating docs * fix integration tests * update documentation * DRY refactor * fix accumulate submission records test * shutdown test provider * create dictionary registration test * dictionary migration tests * tsdocs * pnpm allow build config * re-run migration only when. prev has failed * remove unused code * use common fixtures and assertion function * missing imports * unit test renaming * optional worker pool configuration * Testing workerpool configuration * change pipeline name case * publish committed records to Kafka document topic (#208) * bump version * fix: surface file parse errors with fault isolation and structured logging (#212) Each file in a batch submission is now processed independently: a stream error or schema validation failure on one file is caught and reported without blocking the remaining files. Parse results use a three-state discriminator (ok/invalid/error) and are returned synchronously in the API response, so callers know immediately which files failed and why. Structured logging separates human-readable console output (key=value) from a JSON file transport (LOG_JSON env var) for log aggregator ingestion. Field names and line numbers are logged for schema errors; field values are not (OWASP A03). Bonus: Adds optional sync=false query parameter to the file upload endpoint to allow background parsing for very large batches; when omitted or true, behaviour is synchronous and unchanged from the existing default. - New submissionUtils.spec.ts: 5 tests for submissionInsertDataFromFiles covering ok/invalid/error paths, fault isolation, and multi-file accumulation - Extended schemas.spec.ts with 4 tests for the sync query parameter - Fixed collectRows: source ReadStream errors (ENOENT) were not caught when only the piped csvParse transform had an error listener; both streams now handle errors to prevent uncaught rejections - Updated .mocharc.json to discover co-located src/**/*.spec.ts files (following modules/barrels simplifies relocation later if needed) - Updated tech-debt entry for test placement with concrete migration details --------- Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com>

* Build app using Node 22 (#205) * pnpm allow builds * update Dockerfile with node 22 * fix corepack not being found after deployment without egress * add devctx * #169: Data Dictionary Migration (#210) * update docs * Dictionary migration - Part 1 - Database changes (#182) * database changes * update dbml * updates migration repository * update docs * migration table index * adding ts config noUncheckedIndexedAccess * rename migration enum status IN_PROGRESS * validate undefined query response * Dictionary migration - Part 2 - Execution migration (#183) * database changes * update dbml * migration services * updates migration repository * update migration service * data validation * refactor migration process * block commit submission * fix sort imports * update docs * migration table index * fix typos and logs * migration audit and logs * migration on worker thread * adding ts config noUncheckedIndexedAccess * rename migration enum status IN_PROGRESS * refactoring migration function with Result * Dictionary Migration - Get migration endpoints (#200) * database changes * update dbml * migration services * updates migration repository * update migration service * data validation * refactor migration process * block commit submission * fix sort imports * update docs * migration table index * fix typos and logs * migration audit and logs * migration on worker thread * GET migration endpoints * fix unit tests * GET migration records * adding ts config noUncheckedIndexedAccess * rename migration enum status IN_PROGRESS * refactoring migration function with Result * default constants pagination * change logs and response NotFound * fix swagger docs * include errors or migration changes * type paginated result * using type paginated result * unit test migration formatter functions * remove unhandled error thrown * return 200 json response * tsdocs updates * swagger docs update * retry dictionary registration (#201) * database changes * update dbml * migration services * updates migration repository * update migration service * data validation * refactor migration process * block commit submission * fix sort imports * update docs * migration table index * fix typos and logs * migration audit and logs * migration on worker thread * GET migration endpoints * fix unit tests * GET migration records * adding ts config noUncheckedIndexedAccess * rename migration enum status IN_PROGRESS * refactoring migration function with Result * default constants pagination * change logs and response NotFound * fix swagger docs * include errors or migration changes * type paginated result * using type paginated result * unit test migration formatter functions * remove unhandled error thrown * retry dictionary registration * updating docs * update documentation * DRY refactor * re-run migration only when. prev has failed * Dictionary Migration - Integration tests (#202) * database changes * update dbml * migration services * updates migration repository * update migration service * data validation * refactor migration process * block commit submission * fix sort imports * update docs * migration table index * fix typos and logs * migration audit and logs * migration on worker thread * GET migration endpoints * fix unit tests * GET migration records * adding ts config noUncheckedIndexedAccess * rename migration enum status IN_PROGRESS * refactoring migration function with Result * default constants pagination * change logs and response NotFound * fix swagger docs * include errors or migration changes * type paginated result * using type paginated result * unit test migration formatter functions * remove unhandled error thrown * retry dictionary registration * updating docs * fix integration tests * update documentation * DRY refactor * fix accumulate submission records test * shutdown test provider * create dictionary registration test * dictionary migration tests * tsdocs * pnpm allow build config * re-run migration only when. prev has failed * remove unused code * use common fixtures and assertion function * missing imports * unit test renaming * optional worker pool configuration * Testing workerpool configuration * change pipeline name case * publish committed records to Kafka document topic (#208) * fix: surface file parse errors with fault isolation and structured logging (#212) Each file in a batch submission is now processed independently: a stream error or schema validation failure on one file is caught and reported without blocking the remaining files. Parse results use a three-state discriminator (ok/invalid/error) and are returned synchronously in the API response, so callers know immediately which files failed and why. Structured logging separates human-readable console output (key=value) from a JSON file transport (LOG_JSON env var) for log aggregator ingestion. Field names and line numbers are logged for schema errors; field values are not (OWASP A03). Bonus: Adds optional sync=false query parameter to the file upload endpoint to allow background parsing for very large batches; when omitted or true, behaviour is synchronous and unchanged from the existing default. - New submissionUtils.spec.ts: 5 tests for submissionInsertDataFromFiles covering ok/invalid/error paths, fault isolation, and multi-file accumulation - Extended schemas.spec.ts with 4 tests for the sync query parameter - Fixed collectRows: source ReadStream errors (ENOENT) were not caught when only the piped csvParse transform had an error listener; both streams now handle errors to prevent uncaught rejections - Updated .mocharc.json to discover co-located src/**/*.spec.ts files (following modules/barrels simplifies relocation later if needed) - Updated tech-debt entry for test placement with concrete migration details * docs: log a roadmap item for fixing invalid submissions in place An effort-estimate pass for a submitter complaint found the state machine, re-validation trigger, and delete-by-index scaffolding already support this; the actual gap is that resubmission appends rather than replaces an entity's batch. Logged for a future agent to pick up. * feat: category alias, plus routing Kafka messages by categoryId when a topic is shared (#213) Adds category alias as a first-class feature: assignable at creation, resolvable via GET, and assignable/unassignable after the fact for categories created before this feature existed. Publishes categoryId/categoryAlias on every committed Kafka message, so a consumer sharing one topic across multiple categories, like Maestro, can route correctly, the wrong-index bug this whole thing started from. Review-driven correctness fixes folded in: - Numeric category id always wins over a colliding alias on lookup, ids are permanent, aliases can be assigned/cleared/reassigned - An alias can never be purely numeric, eliminating that collision at the source rather than only handling it gracefully after the fact - categoryId-or-alias resolution now applied consistently across every controller that accepts a categoryId path param (dictionary, submission, audit, migration, submittedData, validator routes); a nonexistent category id or alias now consistently returns 404 everywhere instead of a mix of 400 and 404 * bump version 0.19.0 --------- Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com>

leoraba added 30 commits January 9, 2026 11:10

database changes

74d088a

update dbml

28f683a

migration services

e1ada63

updates migration repository

c1f87dd

Merge branch 'feat/dictionary_migration_db' into feat/migration_imple…

118fc42

…mentation

update migration service

0a26540

data validation

1ec4dc4

refactor migration process

afc4f87

block commit submission

58d4ad8

fix sort imports

b081dd9

Merge branch 'feat/dictionary_migration' into feat/dictionary_migrati…

e6dc14d

…on_db

Merge branch 'feat/dictionary_migration_db' into feat/migration_imple…

608c096

…mentation

Merge branch 'feat/dictionary_migration' into feat/dictionary_migrati…

b23d648

…on_db

update docs

8688240

migration table index

2487bbe

Merge branch 'feat/dictionary_migration_db' into feat/migration_imple…

96b6c63

…mentation

fix typos and logs

d351b2e

migration audit and logs

fb05263

migration on worker thread

b5d6756

GET migration endpoints

f5af236

fix unit tests

d7d17d0

Merge branch 'feat/migration_implementation' into feat/get_migration_…

2d05348

…endpoints

GET migration records

2758202

adding ts config noUncheckedIndexedAccess

b8cd961

rename migration enum status IN_PROGRESS

cdf32cc

Merge branch 'feat/dictionary_migration_db' into feat/migration_imple…

2c889d1

…mentation

Merge branch 'feat/migration_implementation' into feat/get_migration_…

430ea27

…endpoints

Merge branch 'feat/dictionary_migration' into feat/migration_implemen…

c512349

…tation

refactoring migration function with Result

f5994eb

Merge branch 'feat/migration_implementation' into feat/get_migration_…

280395f

…endpoints

leoraba added 3 commits April 30, 2026 14:38

unit test migration formatter functions

e588d0b

remove unhandled error thrown

d8b930d

retry dictionary registration

a7cfcf1

leoraba requested review from JamesTLopez and joneubank May 4, 2026 16:48

leoraba commented May 4, 2026

View reviewed changes

updating docs

a2b05e8

leoraba linked an issue May 4, 2026 that may be closed by this pull request

[Part 3 - Data migration] Retry migration #177

Open

leoraba mentioned this pull request May 4, 2026

Dictionary Migration - Integration tests #202

Merged

joneubank reviewed May 11, 2026

View reviewed changes

Base automatically changed from feat/get_migration_endpoints to feat/dictionary_migration May 12, 2026 16:56

leoraba added 3 commits May 12, 2026 13:07

Merge branch 'feat/dictionary_migration' into feat/retry_failed_migra…

f28c9aa

…tions

update documentation

feb65f5

DRY refactor

ec4a580

leoraba requested a review from joneubank May 29, 2026 15:45

Merge branch 'feat/dictionary_migration' into feat/retry_failed_migra…

7bc39c6

…tions

leoraba requested a review from demariadaniel June 9, 2026 13:44

re-run migration only when. prev has failed

3871c59

leoraba commented Jun 9, 2026

View reviewed changes

joneubank approved these changes Jul 8, 2026

View reviewed changes

leoraba merged commit 62b0e23 into feat/dictionary_migration Jul 9, 2026
2 checks passed

leoraba deleted the feat/retry_failed_migrations branch July 9, 2026 15:13

leoraba mentioned this pull request Jul 9, 2026

#169: Data Dictionary Migration #210

Merged

5 tasks


		If a category is already using the same dictionary name and version, registration returns `409 Conflict`.

		Set the `force` query parameter to `true` to allow re-registration and trigger a migration (or retry) to revalidate existing category data against that dictionary.


		Category groups data that is related and shares the same data structure, for that reason, a category must be associated to a registered dictionary. Over time, if the dictionary requires an update, the category needs to be updates accordingly, See [Dictionary Migration](#dictionary-migration) for more details.

		If a category is already using the same dictionary name and version, registration returns `409 Conflict` by default. If the `force` query parameter is set to `true`, Lyric retries only when the same dictionary is already registered and the latest migration previously failed. If no prior failed migration exists, the `force` flag is ignored.

Uh oh!

Conversation

leoraba commented May 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Details:

Related tickets:

This PR depends on:

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

leoraba commented May 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

joneubank commented Jun 9, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

leoraba commented Jun 9, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

leoraba commented May 4, 2026 •

edited

Loading

leoraba commented May 29, 2026 •

edited

Loading