Jules changes and some model renaming for clarity#14
Open
akutuva21 wants to merge 234 commits into
Open
Conversation
Co-authored-by: google-labs-jules[bot] <161369871+google-labs-jules[bot]@users.noreply.github.com>
* Exported expectString from validate-metadata.js * Added negative test cases to validate-metadata.test.js including: * Non-string values * Null values * Empty strings * Whitespace only strings * Valid strings Co-authored-by: google-labs-jules[bot] <161369871+google-labs-jules[bot]@users.noreply.github.com>
Avoid redundant string allocations by caching the result of \`rawLine.trim()\` in \`parseMetadataYaml\`. This eliminates 2 redundant \`.trim()\` operations per line, drastically reducing unnecessary allocations and processing. Co-authored-by: google-labs-jules[bot] <161369871+google-labs-jules[bot]@users.noreply.github.com>
Replace `content.split(/\r?\n/)` with a zero-allocation `indexOf('\n')` loop in `parseMetadataYaml` inside `scripts/utils.js`. This reduces memory allocations and garbage collection overhead by avoiding the creation of an intermediate array of tokens for every line in the file.
Co-authored-by: akutuva21 <44119804+akutuva21@users.noreply.github.com>
Moved the string/path manipulation logic used to generate the `id` for metadata out of `generateMetadata` and into a new, dedicated `generateId` function. This simplifies the `generateMetadata` function and improves code readability. Co-authored-by: akutuva21 <44119804+akutuva21@users.noreply.github.com>
Adds extensive testing to `scripts/utils.test.js` covering the `parseScalar` function's behavior with arrays, booleans, and edge cases, ensuring regex string replacements like `replace(/^"|"$/g, '')` apply as expected across nested comma elements. Co-authored-by: akutuva21 <44119804+akutuva21@users.noreply.github.com>
Co-authored-by: akutuva21 <44119804+akutuva21@users.noreply.github.com>
This adds a test to verify that `generate-gallery.js` correctly catches errors from `fs.readFileSync` or YAML parsing errors and handles them gracefully by skipping the malformed files without crashing the process. Co-authored-by: akutuva21 <44119804+akutuva21@users.noreply.github.com>
Co-authored-by: akutuva21 <44119804+akutuva21@users.noreply.github.com>
Removed `console.log(yamlContent)` from the `if (dryRun)` block to reduce noise in the standard output. Tested by running `node --test` with all applicable unit tests. Co-authored-by: akutuva21 <44119804+akutuva21@users.noreply.github.com>
…s.js Removed a debugging `console.log` statement from `scripts/apply-gallery-assignments.js` that was generating noise and considered technical debt. Co-authored-by: akutuva21 <44119804+akutuva21@users.noreply.github.com>
…ignments.js Co-authored-by: akutuva21 <44119804+akutuva21@users.noreply.github.com>
Co-authored-by: akutuva21 <44119804+akutuva21@users.noreply.github.com>
Removed a single `console.log` line that printed out the number of bngl files found, improving script command line output cleanliness without changing core functionality. Co-authored-by: akutuva21 <44119804+akutuva21@users.noreply.github.com>
Co-authored-by: akutuva21 <44119804+akutuva21@users.noreply.github.com>
Co-authored-by: akutuva21 <44119804+akutuva21@users.noreply.github.com>
…ments.js Co-authored-by: akutuva21 <44119804+akutuva21@users.noreply.github.com>
Co-authored-by: akutuva21 <44119804+akutuva21@users.noreply.github.com>
Co-authored-by: akutuva21 <44119804+akutuva21@users.noreply.github.com>
Co-authored-by: akutuva21 <44119804+akutuva21@users.noreply.github.com>
Removed noisy output `console.log` statements at the start of the `main()` function in `scripts/backfill-metadata.js`. Co-authored-by: akutuva21 <44119804+akutuva21@users.noreply.github.com>
Added edge case assertions to scripts/utils.test.js for the `parseScalar` utility function. Test cases include handling of mismatched string quotes, mismatched brackets, irregular boolean casings, nested spaces in arrays/strings, and explicit decimal numbers. Co-authored-by: akutuva21 <44119804+akutuva21@users.noreply.github.com>
Removed the extraneous `console.log` statements that print a summary at the end of the `main` function in `scripts/backfill-metadata.js`. This reduces noise in the output and improves code health. Co-authored-by: akutuva21 <44119804+akutuva21@users.noreply.github.com>
…ction issue Co-authored-by: akutuva21 <44119804+akutuva21@users.noreply.github.com>
Replaces the synchronous `fs.readdirSync` depth-first search in `listMetadataFiles` with an asynchronous `fs.promises.readdir` implementation. The refactor leverages `Promise.all` to concurrently scan directories and `try...catch` for robust `ENOENT` handling without redundant `fs.existsSync` checks. This optimizes I/O usage by preventing event loop blocking during file scanning across the repository's `SEARCH_ROOTS`. Co-authored-by: akutuva21 <44119804+akutuva21@users.noreply.github.com>
…mmary Replaces console.log with console.info for the summary output in scripts/backfill-metadata.js. This resolves the code health issue regarding extraneous console logs while preserving the essential summary functionality of the script. Co-authored-by: akutuva21 <44119804+akutuva21@users.noreply.github.com>
Replaced synchronous `listModelFiles` with `listModelFilesAsync` within the async context of `validateMetadataFile` to avoid blocking the Node.js event loop during heavy I/O operations. Co-authored-by: akutuva21 <44119804+akutuva21@users.noreply.github.com>
Replaced synchronous `fs.readFileSync` with asynchronous `fs.promises.readFile` in `loadGalleryCategories` inside `scripts/generate-gallery.js` to prevent blocking the Node.js event loop during initialization. Changed main to await the category load. Co-authored-by: akutuva21 <44119804+akutuva21@users.noreply.github.com>
Co-authored-by: akutuva21 <44119804+akutuva21@users.noreply.github.com>
Co-authored-by: akutuva21 <44119804+akutuva21@users.noreply.github.com>
…versal-15062287209927623304 🔒 Fix Path Traversal in generate-manifest.js output flag
…974916847795327544 🧪 [Testing] Add edge case tests for inferCategory
…ionLine-9573349664910203796 🧪 Add tests for processActionLine to improve backfill metadata coverage
…36943815230223 🧪 test: add tests for processModelLine in backfill-metadata.js
…eid-12012714592641404061 🧪 [testing improvement] Add tests for generateId in backfill-metadata.js
…736384652318 🧪 Add missing tests for expectEnum
…965731491338 🧪 test: add unit tests for getIgnoreDirs
…123594865509 🧪 Add tests for validateMetadataFile
…5298177206964 🧪 Add tests for isCollectionEntry in generate-manifest.js
…3728183734872069 🧹 Code Health: Remove unused `getIgnoreDirs` function and associated dead code
…0449047627298925 🧪 [test] add unit tests for expectArray
…525129266671774 🔒 fix: prevent path traversal on input file read
…0554265 🧪 Add tests for isIgnoredDir in generate-manifest
Update test duplicates in scripts/tests/ and tests/ that were left behind after PR merges changed functions to async and updated normalizeModelKey to preserve hyphens instead of stripping them.
…-manifest.js Add missing DEFAULT_IGNORE_DIRS constant and isIgnoredDir function that were referenced but never defined, causing ReferenceError in CI. Also fix syntax error in test file (extra closing braces).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Harden, optimize, and test RuleHub's metadata/manifest tooling
Summary
This PR is a large fork-sync covering 125 commits to RuleHub's
scripts/build tooling and the generated metadata it produces. Work spans four areas: security hardening, performance (mostly sync→async I/O), test coverage, and cleanup/refactors, plus regeneration of manifests, gallery data, andREADME.md/metadata.yamlfiles to keep CI green.Most commits were authored by @akutuva21, with several co-authored by AI coding agents (
google-labs-jules[bot]).What changed
Security
setNested(block__proto__/constructor/prototype; use plain objects).generate-manifest.Performance
backfill-metadata,generate-manifest,generate-gallery,apply-gallery-assignments, and metadata validation (listModelFiles,findBnglFiles,findAllMetadataFiles,extractModelIds, etc.).Setconstruction out of loops, cachetrim()and RegExp creation, optimizeparseMetadataYamlstring splitting.Tests
expectString/expectArray/expectBoolean,parseScalar,parseMetadataYaml,normalizeModelKey,generateId,inferOrigin/inferCategory,processModelLine/processActionLine,isCollectionEntry,getIgnoreDirs/isIgnoredDir, gallery parsing, migration scripts, and error/edge paths throughout.Refactors & cleanup
parseBnglandextractCategoryMappings.console.logcalls (converted toconsole.infowhere appropriate); drop dead code (getIgnoreDirs, redundant nesting, unused vars).Generated data / CI
README.mdfiles andmetadata.yamlupdates for Published models.manifest.json,manifest-slim.json, andgallery.generated.json; exempt legacy Published models to resolve manifest-drift CI failures.Primary files touched
scripts/backfill-metadata.js,scripts/generate-manifest.js,scripts/apply-gallery-assignments.js,scripts/validate-metadata.js,scripts/generate-gallery.js,scripts/utils.js, their.test.jscounterparts,scripts/migration/*, plus regeneratedmanifest*.json,gallery.generated.json, and manyPublished/**/README.mdandmetadata.yamlfiles.Notes for reviewers
scripts/changes first.Commit breakdown
(Counts overlap where a commit spans categories; total commits = 125.)