Dev: Build a CSV parsing utility - Historical data parser

[naman0r]

ℹ️ Issue

Closes #10

📝 Description

Adds a parser for the historical records spreadsheet (the All Daily Records tab). The upload lambda can call it to check a file before anything is written to daily_records. It returns either one record per day or a list of every problem in the file, each with the sheet row and column, so staff can fix everything in one pass. The error list matches the shape of imports.errors.

Briefly list the changes made to the code:

1. Added parseHistorical(csv) in apps/backend/upload-lambda/src/parsers/historical.ts. It finds the header row even when rows sit above it (the Drive export has a link and a blank row first), looks up columns by header name, and checks that:

   • Month # and Day are a real calendar day (Feb 29 included), and the Month name agrees with Month #
   • each day appears once
   • every value has its year(s), and blank or "None" values have none ("None" is stored as null for both)
   • years are four digits, wind direction is one of the 16 compass points, and Estimated? is TRUE or FALSE

   Precip and snow stay strings ("5.30"), matching how pg reads numeric. Source text is not stored, since it repeats the other columns.

2. Added shared types in libs/weather-parsing/src/types.ts: ParseResult<T>, RowError, and HistoricalRecord. HistoricalRecord uses the daily_records column names, so a parsed row plus stationId and importId can go straight into an insert. The daily parser (Dev: Build a CSV parsing utility - Daily hourly-log parser #9) can reuse ParseResult and RowError.

3. Added a small CSV reader in parsers/csv.ts (no new dependency). It handles quoted fields, which year lists like "2014, 1977" need. Dev: Build a CSV parsing utility - Daily hourly-log parser #9 can use it too.

4. Mapped @bho/weather-parsing in tsconfig.base.json. The lambdas list it as a file: dependency, but nothing installs it, so the import didn't resolve before.

✔️ Verification

Unit tests cover the CSV reader and the parser. The parser tests use the synthetic test sheet, exported from Drive as parsers/__fixtures__/historical.csv (366 rows, every field of Jan 1 checked), plus small sheets for each rule above.

yarn install
npx jest apps/backend/upload-lambda   # 19 tests

To run the parser on any exported CSV and print the rows or the errors:

npx ts-node --transpile-only -O '{"module":"commonjs"}' -e "
const { parseHistorical } = require('./apps/backend/upload-lambda/src/parsers/historical');
const r = parseHistorical(require('fs').readFileSync(process.argv[1], 'utf8'));
console.log(r.ok ? r.rows.length + ' rows' : JSON.stringify(r.errors, null, 2));
" path/to/export.csv

No UI changes.

🏕️ (Optional) Future Work / Notes

Open questions for the team (current behavior in parentheses):

• Should a blank value with blank years count as no record, or as an error? (No record, following the schema doc's "empty cell means null".)
• Can a record be a trace (T) for precip or snow? (Rejected.)
• Must an upload cover all 366 days? (Partial files are accepted.)
• Is a blank Estimated? the same as FALSE? (Yes.)

Found along the way:

• nx build upload-lambda fails with TS6059 because the lambda's rootDir: ./src doesn't allow compiling files from libs/. The build was already failing on main (the lib had no source files). The fix is to drop rootDir or bundle the lambdas with esbuild, which fits better with Dev: Empty Upload Lambda #13.
• tsc writes dist/**/*.spec.js even when it errors, and root jest then runs those copies. Adding dist to jest's ignore paths would stop that.
• The lambdas don't set strictNullChecks, so !result.ok doesn't narrow a ParseResult. Use result.ok === false (noted on the type).
• The repo's lint:check matches no files under apps/backend/*/src, so lint currently checks nothing there.