Fix lexer column tracking

[ydah]

Lexer#column used StringScanner#pos directly, causing columns after multibyte characters to be reported as byte offsets.

Whitespace scanning also consumed newlines together with indentation from the following line. This moved the line-start position past the indentation and incorrectly reported column zero.

Calculate columns as character offsets and process newlines separately from other whitespace. Add regression coverage for indented lines with trailing whitespace and multibyte input.

[Copilot]

🟢 Approval recommended

The changes directly address the described column-tracking defects and include targeted regression tests for both indentation and multibyte scenarios.

Pull request overview

This PR fixes incorrect lexer column reporting by computing columns as character offsets (not byte offsets) and by avoiding newline consumption as part of generic whitespace scanning, which previously caused indentation columns to be misreported after line breaks.

Changes:

• Compute Lexer#column as a character count from the current line start to the scanner position.
• Scan \n separately from other whitespace so indentation after a newline doesn’t reset the line-start pointer incorrectly.
• Add regression specs for indented trailing-whitespace lines and multibyte input.

File summaries

File	Description
spec/lrama/lexer_spec.rb	Adds regression coverage for indentation-after-newline column reporting and multibyte character column offsets.
lib/lrama/lexer.rb	Updates column calculation and refines whitespace/newline scanning to preserve correct line-start tracking.

Review details

• Files reviewed: 2/2 changed files
• Comments generated: 0
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.