Skip to content

ascot integration - #166

Open
isc-klu wants to merge 9 commits into
intersystems-community:masterfrom
isc-klu:ascot-integration
Open

isc-klu wants to merge 9 commits into
intersystems-community:masterfrom
isc-klu:ascot-integration

Conversation

@isc-klu

@isc-klu isc-klu commented Oct 26, 2026 •

Copy link
Copy Markdown

This PR integrates ascot 0.4.2, the latest release as of 2026-10-29. ascot aims to find errors the IRIS compiler can't, and to give more detail on the errors the compiler already finds.

My test set has 579 programs that IRIS rejects, at compile time or at runtime, and 543 that run without error:

  • The compiler and ascot both report an error in 436 of the 579.
  • ascot also reports an error in the other 143, which IRIS only rejects at runtime.
  • ascot reports no error on any of the 543 that run without error.

A preliminary agent study used 47 code-generation tasks written by an independent author. With ascot, Haiku 4.5 used 33% fewer tool calls. Opus 4.8 saw no change, since it already needs about one call per task on average.

How much ascot helps in iris-agentic-dev's own setting is still unknown. Along with the integration, this PR includes benchmark results on iris-agentic-dev's tests/e2e/tasks/benchmark/corpus task set.

@isc-klu
isc-klu force-pushed the ascot-integration branch 10 times, most recently from 2974b09 to b95841d Compare October 27, 2026 18:00
One tool, one file: `iris_check` loads a .cls/.mac/.int/.inc from the project directory
into an ascot workspace and returns its diagnostics. What the file refers to but does not
define is resolved on the target namespace through ascot's host `Atelier` trait, which
`tools/check.rs` implements over iad's own connection with the two requests iad already
makes (`/action/query`, `/doc/{name}`), so ascot opens no connection of its own. Without
a connection those references come back as hints. Power users are pointed at `ascot mcp`
and `ascot lsp`.

ascot 0.4.0 is the first release with the trait. It needs rmcp's `ServerConfig` (lock
moves 3.1.3 -> 3.5.1, `get_info` follows the rename; a client naming protocol 2026-07-28
over `initialize` is now answered with 2025-11-25, the newest version that has one) and
`if let` match guards, so the toolchain pin and the CI action pins move 1.92.0 -> 1.95.0.
Clippy 1.95 flagged one sort in test_changelog.rs.

Smoke tests at the three layers: a no-IRIS check (unit), the advertised contract and one
stdio call (binary), a wrong `%SYSTEM.Version` member found through the connection (live).
The build script's `rerun-if-changed` named `.git/HEAD` and `.git/refs` relative to the crate
directory, where no `.git` exists. Cargo treats a rerun path that does not exist as always
changed, so the script and its `rustc-env` reran on every invocation and `cargo run -- tool`
recompiled `iris-agentic-dev-core` and the binary each time. The paths now come from
`git rev-parse --absolute-git-dir`.
@isc-klu isc-klu changed the title feat: iris_check runs ascot over iad's IRIS connection ascot integration Oct 28, 2026
@isc-klu

isc-klu commented Oct 29, 2026 •

Copy link
Copy Markdown
Author

I ran iris_check (ascot 0.4.2) on the erroneous and clean version of each of the 42 tasks in tests/e2e/tasks/benchmark/corpus, against IRIS Community 2026.2. No model was involved. The erroneous version is the fixture the agent is given; the clean version is the task's reference solution.

Summary numbers

On the 42 erroneous programs:

  • ascot reports errors on none, and warnings on 13, of which 2 are relevant to the bug. See the discussion after the table about the irrelevant warning.
  • The compiler reports errors on none.

On the 42 clean programs:

  • ascot reports errors on none, and warnings on 10.
  • The compiler reports errors on none.

In short:

  • Recall: 2 of 42. ascot reports a warning relevant to the bug on CORPUS-20 and CORPUS-37.
  • Precision: 2 of 13. ascot flags 13 erroneous programs, and on 2 of them a warning points at the bug.
  • Latency per file: median 876 ms cold (a fresh process per file) and 406 ms warm (one running server), from a debug build of iad. Both include the IRIS lookups ascot makes for the file.

Per-task results

Every diagnostic ascot reported is a warning. Line numbers are 1-based within each class. The other 28 tasks got no diagnostics in either version.

Task Erroneous version Clean version Relevant to the bug
CORPUS-03 L7, L8, L9: This call returns %Status, which Do discards -- consider Set to capture it, or checking it directly Same No
CORPUS-05 L7, L8: This call returns %Status, which Do discards -- consider Set to capture it, or checking it directly
L9: Too few arguments: 'sc' is required. Only arguments with default values and Output arguments are considered optional.
L9: The parameter 'len' is declared as a by-ref parameter but got a normal argument
Same, except the 'sc' warning is at L11 and the 'len' warning is gone No
CORPUS-09 L9: Too few arguments: 'deletecount' is required. Only arguments with default values and Output arguments are considered optional. Same No
CORPUS-10 None L8: Variable 'pInsert' is assigned but never read No
CORPUS-17 L9: Too few arguments: 'deletecount' is required. Only arguments with default values and Output arguments are considered optional.
L21: Too few arguments: 'pSelectMode' is required. Only arguments with default values and Output arguments are considered optional.
Same No
CORPUS-18 L8: Too few arguments: 'deletecount' is required. Only arguments with default values and Output arguments are considered optional. Same No
CORPUS-20 L14: Variable 'tSC' is assigned but never read None Yes
CORPUS-21 L14: This call returns %Status, which Do discards -- consider Set to capture it, or checking it directly Same No
CORPUS-23 L4: Variable 'pN' is assigned but never read None No
CORPUS-26 L9, L28: Too few arguments: 'deletecount' is required. Only arguments with default values and Output arguments are considered optional. Same, the second one at L29 No
CORPUS-28 L8: Too few arguments: 'deletecount' is required. Only arguments with default values and Output arguments are considered optional. Same No
CORPUS-31 L4: Variable 'pClass' is assigned but never read None No
CORPUS-37 L6: ObjectScript has no operator precedence and evaluates strictly left-to-right; this may not compute what you expect None Yes
CORPUS-40 L6: Variable 'pInsert' is assigned but never read Same No

Warnings not relevant to the bug

They fall into three groups.

"Do discards %Status" (CORPUS-03, 05, 21). These are correct. SetAt and Rewind return a status that the code throws away. That is a bad pattern worth flagging, even though it isn't the bug here. CORPUS-20's bug is the same mistake: a returned status that is never checked.

"Too few arguments" (CORPUS-05, 09, 17, 18, 26, 28). These are false positives caused by imprecise library method signatures:

  • An argument is used as Output but declared as ByRef:
    • deletecount of %Library.Persistent.%DeleteExtent, in CORPUS-09, 17, 18, 26 and 28
    • sc of %Stream.TmpCharacter.Read, in CORPUS-05
  • An optional argument effectively defaults to "":
    • pSelectMode of %SQL.Statement.%OnNew, reached through %New() in CORPUS-17, which reads it only through $get(pSelectMode)

"Assigned but never read" (CORPUS-10, 23, 31, 40). In the stubs in CORPUS-23 and 31, the parameter really is unused. In the %OnBeforeSave and %OnDelete callbacks in CORPUS-10 and 40, the callback signature requires the parameter, so the warning is more debatable. Here ascot trades some precision for recall.

CORPUS-43 to CORPUS-56 each plant a bug IRIS compiles clean and trips on at runtime: a variable
set on one path, an accumulator set inside a loop that may not run, `$DATA` answering 10, `Quit`
with a value inside `For`, a `%Status` method quitting with no value, one argument too many, a
utility declared `Method` and called through `##class`, a property typo through an object
variable, a misspelled `%SYSTEM.Version` method, JSON text returned for a `%DynamicObject`, a
collection property subscripted, `$CASE` with no default, an `Output` parameter never assigned,
and a `$ORDER` loop with its empty check inverted. The kinds come from the runtime-proven tests
of objectscript-typecheck-corpus; none involves a routine.

Every fixture compiles on IRIS 2026.2, every check fails on the fixture and passes on the
reference, and `iris_check` reports the planted line on all fourteen fixtures and nothing on any
reference. All fourteen are holdout from the day they were written.
The one control among the fifteen new tasks: `$$$ISERROR` where `$$$ISERR` was meant. The
compiler rejects it with MPP5610 and two cascade errors at routine offsets; ascot reports one
error at the class line with `ISERR` as the first suggestion. Holdout from the day it was written.
@isc-klu

isc-klu commented Oct 29, 2026 •

Copy link
Copy Markdown
Author

I added 15 tasks to the corpus, CORPUS-43 to CORPUS-57 (commits b6f4f25 and 0ae3ca7): bugs the IRIS compiler accepts and the program trips on at runtime, drawn from the runtime-proven tests of my type-checking corpus, plus the misspelled macro you asked for. All 15 are holdout from the day they were written. I then ran iris_check (ascot 0.4.2) on the erroneous and clean version of each, against IRIS Community 2026.2, no model involved.

On the 15 erroneous programs:

  • ascot reports errors on 10 and warnings on the other 5; in all 15 the diagnostic is on the planted line.
  • The compiler reports an error on one, CORPUS-57 (misspelled macro); see below the table.

On the 15 clean programs:

  • ascot reports nothing.
  • The compiler reports errors on none.

In short:

  • Recall: 15 of 15.
  • Precision: 15 of 15.
  • Latency per file: median 864 ms cold (a fresh process per file) and 416 ms warm (one running server), from a debug build of iad. Both include the IRIS lookups ascot makes for the file.

These 15 are tasks I wrote around defects ascot detects, so they show what the tool covers.

Per-task results

Task Bug Compiler, erroneous version ascot, erroneous version ascot, clean version
CORPUS-43 variable set in one If branch, returned on both None L9: This '%String' value may be undefined None
CORPUS-44 accumulator set only inside a For that may run zero times None L9: This '%DataType' value may be undefined None
CORPUS-45 $DATA answering 10 treated as a value None L14: This 'Any' value may be undefined None
CORPUS-46 Quit with a value inside For None L9: Quit with %String inside control structure None
CORPUS-47 %Status method quits with no value None L9: Expected return value of type '%Status' but got nothing None
CORPUS-48 one argument too many to an own method None L6: Too many arguments None
CORPUS-49 utility declared Method, called through ##class None L20: Cannot reference instance-only member 'ToFahrenheit' in a class method None
CORPUS-50 wrong property name set through an object variable None L18: Cannot find member 'Turtle' in Bench.Card; did you mean: Title, Owner, %%OID, %GetParameter? None
CORPUS-51 misspelled %SYSTEM.Version method None L6: Cannot find member 'GetMajorVersion' in Class '%SYSTEM.Version'; did you mean: GetMajor, GetVersion, GetCompilerMajorVersion, GetCompilerVersion, GetMinor? None
CORPUS-52 JSON text returned where %DynamicObject is declared None L6: %String is incompatible with %DynamicObject None
CORPUS-53 collection property subscripted None L13: Property 'Items' is not MultiDimensional, so it takes no subscripts None
CORPUS-54 $CASE with no default None L6: This '%String' value may be undefined None
CORPUS-55 Output parameter never assigned None L4: Output parameter 'pCount' is never assigned; its caller will get an undefined value
L6: Variable 'tCount' is assigned but never read
None
CORPUS-56 $ORDER loop with its empty check inverted None L23: This value is undefined: tQty None
CORPUS-57 misspelled macro 3 errors, see below L7: The macro 'ISERROR' is undefined; did you mean: ISERR, ... None

The erroneous program in CORPUS-57 is the only one the compiler rejects:

ERROR:  Bench.Guard.1(4) : MPP5610 : Referenced macro not defined: 'ISERROR'
ERROR: Bench.Guard.cls(Save+3) #1002: Invalid character in tag : '$$$ISERROR(tSC)' : Offset:10
ERROR: Bench.Guard.cls(Save+6) #1026: Invalid command : '1 }' : Offset:11

Only the first line is the bug and no fix is suggested. ascot reports one error at the exact class line and suggests the right macro, ISERR.

`name` replaces `path`: the document with its extension, read from the target namespace through
the same `/doc` request the bridge already makes for lookups, or taken from `source` when the
caller holds text it has not saved yet. An MCP client no longer needs a file inside the project
directory, which is where the earlier shape failed: the classes a task is about live on the
server, and the code an agent wants checked is in its hands before it is written anywhere.

Without a connection the response now carries a `note` saying that external references were not
checked; without a connection and without `source` the call is refused with IRIS_UNREACHABLE
like the other IRIS tools, and a `name` the server lacks answers DOCUMENT_NOT_FOUND.
The tool-name gate reads an imported `iris_unreachable` in a `use` list as a tool reference;
every other caller writes it as a call, which the gate exempts.
@isc-klu

isc-klu commented Oct 30, 2026 •

Copy link
Copy Markdown
Author

The last two commits improves how ascot interact with humans and AI agents: it now takes source code rather than a path to a file system. The new way allows users to check ObjectScript sources without touching the file system. The second commit is a simple change to make a CI happy.

They don't change the diagnostics that ascot provides.

@isc-klu
isc-klu marked this pull request as ready for review October 30, 2026 13:28
@isc-klu
isc-klu requested a review from isc-tdyar as a code owner October 30, 2026 13:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant