diff --git a/demo.cls b/demo.cls new file mode 100644 index 0000000..55bd99b --- /dev/null +++ b/demo.cls @@ -0,0 +1,10 @@ +Class Demo.X +{ + +ClassMethod M() As %String +{ + set x = ##safeexpression("""a""") + quit x +} + +} diff --git a/objectscript-lsp/README.md b/objectscript-lsp/README.md index fa9e634..402e4a6 100644 --- a/objectscript-lsp/README.md +++ b/objectscript-lsp/README.md @@ -6,13 +6,15 @@ We built this language server to provide editor-independent ObjectScript semanti ## Current Features -- Workspace indexing for `.cls`, `.inc`, `.rtn`, `.mac`, and `.int` -- Cold workspace construction uses a bulk declaration/linking pipeline; live documents are +- Workspace indexing for `.cls`, `.inc`, `.rtn`, `.mac`, `.int`, and `.xml` +- Parallel cold workspace indexing (see [Parallel Indexing](#parallel-indexing)); live documents are rebuilt incrementally. - Multi-workspace support through LSP workspace folders, with deepest-parent routing per document - Go-to-definition for ObjectScript variables, orefs, methods, properties, classes, parameters with ProcedureBlock-aware private/public resolution - Go-to-implementation for inherited and overridden methods and classes -- Syntax diagnostics for tracked ObjectScript documents +- Semantic model for every class member type: methods, properties, parameters, relationships, + foreign keys, queries, indices, triggers, XData, projections, and storage +- Syntax diagnostics for tracked ObjectScript documents, plus duplicate-class diagnostics - Mixed-language diagnostics for ObjectScript captured from XML `Implementation` blocks - Refactor code actions for: - Legacy dotted `DO` rewrites @@ -21,7 +23,9 @@ We built this language server to provide editor-independent ObjectScript semanti - document-scoped and workspace-scoped edits - Inheritance modeling and override index build - Dependency modeling and dependencyGraph build (shows all paths to a given method) - +- Granular, ID-independent class and member comparison between two workspaces + (`objectscript_core::workspace_diff`) +- An MCP bridge binary (`objectscript-mcp`) that exposes the server to MCP clients such as Claude Code ## Architecture Summary @@ -32,18 +36,48 @@ We built this language server to provide editor-independent ObjectScript semanti - Public symbols live in `GlobalSemanticModel` - Private symbols are tracked through `LocalSemanticModel` and `ScopeTree` - XML documents are tracked for diagnostics, but they do not enter the class/routine semantic rebuild pipeline -- The bulk indexing phases, ownership model, SYS classification, and performance changes are - documented in [documentation/bulk-workspace-indexing.md](documentation/bulk-workspace-indexing.md). -- Remaining measured bottlenecks and optimization priorities are documented in - [documentation/indexing-optimization-roadmap.md](documentation/indexing-optimization-roadmap.md). -- Granular, ID-independent class and member comparison is documented in - [documentation/class-comparison.md](documentation/class-comparison.md). +- See [crates/objectscript-core/README.md](crates/objectscript-core/README.md) for the core + library's module layout and public API. + +### Parallel Indexing + +Cold workspace indexing (run for each workspace folder on `initialized`) runs on Tokio's +blocking thread pool so the LSP event loop stays responsive, and uses `rayon` for data +parallelism. The rayon global pool is used, so the worker count defaults to the number of +logical CPUs (override with `RAYON_NUM_THREADS`). + +1. **Discover** (serial): `get_paths` walks the workspace root and classifies supported files by + extension. +2. **Read + parse** (parallel, per file): each rayon worker reads a file and parses it with its own + `IndexParsers` instance (created once per worker via `map_init`), producing a + `BulkIndexDocument`. Files that fail to read or parse are logged and skipped. +3. **Register** (serial): under the project write lock, `ProjectData::begin_bulk_index()` + registers each document, assigns `ClassId`s, and records duplicate-class diagnostics. +4. **Build classes** (parallel, per document): `BulkWorkspaceIndex::finalize` builds each class's + members and method bodies (variables, unresolved calls, unresolved oref calls) in parallel. + Workers own their document and never observe `ProjectData`. Inside each class, + `Class::build_class` also builds individual member definitions in parallel. +5. **Commit and link** (serial): results are committed into the shared semantic model, then the + complete inheritance graph is built, followed by the override index (parent-before-child), + ordinary method call edges, and finally oref call resolution. The order matters: oref + resolution walks incoming call edges, so it must run after every ordinary edge exists. + +All shared semantic/index mutation stays serial, so the result does not depend on scheduling. +Progress and per-phase timings are logged to stderr with an `[index]` prefix. + +Workspace comparison is also parallel: `compare_classes_parallel` and +`compare_workspaces_parallel` diff classes across two `ProjectData` snapshots with `par_iter`. ## Workspace Layout - `objectscript-lsp`: LSP transport layer in [src/main.rs](src/main.rs), [src/lsp.rs](src/lsp.rs), and [src/server.rs](src/server.rs) -- `crates/objectscript-core`: parsing, semantic model, workspace state, refactors, and dependency tracking + - [src/bin/objectscript-mcp.rs](src/bin/objectscript-mcp.rs): MCP bridge that spawns + `objectscript-lsp` and exposes it as MCP tools +- `crates/objectscript-core`: parsing, semantic model, workspace state, refactors, dependency + tracking, and class comparison ([README](crates/objectscript-core/README.md)) +- `claude-code-objectscript-lsp/`: Claude Code LSP plugin ([README](claude-code-objectscript-lsp/README.md)) - `objectscript-tests/`: fixture corpus for inheritance, dependencies, navigation, and related regressions +- `documentation/`: feature docs, configuration, benchmarks, and design notes ## LSP Surface @@ -51,34 +85,56 @@ We built this language server to provide editor-independent ObjectScript semanti - `textDocument/definition` - `textDocument/implementation` - `textDocument/diagnostic` -- Code actions and execute commands for refactor rewrites + - `workspace/diagnostic` + - `textDocument/codeAction` (`refactor.rewrite`) + - `workspace/executeCommand` +- Notifications: `didOpen`, `didChange` (incremental sync), `didChangeConfiguration`, + and `didChangeWatchedFiles` +- Execute commands: + - `objectscript.refactorDocument` + - `objectscript.refactorWorkspace` + - `objectscript.refactorWorkspaceDottedDo` (legacy) +- Experimental capability: `objectscriptDependenciesProvider` + +## MCP Tools + +`objectscript-mcp` exposes: `objectscript_initialize_workspace`, `objectscript_diagnostics`, +`objectscript_workspace_diagnostics`, `objectscript_goto_definition`, +`objectscript_code_actions`, `objectscript_execute_command`, and `objectscript_lsp_status`. ## Configuration Editor-specific configuration examples for Zed, Neovim, and VS Code are documented in [documentation/configuration.md](documentation/configuration.md). - ## Build and Test ```bash -cargo build -cargo test +cargo build # builds objectscript-lsp and objectscript-mcp +cargo install --path . # installs both binaries onto PATH +cargo test --workspace # LSP tests + objectscript-core tests ``` +Benchmarks live in `crates/objectscript-core`; see its [README](crates/objectscript-core/README.md#benchmarks) +and [documentation/benchmarks/update-document.md](documentation/benchmarks/update-document.md). + ### Go-To Definition Go-to definition works for classes, class methods, orefs, procedures, subroutines, instance methods, public local variables, private local variables, and global variables. For variables, the way the definition(s) is determined depends on the case: -**CASE 1: Variable is defined the current scope. ** + +**CASE 1: Variable is defined the current scope.** In this case, the variable definition in the given scope is returned. -**CASE 2: Variable is NOT defined the current scope. ** +**CASE 2: Variable is NOT defined the current scope.** + **CASE 2A: Variable is Private** This means that the variable is undefined. No definition is returned. + **CASE 2B: Variable is Public** In this case, the `DependencyGraph` is used to determine all possible paths to the current scope. For each node (scope) on the path, we check if the wanted variable is defined in that scope, and if so we track that location. All possible locations are returned. +More detail is in [documentation/features](documentation/features). ## Grammar Baseline @@ -93,6 +149,7 @@ In this case, the `DependencyGraph` is used to determine all possible paths to t - Semantic diagnostics (undefined variables, unresolved symbols) - Broader mixed-language support beyond XML `Implementation` blocks - More lifecycle and incremental edit coverage -- Expanded semantic support for properties, parameters, queries, triggers, and storage + (see [documentation/incremental-workspace-index-maintenance-spec.md](documentation/incremental-workspace-index-maintenance-spec.md)) +- Go-to-definition / implementation for the newer member types (queries, triggers, indices, storage, etc.) - Find references and symbol-oriented LSP features - Formatting support beyond the current refactor rewrites diff --git a/objectscript-lsp/crates/objectscript-core/README.md b/objectscript-lsp/crates/objectscript-core/README.md new file mode 100644 index 0000000..9bff356 --- /dev/null +++ b/objectscript-lsp/crates/objectscript-core/README.md @@ -0,0 +1,133 @@ +# objectscript-core + +The in-memory representation of an InterSystems ObjectScript workspace. `objectscript-core` +parses `.cls`, `.mac`, `.inc`, `.rtn`, `.int`, and `.xml` sources with `tree-sitter`, builds a +semantic model of classes, routines, members, and variables, and keeps the indexes needed for +navigation, diagnostics, refactoring, and dependency analysis. + +It has no LSP transport of its own. [`objectscript-lsp`](../../README.md) wraps it as a language +server, but it can also be used directly (for example, to compare two workspaces offline). + +## Concepts + +- **`ProjectState`** (`workspace.rs`): concurrency wrapper for one workspace. Holds the root path, + a lock-protected `ProjectData`, and shared Tree-sitter parsers. Entry point for opening and + updating documents (`handle_document_opened`, `update_document`) and refactors. +- **`ProjectData`** (`workspace.rs`): the workspace "database": documents, configuration, + semantic models, symbol indexes, the dependency graph, the override index, and diagnostics. + Also answers definition/implementation queries (`get_method_definition`, + `get_variable_definition`, `get_class_implementations`, and similar). +- **`GlobalSemanticModel`** (`global_semantic.rs`): public, workspace-wide symbols (classes, + methods, properties, parameters, and every other class member type), keyed by stable refs + such as `ClassId`, `MethodRef`, and `PropertyRef`. +- **`LocalSemanticModel`** + **`ScopeTree`** (`local_semantic.rs`, `scope_tree.rs`): private, + per-class symbols and ProcedureBlock-aware lexical scopes. +- **`DependencyGraph`** (`dependency_tracker.rs`): caller → callee edges between methods, used to + find every path into a scope (e.g. resolving public variables). +- **`OverrideIndex`** (`override_index.rs`): inherited/overridden members per class, built + parent-before-child. +- **Class comparison** (`workspace_diff.rs`): ID-independent snapshots and diffs of classes and + their members across two `ProjectData` instances. + +## Module Layout + +| Module | Responsibility | +| --- | --- | +| `workspace` | `ProjectState`, `ProjectData`, bulk indexing, incremental updates, queries | +| `workspace_diff` | `snapshot_class`, `compare_class`, `compare_classes_parallel`, `compare_workspaces_parallel` | +| `class` | Building a `Class` and its members from a parse tree | +| `method`, `variable` | Method bodies, variable definitions, unresolved calls | +| `property`, `parameter`, `relationship`, `foreignkey`, `query`, `index`, `trigger`, `xdata`, `projection`, `storage` | Per-member-type extraction | +| `parse_structures` | Shared data types (`ClassId`, `FileType`, `Method`, `Property`, ...) | +| `global_semantic`, `local_semantic`, `scope_tree`, `scope_structures` | Semantic models and scopes | +| `dependency_tracker`, `override_index` | Call graph, dependents, and override tables | +| `refactor` | Legacy dotted `DO`, `IF/Else`, and `FOR` rewrites | +| `document` | Parsed document state (content, tree, version, file type) | +| `config` | Workspace configuration | +| `common` | Tree-sitter helpers, cached queries, position/byte conversion | + +## Indexing Model + +There are two build paths: + +1. **Bulk (cold) indexing** — `ProjectData::begin_bulk_index()` returns a `BulkWorkspaceIndex`. + Callers `register` every pre-parsed `BulkIndexDocument` (assigning `ClassId`s and detecting + duplicate documents/classes), then call `finalize`: + - **Parallel:** each class/routine document is built independently with `rayon` + (`into_par_iter`): class members, method bodies, variables, and unresolved calls/orefs. + Workers own their document and never touch `ProjectData`. `Class::build_class` additionally + builds individual member definitions in parallel. + - **Serial:** results are committed, then the full inheritance graph is built, then the + override index, then ordinary call edges, then oref call resolution. Oref resolution walks + incoming call edges, so it must follow the ordinary edges for the result to be + deterministic. + + Reading and parsing the files themselves is the caller's job; `objectscript-lsp` does that in + parallel too, with one parser set per rayon worker. + +2. **Incremental updates** — `ProjectState::update_document` applies edits to a single document + and rebuilds only the affected classes, members, inheritance, override entries, and call + edges. See + [`documentation/incremental-workspace-index-maintenance-spec.md`](../../documentation/incremental-workspace-index-maintenance-spec.md). + +`ProjectData::mark_current_classes_as_sys()` can be called after indexing an IRIS SYS source +root and before indexing a customer workspace; `get_sys_dependencies()` then reports the +customer code's direct and transitive SYS dependencies. + +## Usage + +```rust +use objectscript_core::parse_structures::FileType; +use objectscript_core::workspace::ProjectState; +use objectscript_core::workspace_diff::compare_class; +use tower_lsp::lsp_types::Url; + +let baseline = ProjectState::new(); +baseline.handle_document_opened( + Url::parse("file:///workspace/Demo.T.cls").unwrap(), + source.to_string(), + FileType::Cls, + 1, +); + +let target = ProjectState::new(); +// ... open documents in `target` ... + +let comparison = compare_class(&baseline.data.read(), &target.data.read(), "Demo.T"); +``` + +See [`tests/class_comparison.rs`](tests/class_comparison.rs) for more complete examples. + +## Testing + +```bash +cargo test -p objectscript-core +``` + +`tests/class_comparison.rs` covers class and member comparison. Tests for known gaps are +`#[ignore]`d with a reason; list them with `cargo test -p objectscript-core -- --ignored`. + +## Benchmarks + +```bash +# Incremental update_document benchmark (criterion) +cargo bench -p objectscript-core --features update-bench --bench update_document + +# Incremental update benchmark as a standalone example +cargo run -p objectscript-core --release --features update-bench --example update_bench + +# Cold workspace construction; pass a workspace path, or omit it to use a generated workspace +cargo run -p objectscript-core --release --example cold_workspace_bench -- /path/to/workspace +``` + +The `update-bench` feature enables instrumentation (e.g. `full_update_document_call_count`) that +is compiled out of normal builds. Inputs and options for the criterion bench are documented in +[`documentation/benchmarks/update-document.md`](../../documentation/benchmarks/update-document.md). + +## Dependencies + +- `tree-sitter 0.26.6` with `tree-sitter-objectscript`, `tree-sitter-objectscript-routine`, + `tree-sitter-objectscript-playground` (`1.10.1`), and `tree-sitter-xml` (`0.7.0`) +- `rayon` for parallel indexing and comparison +- `petgraph` for the dependency graph +- `tower-lsp` for shared LSP types (`Url`, `Range`, `Diagnostic`)