Skip to content

build.mcpp: four channels a rule package had no way to reach the engine through - #587

Merged
Sunrisepeak merged 15 commits into
mainfrom
feat/vulkan-portability-enumeration
Sep 7, 2026
Merged

build.mcpp: four channels a rule package had no way to reach the engine through#587
Sunrisepeak merged 15 commits into
mainfrom
feat/vulkan-portability-enumeration

Conversation

@Sunrisepeak

@Sunrisepeak Sunrisepeak commented Sep 7, 2026

Copy link
Copy Markdown
Member

Four channels a rule package had no way to reach the engine through, the layer
above build.mcpp those channels make possible, and the graphics example's half
of raising CI from "builds" to "runs".

Each of the four is small. What they have in common is the shape of the gap: one
side already knew something, and nothing carried it to the side that had to
decide.

mcpp::action gains depfile

An action's inputs are fixed when build.mcpp runs, before its command has
executed. A compiler that learns its include graph by parsing the source
therefore has no channel to report it: editing a file the command merely READ
rebuilds nothing, and mcpp build stays green over a stale artifact.

a.depfile = dep.c_str();
a.arg("--depfile").arg(dep.c_str());

mcpp emits depfile = and deps = gcc for that edge. Every device compiler
this matters for already emits one -- glslangValidator --depfile,
glslc -MD -MF, slangc -depfile, nvcc and clang -MD -MF -- so the
missing half was always the receiving end.

The depfile must not also be a declared output: deps = gcc makes ninja consume
and delete it, so an edge promising it would be permanently dirty. Documented at
the field, at the decoder and at the emission site.

A rule package declares the extensions it compiles, and the module that does it

A constrained glob's accel key does not make a file a device source;
kDeviceExtensions does. Until this change a rule package could not introduce
a device language on its own
: measured against 2026.9.6.6 with a working rule
and a working xim:slang, the payload installed, the rule compiled, and the file
still fell through to the ordinary source scan --

warning: [build] accel names vulkan but no constrained glob matched a `.slang`
error: 'scale.slang' is listed in [build] sources, and mcpp has no role for
       the extension '.slang'.

The first version of this branch fixed that by adding .slang to the engine's
table. That inverts the dependency: a general build system would hold a list
of an external plugin's languages, and a third-party rule -- someone else's CUDA
rule, someone else's shader rule -- would still wait on an engine release. The
shape that shipped puts both facts on the rule package's own feature:

[features.rules-slang]
sources           = ["rules/slang.cppm"]
device_extensions = [".slang"]
rule_module       = "mcpp.rules.slang"

The engine holds no package name, no feature spelling and no module name. The
two keys are required together: a feature that claims an extension without
naming who compiles it is refused, rather than letting the file fall out
somewhere later.

The criterion is direct. .slang was removed from kDeviceExtensions, and
mcpp-plugins' tests/slang-consumer builds and runs unchanged. A new device
language now costs no engine release.

Collected after feature activation and before the extension table that narrows
constrained globs, and that position is load-bearing: placed after it, the
declared extensions classify nothing, the device source list comes out empty,
the rule is handed no files, and the failure surfaces three edges away as
failed to read compiled module. Measured.

A project that declares its rules need not write build.mcpp

Once the rule modules are declared, the program's content is determined: import
them, call compile() on each, return non-zero if any refused. mcpp writes it
into the build directory.

A project that takes it over copies the file to its root and edits it, and
synthesis stops -- the declaration is a layer ABOVE build.mcpp, not a second
mechanism beside it. A package that already has one keeps it, because two
programs submitting the same actions is a submission nobody asked for.

A synthesis that cannot write its program now refuses. Returning "nothing to do"
left the device sources handed to no one and the build otherwise successful,
with the first symptom an unresolved name in a consumer three edges away.

[language] modules is reported as MCPP_LANGUAGE_MODULES

A rule that GENERATES a consumer-facing declaration has to choose between a
module interface and a header, and the project has already said which it builds.
Deriving it any other way would be a second spelling of one decision.

An engine that does not report it leaves the variable absent, which a rule reads
as "header" -- the behaviour every consumer had before -- so the upgrade needs no
project to declare anything.

A build program is told which package it is building

MCPP_PKG_NAME and MCPP_PKG_NAMESPACE, read through mcpp::package_name()
and mcpp::package_namespace().

Every name a rule generates is derived from the package's name -- the module a
consumer imports, the namespace the accessors sit in, the symbols in a generated
header -- and nothing answered that question. The closest available answer was
the leaf of MCPP_MANIFEST_DIR, a DIRECTORY name, and the two differ whenever a
package sits under a generic folder. This repository is such a case:
examples/09-heterogeneous/vulkan/app/ declares name = "vulkan-saxpy" and
mcpp.rules.spirv generated app.shaders, which every <something>/app/ in a
workspace would also claim -- and which changes when a folder is renamed.

The e2e distinguishes the two derivations by construction: its fixture's
directory is app and its package is vulkan-saxpy. A fixture whose name
matched its folder passes against either implementation, which is what every
existing one did.

Portability drivers are invisible by default

The Vulkan loader does not hand portability drivers to
vkEnumeratePhysicalDevices unless the instance both enables
VK_KHR_portability_enumeration and sets the matching create-info flag. The
offscreen example now does, so the macOS job runs the render on MoltenVK rather
than only building it, and the example itself declares the driver under
cfg(macos) -- a checkout gets a machine that can run it, not only the runner
that had an extra command.

Asked as a capability rather than as #ifdef __APPLE__: the property is "this
loader is showing me a portability driver", which a Linux machine running a
translation layer also has and a macOS build against a native driver does not.
Linux/lavapipe readings are unchanged, centre pixel (124, 70, 62, 255), device
llvmpipe.

Windows stays at "builds", and the recorded reason was wrong

The run step was written, pushed and measured. xim:mesa-lavapipe installed, the
ICD manifest was found in the store, and the program printed render unavailable -- src/main.cpp reporting that the render function returned
nothing.

An earlier revision of this branch recorded a missing vulkan-1.dll as the
cause. The log refutes it: the Vulkan leg imports vkCreateInstance from that
DLL through the import library, so a process that could not find it would fail
during image load and print nothing at all. It printed. mcpp-index's own
vulkan-tests member calls vkEnumerateInstanceVersion on the windows shards
and passes, which says the same thing from the other side.

A separate measurement removes a second suspect: the Khronos loader in
compat:vulkan cross-builds into a working vulkan-1.dll from the source the
index already carries -- 265 exports matching upstream's vulkan-1.def name for
name, DllMain present, importing only ADVAPI32, CFGMGR32, KERNEL32 and msvcrt.
The descriptor's note argues that a Windows loader must be a DLL, not that it
cannot be built. So a hermetic Windows loader is available whenever it is
wanted, and it is not what this step is waiting on.

What is open is why the lavapipe payload's ICD enumerates no device under a
process mcpp launched. Both notes that carried the wrong cause are corrected in
place rather than deleted, so a reader sees what was believed and what refuted
it.

What is not here

examples/09-heterogeneous moving to the generated boundary and the module
payload surface, which consumes mcpp:plugins 0.3.0. That version needs a
released mcpp to build against, and mcpp's own CI resolves the plugin collection
from the index, so those changes cannot be green until this ships. They are
written, verified on this host against the plugins worktree -- cuda and sycl
build and run on an RTX 4080 with the generated boundary on the island's own
command line, vulkan reaches its SPIR-V as a module -- and parked on
feat/examples-module-surface.

四期(软件设备把 CI 上限抬到运行)的程序侧那一半。

接设备之前先测了上游产物,而第一行的读数就是本条的理由:Khronos 官方的
`MoltenVK-macos.tar` v1.4.2 里,ICD 带着 **`"is_portability_driver": true`**。
**loader 默认不把可移植性驱动交给 `vkEnumeratePhysicalDevices`** —— 一个照着原生
驱动写的 Vulkan 程序因此在 macOS 上一个设备都找不到,并把它报成「这台机器没有
GPU」。诊断是错的,而它看起来完全合理。

规范要的是成对的两半:实例要**启用** `VK_KHR_portability_enumeration` **并且**置位
`VK_INSTANCE_CREATE_ENUMERATE_PORTABILITY_BIT_KHR`;随后凡是声明了
`VK_KHR_portability_subset` 的设备,必须在 `vkCreateDevice` 时启用它,否则调用失败。

按**能力**问,不按 `#ifdef __APPLE__`:性质是「我面前这个 loader 在展示可移植性
驱动」,而 `#ifdef` 两个方向都会错 —— Linux 上跑翻译层的机器也有它,macOS 上对着
原生驱动构建的程序并不需要它。

实测:Linux/lavapipe 上读数不变,中心像素仍是 `(124, 70, 62, 255)`。

顺带记下 Windows 那一侧的测量:`mesa3d-26.2.0-release-msvc` 的 `vulkan_lvp.dll`
**只导入系统 DLL**(无 MSVCP140/VCRUNTIME140),静态 CRT,机器上不需要装任何东西。

载荷侧(把 MoltenVK 与 Windows 的 lavapipe 收进索引)未做:那要往发布组织推新的二进制
资产,而 GitCode 的资产不可替换不可删除。
…ine through

Each of these is small, and they share a shape: a rule package already knew
something and nothing carried it to a decision the engine makes.

`mcpp::action` gains `depfile`. An action's inputs are fixed when build.mcpp
runs, before its command has executed, so a compiler that learns its include
graph by parsing the source has no channel to report it: editing a file the
command merely READ rebuilds nothing and `mcpp build` stays green over a stale
artifact. With a depfile declared, ninja reads the file and folds what it names
into the edge. Every device compiler this matters for already emits one --
`glslangValidator --depfile`, `glslc -MD -MF`, `slangc -depfile`, `nvcc` and
`clang` `-MD -MF` -- so the missing half was always the receiving end.

The depfile must not also be declared as an output: `deps = gcc` makes ninja
consume and delete it, so an edge that promised it would be permanently dirty.
The comment at the emission site says so.

`.slang` joins the device-source extension table. A constrained glob's `accel`
key does not make a file a device source; `kDeviceExtensions` does. Until it
listed `.slang`, a rule package could not introduce a device language on its
own: the payload installed, the rule compiled, and the file still fell through
to the ordinary source scan and was refused with "mcpp has no role for the
extension". Measured against 2026.9.6.6 with a working rule and a working
`xim:slang`.

Slang is a language rather than a second driver for GLSL -- its own module
system, generics, and targets beyond SPIR-V -- so it has an extension here and
a rule of its own outside.

`[language] modules` is reported as `MCPP_LANGUAGE_MODULES`. A rule that
GENERATES a consumer-facing declaration has to choose between a module
interface and a header, and the project has already stated which it builds;
deriving it any other way would be a second spelling of one decision. An engine
that does not report it leaves the variable absent, which a rule reads as
"header" -- the behaviour every consumer had before -- so the upgrade needs no
project to declare anything.

Tests. `tests/e2e/631` builds a project whose action writes a depfile naming a
file it never declares as an input, and asserts the action reruns when that
file is touched. It carries its own control: a no-op rebuild must not rerun the
action, which rules out a backend that reruns unconditionally. A separate
reverse control was run by hand against the same binary with `a.depfile`
removed and the flag still passed: the action did not rerun, so the assertion
measures the field rather than some other trigger.
It read `grep -q "GENERATE" b3.log && FAIL`, on the assumption that ninja's
description line for the edge reaches that log. It does not. `mcpp build`
passes `--quiet` to ninja whenever it is not `--verbose` and surfaces ninja's
captured stdout only on failure, so a from-scratch build -- which
unquestionably runs the action -- prints no such line either. The grep never
matched, and the check could not fail whether the action reran or not.

Found while designing the depfile test, whose first draft copied this
convention and would have inherited the same emptiness.

The generated file's modification time measures the thing being asserted
directly: the action rewrites it whenever it runs. Verified in both directions
against the same binary -- unchanged after touching an unrelated source, and
the assertion fails naming both timestamps when the touched file is a declared
input of the action instead.
@Sunrisepeak Sunrisepeak changed the title examples/10-graphics: 可移植性驱动默认是看不见的 build.mcpp: three channels a rule package had no way to reach the engine through Sep 7, 2026
macOS and Windows built it and did not run it, which is the shape the
cross-platform work exists to remove: the half of a lane written for a host is
the half that host never exercises. A build asserts that the shader compiler
this platform publishes works and that the Vulkan half links. It cannot assert
that the loader hands the program a device, and on macOS that is the
interesting half.

The example now declares a device for those two platforms as well --
`xim:moltenvk` under `cfg(macos)`, `xim:mesa-lavapipe` under `cfg(windows)` --
rather than leaving CI to install one. The omission was invisible while the
example was never run, because a program that does not run never asks the
loader for a device, and declaring it here is what makes the example complete
for anyone who checks it out rather than only for the runner that had an extra
command.

The two CI steps therefore install nothing. They locate the ICD the build
already provisioned and set `VK_DRIVER_FILES`, so a manifest that failed to
name the driver fails the step. An `xlings install` in the step would have made
it pass either way.

macOS and Windows assert different things, and the difference is the point.
lavapipe is a software rasteriser producing the same pixels by construction, so
the image cannot distinguish it and the device name is what does. MoltenVK is
the host's own GPU through Metal, so its name differs by runner; what
distinguishes reaching a device there is that a centre pixel was reported at
all, which a portability driver the loader declined to show would not produce.

This closes the dependency between the two halves of the portability change:
the package without the enumeration finds no device, and the enumeration
without the package has nothing to find.
@Sunrisepeak
Sunrisepeak force-pushed the feat/vulkan-portability-enumeration branch from 0cbeee3 to 1a3fa43 Compare September 7, 2026 14:50
…missing package

A run step was written, pushed and measured. The manifest declaration worked --
`Provisioning [xlings.workspace] entries (xim:mesa-lavapipe@26.2.0)` -- the ICD
was found in the store, and the program still printed `render unavailable`,
which is its own report that it enumerated no device.

The missing piece is the LOADER, not the driver. `compat:vulkan` ships an import
library on Windows and nothing else, and its own descriptor says why: a
statically linked loader cannot work there, because upstream's
`loader_windows.c` creates its locks in `DllMain` and a static library never
gets one. The runtime `vulkan-1.dll` is expected to come from an installed GPU
driver, and a GitHub Windows runner has none. `xim:mesa-lavapipe`'s Windows
payload is an ICD that imports system DLLs and carries no loader either.

So the example declares no device on Windows: an entry there would download
56 MB that nothing can load. macOS is not in the same position -- `compat:vulkan`
builds the loader from source everywhere except Windows -- and keeps its run
step.

Raising this platform to "runs" needs a Windows `vulkan-1.dll` package, which is
a packaging decision rather than a step in a workflow file. Both the workflow
and the example manifest record the measurement where someone looking for the
gap will find it.
…sts no engine change

Two keys on a feature, and the engine holds no package name, no feature
spelling and no module name:

    [features.rules-slang]
    sources           = ["rules/slang.cppm"]
    rule_module       = "mcpp.rules.slang"
    device_extensions = [".slang"]

`device_extensions` classifies those extensions as device sources in a consumer
that activates the feature. `rule_module` is what a build program imports to
reach the rule. Two things follow.

A NEW DEVICE LANGUAGE NO LONGER TOUCHES THE ENGINE. Adding `.slang` to the
built-in table cost an engine change, a release, and a version bump in the rule
package's CI before the rule could route one file. `.slang` is removed from that
table here and `rules-slang` declares it instead; `tests/slang-consumer` builds
and runs unchanged, which is the only honest test of whether the mechanism
carries a language. The built-in list is now what mcpp knows without being told
-- a compatibility set for languages whose support shipped before the
declaration existed -- rather than a registry a sixth backend joins.

A CONSUMER WRITES ONE EDGE AND NO BUILD PROGRAM. `host-module = true` is implied
by `rule_module`, because a feature naming one has already said that is the only
way to use it. And a package with no `build.mcpp` gets the program its rules
describe written into the build directory:

    [build-dependencies.mcpp]
    plugins = { version = "0.3.0", features = ["rules-spirv"] }

is the whole declaration in `tests/spirv-zero-config`, which compiles a shader
and reaches it through a generated module. A project that writes its own
`build.mcpp` keeps it: synthesis fills an absence and never overrides.

THE FEATURE IS STILL REQUESTED BY NAME. An earlier revision derived it from the
extensions a project's sources carried, so a consumer could name the package
alone. That was withdrawn for two reasons and neither was cost. Two packages may
claim one extension -- a third-party CUDA rule is a thing someone will write --
and derivation would then guess or refuse where `features = [...]` has already
said which. And a manifest's job is to describe the build: a derived feature set
is information the file no longer states, which is worse for a reader and worse
for anything reading the manifest as context.

Which rules ran is said out loud, for the same reason the resolved toolchain is:

       Rules mcpp.rules.spirv (mcpp:plugins)

Two ordering defects were found by running it rather than by reading it. The
collection must sit between feature activation and the extension table that
narrows the constrained globs; placed after, the declared extensions arrived too
late to classify anything and the rule was handed an empty list. And the guard
on the build-program call asked whether the FILE existed while the function
asked whether a program was WANTED, which left the synthesis unreachable. Both
now ask the same question, and the comments say what the failure looked like.
…withdrawn

docs/05 gains the rule-feature section in both languages: what
`device_extensions` and `rule_module` state, the two things that follow
(`host-module` implied, a build program written where there is none), and why
the feature is still requested by name.

docs/20's device extension table is now described as what mcpp knows without
being told rather than as the registry a sixth backend joins, with a note
pointing at the declaration that adds to it. `.slang` has left the table.

The design document records the two designs that were written and withdrawn -- a
`[rules]` section and activating rules from the files present -- with the
objections that retired them, so the next reader does not rediscover the same
two shapes.
…tten

The seam is written by hand and the shader lane's equivalent is generated, which
reads as an inconsistency until the reason is stated. A device translation unit
is code, and its interface is a design decision no generator makes well; a
shader is data, and its interface is an address and a size.

Both are already invisible to a consumer -- only the seam includes the C header,
and everything downstream imports the module -- so the code lanes have been
module-first all along. The shader lane was the exception until 2026.9.7.1,
where the generated header WAS the interface.

Added in both languages.
…costs

Not required by the language: the seam can declare the entry point and the
island can define it, with no header, and that builds and runs. What the header
buys is that the declaration exists once.

Stated because the failure it prevents is the worst one available here. C
language linkage does not mangle, so two copies that disagree are one symbol:
the link is clean and each side reads the arguments by its own ABI. The property
that forces this boundary to be extern "C" is the same property that makes a
split declaration undetectable, and the three alternatives that would remove the
header are listed with why none does.

Both languages.
`Is the extern "C" header required?` and its Chinese twin were question
headings. The check names the rule and both files; the content is unchanged.
`run_build_program` returned success when the `std::ofstream` for the
synthesised program failed to open. The caller reads that as "this package
has no build program", so the device sources are handed to nobody, nothing
is generated, and the build succeeds. The first symptom is an unresolved
name in a consumer that imported an interface the program was to write --
three edges from the write that failed.

Both the open and the close are checked: a stream that opened and then
failed while flushing leaves a truncated program, which compiles into a
different defect.
Three places recorded a missing `vulkan-1.dll` as the cause of
`render unavailable` on the Windows runner. The log refutes it. That line
is printed by `src/main.cpp` after the render function returns nothing,
and the Vulkan leg imports `vkCreateInstance` from `vulkan-1.dll` through
the import library -- a process that could not find that DLL would fail
during image load and print nothing. It printed. mcpp-index's own
`vulkan-tests` member calls `vkEnumerateInstanceVersion` on the windows
shards and passes, which says the same thing from the other side.

The conclusion came from generalising the descriptor's note about STATIC
linkage into a claim about building at all. Measured against that: the
Khronos loader in `compat:vulkan` cross-builds into a working
`vulkan-1.dll` from the source the index already carries -- 265 exports
matching upstream's `vulkan-1.def` name for name, DllMain present,
importing only ADVAPI32, CFGMGR32, KERNEL32 and msvcrt. So a hermetic
Windows loader is available whenever it is wanted, and it is not what this
step is waiting on.

What is open is why the lavapipe payload's ICD enumerates no device under
a process mcpp launched. The notes are corrected in place rather than
deleted, so a reader sees what was believed and what refuted it.
Found by self-review of the module surface. Every name a rule package
generates is derived from the package's name -- the module a consumer
imports, the namespace the accessors sit in, the symbols in a generated
header -- and nothing in the build-program contract answered that
question. The closest available answer was the leaf of MCPP_MANIFEST_DIR,
which is a directory name.

The two differ whenever a project lays a package out under a generic
folder, and `mcpp.rules.spirv` was measurably wrong there:
`examples/09-heterogeneous/vulkan/app/` declares `name = "vulkan-saxpy"`
and generated `app.shaders`, so every `<something>/app/` in a workspace
claimed one module. A user-facing name derived from a folder is also a
name that changes when a folder is renamed.

MCPP_PKG_NAME and MCPP_PKG_NAMESPACE, with `mcpp::package_name()` and
`mcpp::package_namespace()` reading them. Set at both sites that build a
BuildProgramEnv -- the dependency loop and the root; those two are the
whole enumeration.

The e2e distinguishes the two derivations by construction: the fixture's
directory is `app` and its package is `vulkan-saxpy`, so an implementation
that still read the directory writes `app` where the assertion expects
`vulkan-saxpy`. A fixture whose name matched its folder would pass either
way, which is what every existing one did.
…eader

The style check refuses the second person in a reference document. Both
rows said what a rule reads an absent variable as by addressing it
directly; they now state it.
Three corrections to the 2026.9.7.1 entry.

The `.slang` bullet described the first design -- adding the extension to
the engine's built-in table. That inverted the dependency: a general build
system would hold a list of an external plugin's languages, and a
third-party rule would still wait on an engine release. What shipped is
`device_extensions` and `rule_module` on the rule package's own feature,
with `.slang` REMOVED from the engine table and `tests/slang-consumer`
building unchanged as the criterion.

The synthesised `build.mcpp` was not mentioned at all.

The graphics section claimed the example declares a device under
`cfg(windows)` and that two CI steps run it. Neither is true: Windows
declares none and the step was withdrawn. The corrected text also records
why -- the failure is after the loader, not at it.
@Sunrisepeak Sunrisepeak changed the title build.mcpp: three channels a rule package had no way to reach the engine through build.mcpp: four channels a rule package had no way to reach the engine through Sep 7, 2026
@Sunrisepeak
Sunrisepeak merged commit 4d3e969 into main Sep 7, 2026
38 checks passed
@Sunrisepeak
Sunrisepeak deleted the feat/vulkan-portability-enumeration branch September 7, 2026 19:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants