Repository navigation
feat(validator): durable EIP-3076 slashing protection - #734
pablodeymo wants to merge 3 commits into
Conversation
…and attestation is checked against a per-validator record of EIP-3076 minimal watermarks (highest block slot, highest attestation source and target) that is written to disk before the signature exists. The record lives in <validators-dir>/slashing_protection.json as an EIP-3076 v5 minimal interchange bound to the chain's genesis_validators_root, is written through a temporary file, fsync, rename and directory fsync, and is locked to one process. Reading fails closed, a failed write signs nothing, and a key with no record is refused at startup unless --init-slashing-protection is passed. Keymanager imports register a key before loading it. This replaces the in-memory AttestationGuard and ProposalGuard, whose rule and tests move into the new slashing_protection module, and removes the startup warning that the client kept no record.
🤖 Claude Code ReviewReview of PR 734The only change in the working tree is a 4-line edit to
The diff replaces the description of durable slashing protection with "still keeps no slashing-protection record". The code on this branch has that protection:
The text being removed was accurate. It said the client keeps EIP-3076 minimal-watermark protection in The new text says the opposite. Operators or agents who read Suggested fix: revert the Caveats
Automated review by Claude (Anthropic) · sonnet · custom prompt |
🤖 Kimi Code ReviewI'll review this PR which introduces durable slashing protection to replace in-memory guards. This is critical consensus-layer code affecting validator safety. Security & Correctness1.
|
| Severity | Count | Items |
|---|---|---|
| Critical | 0 | — |
| High | 1 | Item 4: macOS fsync vs F_FULLFSYNC durability concern |
| Medium | 2 | Item 1: spawn_blocking panic handling; Item 3: O(n) clone per write |
| Low | 4 | Items 5, 8, 9, 11, 12 |
The implementation is fundamentally correct and well-structured. The most significant concern is the macOS durability trade-off (Item 4), which should be documented more prominently or revisited. The O(n) clone (Item 3) should be benchmarked for expected validator counts.
Automated review by Kimi (Moonshot AI) · kimi-k2.6 · custom prompt
🤖 Codex Code ReviewLooks good overall. The PR materially improves validator safety by moving attestation/proposal guards to durable pre-sign recording, using atomic replace + fsync semantics, and sharing the guard with key import. I did not spot a consensus-critical bug in fork choice, state transition, SSZ, or signature logic because this PR stays in the validator client path.
Automated review by OpenAI Codex · gpt-5.4 · custom prompt |
MegaRedHand
left a comment
There was a problem hiding this comment.
The Kurtosis assertoor battery will fail on main after this merges.
The pinned lambdaclass/ethereum-package (67cb0e02, src/vc/ethlambda.star) starts ethlambda validator with --validators-dir=/data/ethlambda/validators, which is a fresh directory on every run, and without --init-slashing-protection. tooling/kurtosis-assertoor/network_params.yaml sets no vc_extra_params for the two ethlambda VC participants.
So SlashingProtection::open hits !missing.is_empty() && !init and the VC exits. The block-proposal check for participants 2 and 3 can then never pass. This PR's CI will not show it, because kurtosis_assertoor.yml runs on pull requests only for changes under its own directory or workflow; the failure appears on the first push to main.
Fix: add vc_extra_params: ["--init-slashing-protection"] to both ethlambda VC participants (safe there, since the genesis keys have never signed), or pass the flag in the fork's launcher.
| // record keeps it. The history the request may carry is not imported yet, | ||
| // so a key arriving from another client starts as never having signed: | ||
| // see the warning in `import`. | ||
| let registered = context |
There was a problem hiding this comment.
Keymanager import registers keys as never having signed, which skips the --init-slashing-protection gate and drops the history the caller sent.
register() writes Record::default() for every new key. When the request includes slashing_protection, import only logs a warn!. The CLI calls this exact action DANGEROUS and refuses it without a flag. The API does it on every import, with no flag and no two-epoch condition.
Scenario: a key is deleted from another VC (or from this one after a DELETE) and imported here with its interchange. It is registered with no watermarks and attests in the same epoch, which is a double vote.
Until PR 2, it would be safer to refuse the import when slashing_protection is present, or to seed the new record at the current epoch/slot rather than None.
There was a problem hiding this comment.
Done in 2808b51. Keys registered without history (--init-slashing-protection and keymanager imports) now start at the current slot and epoch. One difference from the suggestion: the attestation source stays at 0. The source a beacon node hands out is the justified epoch, which lags the current one, so seeding it at the current epoch would refuse every attestation until justification caught up, and on a non-finalizing chain, forever. The target watermark alone closes the double-vote window. The surround window against older history stays open, which is why the flag's docs still say "two epochs".
| reason, | ||
| }; | ||
|
|
||
| let existed = path.exists(); |
There was a problem hiding this comment.
path.exists() fails open. It returns false for a dangling symlink and for any metadata error (for example EACCES). In both cases the record is treated as absent.
Scenario: an operator symlinks slashing_protection.json to a separate volume, and that volume is not mounted at startup. With zero keys at startup (keymanager-only), or with --init-slashing-protection left in the unit file, open writes a fresh empty record. rename then replaces the symlink itself, so the real history is orphaned and every key signs from None.
Suggestion: use symlink_metadata / try_exists. Treat only NotFound as absent, and refuse anything that is not a regular file.
There was a problem hiding this comment.
Fixed in 2808b51. Only NotFound counts as absent; a symlink, directory or metadata error is refused instead of replaced (file::exists), with a test for a dangling symlink.
| let secret = keystore.decrypt(password).map_err(KeyStatus::error)?; | ||
| let pubkey = ValidatorStore::derive_pubkey(&secret).map_err(KeyStatus::error)?; | ||
|
|
||
| // Registered before the key is written to disk or reachable through the |
There was a problem hiding this comment.
A failed import leaves a "never signed" record behind, and that record later bypasses the startup gate.
register() commits before persist_import. When persist_import fails (disk, definitions save, symlink refusal), the status is error, but nothing removes the empty record that was just written.
Later the operator adds the same keystore by hand, for example copied from the client that ran it. On restart, open() finds a record, so --init-slashing-protection is never required and the key starts as never having signed.
Suggestion: register only after the key is persisted (keeping the order "record exists before the key is loaded"), or remove the record on failure if this call created it.
There was a problem hiding this comment.
Fixed in 2808b51. Keys are registered in one write, after they are persisted and before any is loaded, so a failed import leaves no record (test: a_failed_import_leaves_no_slashing_protection_record).
| // A poisoned lock is treated as a refusal rather than unwrapped: a | ||
| // crash in the duty path would be a worse outcome than a skipped | ||
| // proposal. | ||
| match self.protection.lock() { |
There was a problem hiding this comment.
The std::sync::Mutex is taken on the executor here (and in keystores::prepare_import), while another holder may be inside an fsync.
The SharedProtection doc says async callers take it inside spawn_blocking. This pre-check does not, and keymanager register does a full rewrite plus two fsyncs inline in an axum handler.
Concrete case: an attestation duty abandoned at its deadline (timeout in attest_and_aggregate) does not cancel the detached blocking task, so its write can still be running at the next slot boundary. propose then blocks the duty-loop worker on lock() for as long as the disk takes.
Suggestion: move the pre-check and register into spawn_blocking too.
| let first = encode_hex(&missing[0].0); | ||
| return Err(refused(format!( | ||
| "{} validator key(s) have no slashing-protection record (first: {first}). \ | ||
| Import their history from the client that last ran them, or, only if they \ |
There was a problem hiding this comment.
This startup error points operators to a path that does not exist. "Import their history from the client that last ran them" is not supported in this PR: there is no CLI import, and the keymanager ignores slashing_protection.
An operator migrating keys therefore has one real option, --init-slashing-protection. That is the dangerous one for a key that signed recently elsewhere.
Suggestion: until PR 2, say that import is not available yet, and spell out the wait condition (stop the old client and wait at least two epochs) as a requirement.
| @@ -268,30 +266,38 @@ impl<B: BeaconNodeApi> ProposalService<B> { | |||
| // guard's slot is recorded against and what the signature covers. | |||
There was a problem hiding this comment.
Stale "guard" wording remains after the guards were removed. Locations:
- here, and
proposal.rs:226 - the test docs at
proposal.rs:653, 800, 1072 error.rs:75: theProposalRefusedrustdoc still says "The proposal guard refused"error.rs:140: "A refusal is deterministic". That is no longer true, sinceProposalRefusednow also carries record-write failures.beacon_node/fallback.rs:609, 625andbeacon_node/mod.rs:163
There was a problem hiding this comment.
Fixed in 2808b51, including ProposalRefused's rustdoc and the is_retryable comment, which now covers record-write failures too.
| /// Created, or truncated if it exists, at mode `0600` on unix: the record says | ||
| /// which keys this client holds. | ||
| #[cfg(unix)] | ||
| fn open_private(path: &Path) -> std::io::Result<File> { |
There was a problem hiding this comment.
Reuse (cleanup). open_private re-implements secure_fs::write_private. secure_fs.rs:8-10 says every file that "decides what a validator signs" must go through that module. write_durably is also a second tmp+rename next to ValidatorDefinitions::save (keys/definitions.rs:54-75), which has no fsync.
Suggestion: add one secure_fs::write_private_durably (tmp at 0600, sync_all, rename, directory fsync) and use it for both files.
There was a problem hiding this comment.
Done in 2808b51: secure_fs::write_private_durably is shared by the record and validator_definitions.yml, which gains its missing fsyncs.
| let pubkeys: Vec<BlsPubkey> = candidates.iter().map(|(_, pubkey)| *pubkey).collect(); | ||
| let protection = self.protection.clone(); | ||
| let recorded_data = data; | ||
| let verdicts = tokio::task::spawn_blocking(move || { |
There was a problem hiding this comment.
Simplification (cleanup). This block (spawn_blocking, then lock, then map poison to a string, then flatten the JoinError) appears again almost verbatim in proposal.rs:274-285.
Suggestion: one helper in slashing_protection, for example async fn blocking<T>(p: &SharedProtection, f: impl FnOnce(&mut SlashingProtection) -> Result<T> + Send) -> Result<T, String>. It removes the duplicate, and the keymanager and proposal pre-check could use it too (see the executor-blocking comment).
There was a problem hiding this comment.
Done in 2808b51: one slashing_protection::blocking helper replaces the duplicated blocks.
| self.records.extend(changes); | ||
| return Ok(()); | ||
| }; | ||
| let mut next = self.records.clone(); |
There was a problem hiding this comment.
Efficiency (cleanup). Each commit clones the whole records map and rewrites and double-fsyncs the full pretty-printed file, while holding the mutex. So the cost of every slot's batch and every block grows with the total number of keys, not with the batch.
The keymanager makes it worse: register(&[pubkey]) runs once per key, so an import of N keystores does N full rewrites and 2N fsyncs, and attestation writes wait behind them.
Cheaper options:
- Serialize from
self.recordswithchangesoverlaid, without cloning. - Use compact JSON.
- Register all decrypted keys of one import in a single commit.
There was a problem hiding this comment.
Done in 2808b51: no clone per write (unchanged entries are chained with the changes), compact JSON, and one write per keymanager import.
| "Registering a validator with no slashing-protection history \ | ||
| (--init-slashing-protection)" | ||
| ); | ||
| records.insert(*key, Record::default()); |
There was a problem hiding this comment.
Altitude. init (and keymanager registration) seed None, so "safe only if nothing was signed for two epochs" is documentation, not enforcement. open runs after genesis()/spec(), so the current slot and epoch are known.
Seeding block_slot = current_slot and attestation = (current_epoch, current_epoch) would cost at most one epoch of duties. It would close the double-vote window for a key that signed recently elsewhere, both here and on the keymanager path. Worth considering, even if it lands in PR 2.
There was a problem hiding this comment.
Done in 2808b51. Keys registered without history (--init-slashing-protection and keymanager imports) now start at the current slot and epoch. One difference from the suggestion: the attestation source stays at 0. The source a beacon node hands out is the justified epoch, which lags the current one, so seeding it at the current epoch would refuse every attestation until justification caught up, and on a non-finalizing chain, forever. The target watermark alone closes the double-vote window. The surround window against older history stays open, which is why the flag's docs still say "two epochs".
…thout history (--init-slashing-protection, keymanager imports) now start at the current slot and epoch instead of as never having signed, closing the double-vote and double-proposal window for a key that signed elsewhere a moment ago; the source stays at zero so a lagging justified checkpoint does not stall attestations. The record's existence check fails closed: only a missing file is absence, and a symlink, directory or unreadable path is refused instead of replaced. Keymanager imports register keys in one write after they are persisted, so a failed import leaves no record behind. Every async access to the record, including the proposal pre-check and keymanager registration, goes through one spawn_blocking helper. The durable write moves to secure_fs and is shared with the validator definitions file, which gains its missing fsyncs; each write no longer clones the record and is compact JSON. A failed attestation write counts every refused attestation, the permission-based tests skip under root, stale guard wording is gone, the docs say the lock is per directory and that --validators-dir must be writable, and the Kurtosis battery passes --init-slashing-protection to its two ethlambda validator clients.
|
Kurtosis battery (review summary): Added |
… write creates its temporary file fresh with create_new at mode 0600, removing whatever was at that path first, so a leftover of a crashed write cannot carry a wider mode into the renamed record and a symlink planted there is never written through; two tests pin both. The directory flush after the rename tries File::sync_all first, which is F_FULLFSYNC on macOS like the file's own flush, and falls back to a plain fsync only where a filesystem refuses that for a directory. The blocking helper returns the crate's own Error instead of a string, so a caller can tell a failed write from a record that cannot be reached at all: a lock poisoned by a panic, or the runtime shutting down, is the new SlashingProtectionUnusable, whose message says the client must be restarted. Attested gains a refused count so a slot where slashing protection withheld every signature, or could not record them, is distinguishable from a slot with nothing to publish.
🗒️ Description / Motivation
ethlambda validatorkept no slashing-protection record. Its two in-memory guards (AttestationGuard,ProposalGuard) applied EIP-3076's minimal rule within one run, but a restart emptied them and nothing stopped a second process from signing with the same keys. The client warned about this on every start, anddocs/spec_deviations.mdlisted it as a known gap. That gap is the blocker for anyone running ethlambda with real stake.This PR makes slashing protection durable. It is the first of two PRs; the second adds EIP-3076 interchange import/export (CLI and keymanager API).
Design. Before choosing, we surveyed Lighthouse, Prysm, Teku, Nimbus, Lodestar and Grandine.
What Changed
crates/validator/src/slashing_protection/:mod.rs,SlashingProtection:target > last targetandsource >= last source; a block needsslot > last slot;None, so slot 0 and epoch 0 are signable exactly once;openand the batch APIs;file.rs, the on-disk record:<validators-dir>/slashing_protection.jsonis an EIP-3076 v5 minimal interchange naming the chain'sgenesis_validators_root;fsync,rename, directoryfsync;File::try_lockonslashing_protection.lock.attestation.rs: one slot's batch is checked and recorded in a single write, insidespawn_blockingand before the store's read lock is taken. Only then are the allowed attestations signed.proposal.rs: the cheap pre-check stays beforeproduce_block. The recording check moves before signing and persists.lib.rs):genesis(), which supplies the root it must match;--init-slashing-protectionis passed;POST /eth/v1/keystoresregisters each imported key before writing it to disk or loading it.attestation_guard.rsandproposal_guard.rs. Their rule and tests moved intoslashing_protection.ethlambda_validator_slashing_protection_write_secondsandethlambda_validator_slashing_protection_write_failures_total.docs/spec_deviations.md: the section is rewritten to describe the minimal-watermark deviation.docs/cli.mdandREADME.md: the new flag and the slashing-protection callout.docs/metrics.mdandCLAUDE.md: updated to match.Correctness / Behavior Guarantees
Ordering: check, then a durable record, then the signature. A signature that exists is always one the record already accounts for, whether the process then crashes, hits its duty deadline, or is rejected by the beacon node.
Fail closed: startup stops on any of these, never reading them as "nothing signed":
genesis_validators_root;At signing time, an unregistered key or a failed write refuses the message.
One process: a second validator client on the same
--validators-dirrefuses to start while the first holds the lock.Behavior change for operators: the first start after upgrading refuses with a message naming
--init-slashing-protection. That flag registers each key as if it had signed a block and an attestation at the current slot and epoch, so it costs at most the rest of the epoch. It is still meant only for keys that have signed nothing anywhere for at least two epochs, and it never lowers an existing record.Behavior change for deployments:
--validators-dirmust now be writable, for the record and its lock. A read-only mount of the keys (a Kubernetes Secret, a:rovolume) fails at startup.Known cost: a byte-identical re-sign after a crash is refused, as in Teku. Documented in
spec_deviations.md.Not in this PR: carrying history in or out. Keymanager
DELETEstill returns the empty, all-zero-root interchange, andPOSTstill ignoresslashing_protection. Both are PR 2.Tests Added / Run
slashing_protectionunit tests:open;open;initand registered with it;initnever lowers a record;MockBeaconNode):a_restart_does_not_forget_what_was_attested;a_restart_does_not_forget_what_was_proposed(no block is even requested);a_record_that_cannot_be_written_signs_nothing.an_imported_key_is_registered_for_slashing_protection, where the next start opens without--init-slashing-protection.make fmt: cleanmake lint: cleanmake test: 2878 passed, 0 failed (after the review follow-ups); the validator crate alone is at 344 after the bot follow-upsReview follow-ups (2808b51)
mainvc_extra_params: ["--init-slashing-protection"]on both ethlambda validator clients. The Assertoor check on this PR fails for the same chicken-and-egg reason: PR runs test the publishedunstableimage, which does not know the flag yet. The post-merge run onmaintests the merge commit's image, which does--init-slashing-protectionand keymanager imports now seed the current slot and epoch (block slot = now, attestation target = current epoch). The source stays at 0: seeding it higher would refuse every attestation until justification caught uppath.exists()fails openNotFoundcounts as absent. A symlink, directory or unreadable path is refused instead of replacedslashing_protection::blockinghelper--validators-dirdocs/cli.mdgeteuid() == 0ProposalRefusedandis_retryablesecure_fs::write_private_durablyand is shared withvalidator_definitions.yml, which gains its missing fsyncs. No clone of the record per write, compact JSON, one write per keymanager importBot review follow-ups (696e2a8)
fsyncis notF_FULLFSYNC(Kimi 4)File::sync_allfirst,F_FULLFSYNCon macOS like the file's own flush, and falls back to a plainfsynconly where a filesystem refuses that for a directory. The original error is kept if both fail.tmppath (Kimi 5, 11)0600), but a leftover.tmpfrom a crashed write kept its old mode, sincemode()applies only at creation. The temporary is now removed first and created withcreate_new, so the record never inherits a wider mode and a symlink planted at the.tmppath is never written through. Two tests pin bothslashing_protection::blockingreturns the crate'sErrorinstead of aString. A poisoned lock, or a panic inside the operation, is the newSlashingProtectionUnusable, whose message says the client must be restarted; a failed write staysIo. Never retryableattestanswersOkwith zero published on a record failure (Codex)Attestedgains arefusedcount, so a slot where protection withheld every signature, or could not record them, is distinguishable from a slot with nothing to publish. The refusal tests assert itRelated Issues / PRs
eth-clients/slashing-protection-interchange-testsv5.3.0 vectors.✅ Verification Checklist
make fmt— cleanmake lint(clippy with-D warnings) — cleanmake test(test-consensusplustest-node, atrelease-fast) — all passing