Skip to content

fix(sdk): stop silently truncating v0 key material, bound error bodies, unwedge ct_monitor - #1281

Open
kvinwang wants to merge 4 commits into
nextfrom
fix/sdk-parser-bounds
Open

kvinwang wants to merge 4 commits into
nextfrom
fix/sdk-parser-bounds

Conversation

@kvinwang

Copy link
Copy Markdown
Collaborator

Problem

An audit of every parser in the four SDKs, ct_monitor and the operator CLIs, done by feeding the same malformed response to all four SDKs and tabulating what each did. 39 v1 cases and 20 v0 cases; the harness is added here.

The v1 surface is uniform — all four SDKs accept and reject exactly the same v1 responses, 39/39. Every problem below is on v0 or outside the SDKs.

1. JS v0 silently truncates malformed hex into valid-looking key material

Node's hex decoder stops at the first pair it cannot parse and returns the prefix, with no error. Given a GetKey response whose 64 valid digits are followed by junk:

key the SDK returned
rust error: Invalid character 'G' at position 64
python error: ValueError
go error: encoding/hex: invalid byte: U+0047 'G'
js a plausible 32-byte Uint8Array, no error

toViemAccount turns that into a working Ethereum account at an address nobody chose. v1 already guards this — decode_hex in client-v1.ts has a comment explaining exactly this failure mode — but the frozen v0 client never got the same treatment.

Two more from the same pass: a null or absent signature_chain threw a bare TypeError: Cannot read properties of null (reading 'map'), which names no field and reads like an SDK bug; and info() accepted a numeric tcb_info, because JSON.parse stringifies its argument, handing back 42 typed as TcbInfo with every .mrtd read undefined.

2. Go and Python put the entire response body in the error message

An agent with no route for the path answers with an HTML page. A 20 KB response produced a 20 KB error — 20035 bytes in Go, 20176 characters in Python. Rust already caps this at 512 characters and JS at 300, both with comments saying why.

3. ct_monitor stops detecting after the first failing certificate

The scan loop propagated the first failure with ?, returning before last_checked was ever assigned. Sixty seconds later the same page of up to 10 000 logs was re-fetched and the pass stopped at the same entry — forever. run only logs the error, so the monitor went on looking healthy while checking nothing.

The trigger does not have to be a real mis-issuance: check_one_log fetches each certificate from crt.sh and parses it as PEM, so a single 429 whose HTML body fails to parse wedges it just as effectively. On a first run last_checked is None, so a domain with any certificate history issues up to 10 000 sequential crt.sh fetches — which will itself provoke the rate limiting that causes the wedge. It is also a detection-evasion primitive: one benign unknown key, such as a rotated gateway key, masks every log recorded after it.

Fix

Four commits, one concern each.

  • fix(sdk-js) — move v1's strict decoders into shared.ts, which exists for exactly this, and use them from both surfaces so the two cannot drift again.
  • fix(sdk) — Go and Python adopt Rust's bound and Rust's rule: prefer the prpc error field, fall back to the raw body, count characters not bytes. Go also stops reading at 64 KiB rather than buffering a page it will throw away.
  • fix(ct_monitor) — a failing log is reported and the pass continues, the watermark advances, and failures are summarised into one error. The loop is extracted into scan, away from the HTTP calls, so it can be tested without a network; the crate had no tests at all.
  • test(sdk) — the differential harness.

Compatibility

The SDKs are published packages with external users, so each change is stated explicitly.

change rejects input that previously worked?
JS v0 strict hex Yes — malformed hex now throws instead of returning a truncated Uint8Array. The value it returned was a truncated private key; no working application can depend on it.
JS v0 named list/field errors No — these already threw, with a worse message.
JS v0 tcb_info must be a string Yes — a non-string tcb_info now throws instead of returning a number typed as TcbInfo. Unusable either way.
Go/Python error bounds No — error text only.
ct_monitor No API. A bad certificate is now alerted on once per appearance rather than every minute forever.

A well-formed response, an empty hex string and an empty chain are unchanged in every SDK. The preference throughout was to turn a crash or a silent wrong value into a clean error, never to turn a lenient accept into a reject — the two exceptions above are the cases where the lenient accept handed back unusable key material.

Deliberately not changed

  • GetTlsKeyResponse.asUint8Array returns different bytes in Go than in Python and JS. Same PEM, same call: Go parses PKCS#8 and returns the EC scalar; Python and JS base64-decode the PEM body and return the raw DER prefix — ASN.1 header and curve OID, not the key. So ToEthereumAccount in Go and toViemAccount in JS derive different Ethereum addresses from the same TLS key. Both are published and both are live, so whichever side changes moves someone's assets. This needs a migration story, not a patch.
  • Python and JS zero-pad a short decode into a full-length key. Same frozen-surface argument; client-v0.ts already records the padding as load-bearing.
  • Go v0 reads a missing or null required field as empty, so DecodeKey() returns ([]byte{}, nil) for a response carrying no key. The right fix is the one Go's own v1 client already made with *string, but it rejects responses that today decode to empty and deserves its own deliberate decision rather than a rider here.
  • sdk/go/ratls indexes [:8] on unchecked parser output (ratls.go:114, :229) — the only genuinely hostile input in the SDKs, since it comes from a remote TLS peer. I could not demonstrate a panic: both values originate in dcap-qvl's fixed-size Rust arrays, and the module does not build without a native libdcap_qvl. Worth noting separately that sdk/go/ratls is a distinct Go module that sdk/run-tests.sh never builds or tests.

Verification

Full suites, against a simulator built from this tree:

sdk/rust     11 test binaries, all ok
sdk/go       ok  .../sdk/go/dstack   ok  .../sdk/go/tappd
sdk/python   174 passed
sdk/js       167 passed  (155 before; +12 new)
ct_monitor   6 passed    (0 before -- the crate had no tests)
dstack-cli / dstack-cli-core / dstackup   5, 17, 31 passed
cargo fmt --all --check                   clean
cargo clippy --lib --bins -D warnings     0 warnings

Every fix was reproduced before it was written. The ct_monitor tests were run first against a faithful transcription of the old loop and failed exactly as predicted:

---- tests::a_failing_log_does_not_stop_the_pass stdout ----
assertion `left == right` failed: every log must still be checked
  left: [3, 2]
 right: [3, 2, 1]

cargo clippy --all-targets does not pass on next either (pre-existing lints elsewhere), so the gate used here is the lib/bins target. sdk/rust also has one pre-existing cargo fmt diff in types/src/dstack_v1.rs, present on next and untouched by this branch.

Node's hex decoder stops at the first pair it cannot parse and returns the
prefix, with no error. A `GetKey` response whose 64 valid digits are followed
by junk therefore yielded a plausible 32-byte `key`, and `toViemAccount` turned
that into a working account at an address nobody chose. Rust, Python and Go all
refuse the identical response.

v1 already guards this (`decode_hex` in `client-v1.ts`); the frozen v0 client
never got the same treatment. Move v1's decoders into `shared.ts` -- which
exists for exactly this -- and use them from both surfaces, so the two cannot
drift again.

Also from the same pass: a null or absent `signature_chain` threw a bare
`TypeError: Cannot read properties of null (reading 'map')`, which names no
field and reads like an SDK bug; and `info()` accepted a numeric `tcb_info`,
because `JSON.parse` stringifies its argument, handing back `42` typed as
`TcbInfo` with every `.mrtd` read `undefined`.

Compat: this rejects three response shapes that previously returned a value --
malformed hex, a missing repeated field, and a non-string `tcb_info`. In every
case the value returned was unusable (a truncated private key, an empty chain,
a number typed as a struct), so no working application can depend on it. A
well-formed response, an empty hex string and an empty chain are unchanged.
An agent with no route for the path answers with an HTML page. Go put the whole
body in the error string and Python put the whole body in the exception
message: a 20 KB response produced a 20 KB error, measured at 20035 bytes in Go
and 20176 characters in Python.

Rust already caps this at 512 characters and JS at 300, both with comments
saying why, so this is the two stragglers catching up rather than a new policy.
Use Rust's number and Rust's rule -- prefer the prpc handler's `error` field
when the body is one, fall back to the raw body when it is an HTML page, and
count characters rather than bytes so the bound cannot land inside a multi-byte
sequence.

Go also stops reading at 64 KiB instead of buffering a whole page it is going
to throw away, and now reports `HTTP <status>: <reason>` rather than
`unexpected status code: <status>, body: <everything>`, which matches what the
other three SDKs print. The read bound is deliberately larger than the quote
bound: the quoted text is lifted out of the `error` field *inside* the body, so
a body cut off mid-string is no longer JSON and every large prpc error would
degrade into a raw truncated blob.

Compat: error text only. No response that parsed before parses differently.
The scan loop propagated the first failure with `?`, which returned before
`last_checked` was ever assigned. Sixty seconds later the same page of up to
10 000 logs was re-fetched and the pass stopped at the same entry -- forever.
`run` only logs the error, so the monitor went on looking healthy while
checking nothing.

The trigger does not have to be a real mis-issuance. `check_one_log` fetches
each certificate from crt.sh and parses it as PEM, so a single 429 whose HTML
body fails to parse wedges the monitor just as effectively. On a first run
`last_checked` is `None`, so a domain with any certificate history issues up to
10 000 sequential crt.sh fetches, which will itself provoke the rate limiting
that causes the wedge. It is also a detection-evasion primitive: one benign
unknown key -- a rotated gateway key, say -- masks every log recorded after it.

So a failing log is now reported and the pass continues, the watermark advances,
and the failures are summarised into one error at the end. The trade is that a
certificate that genuinely should not exist is alerted on once instead of every
minute; the alternative was alerting forever about one certificate and never
looking at another.

Extract the loop into `scan`, away from the HTTP calls, so it can be tested
without a network -- the crate had no tests at all. Stopping at the previous
watermark and reporting one that fell off the page are unchanged, and both are
now covered.
Four implementations of one protocol is the shape that produces parser
differentials, and the only way to find them is to feed the same bytes to all
four. This serves one canned guest-agent response over a unix socket and
tabulates what Rust, Python, Go and JavaScript each do with it.

It found the three bugs fixed in this branch, and it is what says the v1
surface agrees on all 39 of its cases -- a negative result worth being able to
re-derive.

Not wired into `sdk/run-tests.sh`: it needs all four toolchains, and it answers
"do these agree?" rather than asserting a fixed expectation. Run it by hand
when the wire format changes or when one SDK's decoding is touched.

It does not use `sdk/simulator`, deliberately. The simulator is the real guest
agent with one trait swapped, which is exactly what makes it trustworthy -- a
test that passes against it cannot be rejected by a real agent on validation
grounds. A fault-injection mode would put test-only branches in the production
handler and cost that property.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant