feat: add lambda-runtime-invocation-id header - #1159
Conversation
b60f865 to
048c1e8
Compare
4c28d43 to
afb6708
Compare
afb6708 to
de1f2e4
Compare
c403f80 to
31c22d2
Compare
31c22d2 to
2cabdac
Compare
| return scenarios | ||
|
|
||
|
|
||
| def get_invocation_id_scenarios(): |
There was a problem hiding this comment.
This is not testing the invocation id cross wiring situation. the assertion is on request id, which is going to be the same. But we want to check that response is coming appropriately.
batches = [
[Request.create(
payload={"command": "invoke-A", "sleep": TIMEOUT + 2},
assertions=[{"transform": ".errorType", "error": "Sandbox.Timedout"}],
headers={"X-Amzn-RequestId": SAME_REQUEST_ID},
)],
[Request.create(
payload={"command": "invoke-B", "sleep": TIMEOUT - 1},
assertions={"response": {"from": "invoke-B"}},
headers={"X-Amzn-RequestId": SAME_REQUEST_ID},
)],
]
fc33a71 to
1152b7c
Compare
jlizen
left a comment
jlizen
left a comment
There was a problem hiding this comment.
Core approach is good, but some small tweaks.
Also: currently we log Lambda function timeout! for any 410, but we now will have a 410 if a stale response is rejected. We should tweak the message.
| let mut req = build_request().method(Method::POST).uri(uri).body(body)?; | ||
|
|
||
| if let Some(id) = self.invocation_id { | ||
| req.headers_mut().insert(LAMBDA_RUNTIME_INVOCATION_ID, id.parse()?); |
There was a problem hiding this comment.
This will error out and crash the runtime if a malformed invocation id header is sent.
We originally decode with String::from_utf8_lossy(), but that replaces non-utf8 bytes with U+FFFD which is anyway not ascii. So then this parse will fail.
I did a quick check and didn't find any other round trips, this new code is the only place impacted.
I know we control the sender but we should be defensive against malformed inputs anyway. I would suggest a rate-limited log warning if we have bad bytes.
| struct Request { | ||
| #[serde(rename = "command")] | ||
| _command: String, | ||
| sleep: u32, |
There was a problem hiding this comment.
This should have #[serde(default)], otherwise the example command in the header will actually have a deserialization failure.
| @@ -1,2 +1,2 @@ | |||
| """ | |||
| Multi-concurrency test scenarios for basic-lambda-concurrent. | |||
There was a problem hiding this comment.
this is stale now i think?
There was a problem hiding this comment.
IE, there are multiple lambda handlers now?
| TIMEOUT = 5 | ||
|
|
||
|
|
||
| def _make_env(concurrency: int = DEFAULT_CONCURRENCY) -> dict: |
There was a problem hiding this comment.
this should accept a handler input too, right now it makes it seem like everything is the basic concurrent
| let mut req = build_request().method(Method::POST).uri(uri).body(body)?; | ||
|
|
||
| if let Some(id) = self.invocation_id { | ||
| req.headers_mut().insert(LAMBDA_RUNTIME_INVOCATION_ID, id.parse()?); |
There was a problem hiding this comment.
nit: we use insert() here but append() on the streaming branch, I would make them consistent
| use tower::{service_fn, Service}; | ||
|
|
||
| #[tokio::test] | ||
| async fn forwards_invocation_id_from_next_response_headers() { |
There was a problem hiding this comment.
It would be good to have a test covering the deserialization error path as well
3 participants
Summary
Add
Lambda-Runtime-Invocation-Idheader support for cross-wiring protection.The RIC now echoes the invocation ID received from RAPID on
/nextback on/responseand/error, enabling RAPID to detect and reject stale responses from timed-out invocations.Problem
On Lambda Managed Instances (LMI) and On-Demand (OD), when an invoke times out, the runtime process continues running in the background. If a new invoke arrives with the same
requestId, RAPID accepts it. The still-running old invocation eventually posts its response, and RAPID matches it to the new invoke — delivering the wrong response (cross-wiring).Solution
RAPID sends a unique per-invoke nonce via
Lambda-Runtime-Invocation-Idheader on/next. The runtime echoes it back on/responseand/error. RAPID validates the match before accepting the response.Backward Compatibility
Fully backward compatible in both directions:
Testing