feat(mesh): upgrade to mesh-llm 0.76.0-rc3 and recommend Qwen3.8 27B - #6189
Open
micspiral wants to merge 2 commits into
Open
feat(mesh): upgrade to mesh-llm 0.76.0-rc3 and recommend Qwen3.8 27B#6189micspiral wants to merge 2 commits into
micspiral wants to merge 2 commits into
Conversation
Bump every mesh-llm git dependency in buzz-relay and buzz-desktop from v0.75.1 to v0.76.0-rc3, and make Qwen3.8 27B Q4_K_M the curated pick for 64GB-class machines. The rc carries the two changes that make Qwen3.8 safe to recommend: Mesh-LLM/mesh-llm#1283 routes Qwen3.8 identities to the qwen35 recurrent family (without it the identity compacts to a string matching neither qwen35 nor qwen36, so a hybrid recurrent model is classified as dense attention-KV), and #1342 restores recurrent shared prefixes. #1343 adds Qwen3.8-27B-Q4_K_M to the compiled MODEL_CATALOG. Because the large pick now exists in MODEL_CATALOG, the synthesized catalog entry Buzz carried for the gemma-4-26B pick is removed along with its size/file/description constants; the entry is canonicalized from the upstream catalog instead. The retired gemma-4-26B alias is kept in canonical_curated_model_id so machines that already selected it keep resolving to a model id Mesh's OpenAI ingress accepts. Dependency resolution notes: iroh moves 1.0.2 -> 1.0.3 to match the rc's requirement, and libsqlite3-sys moves 0.30.1 -> 0.35.0 because the rc's mesh-llm-log-store requires rusqlite 0.37 (only one package may link sqlite3). Verification: cargo check passes for buzz-relay and for buzz-desktop with --features mesh-llm; 2611 buzz-desktop tests pass; cargo fmt --check is clean. Two failures predate this change and reproduce on unmodified origin/main: the global_config inherited_shared_compute_translates_to_supported_agent_transport assertion (expects "auto", gets "mesh") and an items-after-test-module clippy lint in managed_agents/restore.rs. Not runtime-verified against a live Qwen3.8 mesh node from this branch. Co-authored-by: Michael Neale <michael.neale@gmail.com> Signed-off-by: Michael Neale <michael.neale@gmail.com>
buzz-agent's `BUZZ_AGENT_LLM_TIMEOUT_SECS` defaults to 240s. MeshLLM's OpenAI frontend gives a backend call 600s (`OpenAiFrontendConfig::DEFAULT_BACKEND_TIMEOUT`), so on shared compute the client aborts work the server is still legitimately doing. Requests to a local mesh are `stream: false`, so a cold multi-ten-thousand token prefill is one silent request. Measured on an M5 Max serving Qwen3.8-27B Q4_K_M: an 88,318-token cold prompt returns HTTP 200 after 503s at 176 tok/s prefill. At the 240s default the client gives up at four minutes, then retries with an escalated budget (240 -> 480 -> 960), piling load onto a box that is already prefilling the first attempt. Seat the mesh client budget at 660s, just above the frontend's 600s, so the mesh's own error surfaces rather than a client abort racing it. Remote providers are untouched and keep buzz-agent's 240s default. Like the other mesh seeds this is a default, not policy: an explicit user value survives via `insert_default_if_unset` plus the copy-forward list in `relay_mesh_process_env`. Co-authored-by: Michael Neale <michael.neale@gmail.com> Signed-off-by: Michael Neale <michael.neale@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Upgrades every mesh-llm git dependency (buzz-relay + buzz-desktop) from
v0.75.1tov0.76.0-rc3, and makes Qwen3.8 27B Q4_K_M the curated pick for 64GB-class machines.Why this rc, specifically, for Qwen3.8
Qwen3.8 could not be recommended on 0.75.1. Two upstream changes in this rc are correctness prerequisites, not polish:
qwen35recurrent family. Without it,Qwen3.8-...compacts to a string containing neitherqwen35norqwen36, family resolution returnsNone, and the fallback classifies a hybrid recurrent model as dense attention-KV — so the planner treats SSM stages as freely-migratable KV.Qwen3.8-27B-Q4_K_Mto the compiledMODEL_CATALOG.Catalog change
Because the large pick now exists in
MODEL_CATALOG, the synthesized entry Buzz carried for the gemma-4-26B pick is deleted along with its_SIZE/_FILE/_DESCRIPTIONconstants — the entry is canonicalized from the upstream catalog instead. The retired gemma-4-26B alias is kept incanonical_curated_model_idso machines that already selected it keep resolving to a model id Mesh's OpenAI ingress accepts.The small tier (
gemma-4-E4B-it:Q4_K_M) and the 64GB rated-capacity boundary are unchanged.Dependency resolution
Two transitive bumps were forced, both required to resolve at all:
iroh1.0.2 -> 1.0.3 (the rc requires^1.0.3).libsqlite3-sys0.30.1 -> 0.35.0 — the rc'smesh-llm-log-storeneedsrusqlite 0.37, and only one package in the graph maylinks = "sqlite3".Verification
cargo check -p buzz-relay— clean.cargo check --features mesh-llm(buzz-desktop) — clean.cargo test --features mesh-llm(buzz-desktop, full package) — 2611 passed.cargo fmt --check— clean.Two failures predate this change; both reproduce on unmodified
origin/main(verified by stashing this diff atf8692fa9b):managed_agents::global_config::tests::inherited_shared_compute_translates_to_supported_agent_transport— assertsBUZZ_AGENT_MODEL == "auto", gets"mesh".clippy -D warnings:items after a test moduleatmanaged_agents/restore.rs:528.Scope of evidence
This is a compile-and-test-level upgrade. It is not runtime-verified against a live Qwen3.8 mesh node from this branch. I have separately verified Qwen3.8 27B Q4_K_M serving correctly (real completions, hybrid recurrent, 65 blocks,
full_attention_interval4) on a source build at mesh-llmmain, but that was a mesh-llm binary, not this Buzz branch — someone should smoke a share-compute session before release.Related open micspiral PRs
fix(desktop): stop abandoning healthy mesh prefills at four minutes— relevant and likely wanted alongside this. Qwen3.8 27B is a larger first load than the current default, so the four-minute prefill abandonment is more likely to bite after this upgrade, not less. Touches onlyrelay_mesh.rs; no conflict with this PR.Mesh share-compute UX: sidebar card, topology— touches the same share-compute feature area; may need a rebase on whichever of these lands first.Includes #6115 (closed, folded in here)
Per Mic, #6115
fix(desktop): stop abandoning healthy mesh prefills at four minutesis closed and its content lands in this PR. Cherry-picked clean as a second commit, one file (managed_agents/relay_mesh.rs).That fix is a genuine prerequisite for the Qwen3.8 default rather than an unrelated ride-along: buzz-agent's
BUZZ_AGENT_LLM_TIMEOUT_SECSdefaults to 240 s while MeshLLM's frontend gives a backend call 600 s, and mesh requests arestream: false— so a cold prefill is one silent request. Measured on an M5 Max serving Qwen3.8-27B Q4_K_M: an 88,318-token cold prompt returns HTTP 200 after 503 s. At the 240 s default the client abandons it at four minutes and retries with an escalated budget (240 -> 480 -> 960), piling load onto a box already prefilling the first attempt. Since this PR makes that exact model the 64GB-class default, shipping the upgrade without the fix would make the timeout more likely to bite, not less. Mesh is seated at 660 s; remote providers keep 240 s.Combined-branch verification at
777426ba8: 2615 desktop tests pass (4 more than the upgrade commit alone — the fix's own tests),cargo fmt --checkclean, same single pre-existingglobal_configfailure noted above and nothing new.