[Performance]Reduce full names snapshot rebuild cost on HMS events

[wenzhenghu]

Background

This issue is a follow-up for PR #65126.

PR #65126 refactors the external metadata names cache to use immutable snapshots with copy-on-write style updates. That change was introduced for correctness rather than performance.

The main goal of this refactor is to make name-cache publication and invalidation semantics safe under concurrent loads and incremental HMS events. In particular, the PR needs a names-cache representation that can:

• publish a consistent snapshot to readers
• avoid in-place mutation of shared cache state
• cooperate with generation-based invalidation / stale-load fencing
• prevent lagging async/manual loads from overwriting newer HMS event updates

Using immutable snapshots helps make these correctness guarantees explicit and avoids readers observing partially updated name/index state during concurrent cache refresh or event handling.

At the same time, this refactor also changes the HMS event update path from the previous incremental-style update to a full snapshot rebuild in some hot-name-cache cases. As a result, it introduces a real performance regression on HMS CREATE/DROP event handling for very large metadata sets.

This follow-up is only relevant if PR #65126 is merged.

Problem

On HMS table/database name events, the current update path may rebuild the full names snapshot for a single name change. In large external catalogs, one CREATE or DROP event may:

• scan the full names list
• copy the full list
• recreate existing entries
• rebuild derived indexes and maps

This makes a single HMS event cost O(n) in the size of the hot names snapshot.

Impact

This is a performance regression introduced by PR #65126, not a correctness blocker.

The impact is most visible on the HMS event path with very large metadata sets. In normal usage, CREATE/DROP is usually not a high-frequency operation, and most users will not have extremely large numbers of databases or tables under a single catalog/database. The problem is therefore concentrated in large-scale HMS event scenarios.

Also, this copy-on-write overhead has already been discussed in earlier review rounds, and parts of the PR series have already reduced the impact on more frequent execution paths. The remaining concern is mainly the HMS event path.

Why this is tracked separately

By the current review stage of PR #65126, the remaining issues are already being narrowed down progressively. Fixing this regression properly likely requires a broader redesign of the names-cache update/publication strategy, which would significantly expand scope and increase late-stage rework risk.

So instead of mixing that architectural change into the current PR, this issue tracks the follow-up performance fix after PR #65126 is merged, or together with other known HMS event follow-up work.

Candidate directions

Possible directions under consideration:

1. Stripe/shard the names cache

   • split the names snapshot/index into multiple stripes
   • limit copy/rebuild work to the affected subset for each event
   • preserve the current snapshot visibility guarantees

2. Control deep-copy vs reference reuse

   • add an internal mechanism to choose whether immutable entries are deep-copied or directly reused
   • reduce redundant object recreation and index rebuild work
   • keep publication safety and immutability guarantees explicit

These are candidate ideas only and still need design validation.

Follow-up expectations

The follow-up should:

• reduce the O(n) rebuild cost for ordinary single-name HMS events
• preserve concurrency and snapshot correctness
• include targeted benchmark or regression coverage for large hot snapshots and event bursts
• clearly document the invariants of the final cache design