SOLR-18315: Optimize UTF conversions by using standard Java APIs

[psalagnac]

https://issues.apache.org/jira/browse/SOLR-18315

Description

This replaces class ByteUtils by Java API calls, mostly String constructor and getBytes() method when doing short-length byte[] <-> String conversions, or using CharsetEncoder for longer data.

Tests

I validated the perf improvement with RequestWriters benchmark (note it just measures serialization of update requests, whithout processing the requests).

I get a 8.5x improvement in throughput.

### BASELINE
Benchmark                   (batchSize)   (type)   Mode  Cnt     Score     Error  Units
RequestWriters.writeUpdate           10  javabin  thrpt    5  5660,213 ± 277,626  ops/s
RequestWriters.writeUpdate          100  javabin  thrpt    5   572,390 ±   3,854  ops/s
RequestWriters.writeUpdate         1000  javabin  thrpt    5    57,397 ±   0,270  ops/s
RequestWriters.writeUpdate        10000  javabin  thrpt    5     5,704 ±   0,039  ops/s

### AFTER
Benchmark                   (batchSize)   (type)   Mode  Cnt      Score      Error  Units
RequestWriters.writeUpdate           10  javabin  thrpt    5  49797,001 ± 2110,121  ops/s
RequestWriters.writeUpdate          100  javabin  thrpt    5   5245,920 ±  143,421  ops/s
RequestWriters.writeUpdate         1000  javabin  thrpt    5    514,228 ±   15,048  ops/s
RequestWriters.writeUpdate        10000  javabin  thrpt    5     48,803 ±    0,875  ops/s

[epugh]

Bug fix here? Nice to see the benchmarks get used and therefore finding oversights like this.

[psalagnac]

Yes, bug fix. I added this benchmark a while ago and I assume I forgot to change this name before merging. If I recall well, the class was renamed while I was working on this.

[epugh]

cool! thanks for responding.

[dsmiley]

I reviewed the code. I had suspicions the historic developments that lead to it's current state had insights that the change here reverts. I had Claude review and it came up with the following report:

---

Reviewed the write-side changes in JavaBinCodec.writeStr — found what looks like a correctness regression in the large-string (>64KB) path.

Potential infinite loop on malformed UTF-16 input

The double-pass encode at JavaBinCodec.writeStr (the streaming branch for strings whose worst-case UTF-8 size exceeds MAX_UTF8_SIZE_FOR_ARRAY_GROW_STRATEGY) calls the low-level CharsetEncoder.encode(CharBuffer, ByteBuffer, boolean) but never inspects the returned CoderResult:

while (!endOfInput) {
  endOfInput = !cb.hasRemaining();
  scratch.clear();
  encoder.encode(cb, scratch, endOfInput); // result discarded
  sz += scratch.position();
}

The default CodingErrorAction for a CharsetEncoder is REPORT. On malformed input (e.g. an unpaired surrogate), REPORT returns a MALFORMED result without consuming the offending char — so cb's position never advances, cb.hasRemaining() stays true forever, and this spins indefinitely (verified locally: a 30k-char string with one unpaired surrogate at index 15000 loops forever, stuck at the same position every iteration).

The deleted ByteUtils.writeUTF16toUTF8/UTF16toUTF8 explicitly detected unpaired/out-of-order surrogates and substituted U+FFFD, which guaranteed forward progress. That guard is gone here, so any large field value containing an unpaired surrogate (corrupted imports, non-validating JSON/XML producing a lone \uD800-range char, etc.) will hang the thread serializing it via javabin — this reads as a DoS risk, not just a perf nit.

testLongString (added in this PR) uses TestUtil.randomUnicodeString, which never produces unpaired surrogates, so it wouldn't catch this. Might be worth adding a case with a deliberately malformed surrogate to cover it.

Suggested fix: check the CoderResult and handle isMalformed()/isUnmappable() explicitly (e.g. write a replacement char and advance past the bad input), mirroring what the old ByteUtils code did.

Two smaller, lower-confidence points:

• The small-string branch now allocates a fresh byte[] on every writeStr call (s.toString().getBytes(StandardCharsets.UTF_8)) instead of reusing the pooled bytes field the surrounding code maintains via getBufferSize. That buffer is still reused on the read path and in the >64KB write path, just not here. Might be a net win from the JIT intrinsic, but given this is the hot path for the overwhelming majority of field writes, a quick JMH comparison against the old buffer-reuse approach would be good to confirm it's actually a net improvement rather than trading CPU for extra allocation/GC pressure.
• Also noting String.getBytes(UTF_8)'s default replacement for an unpaired surrogate is a single 0x3F ('?'), vs. the old code's 3-byte U+FFFD (EF BF BD). Minor, silent wire-format behavior change for malformed data, probably fine to ignore unless someone depends on it.

---

IMO the allocation is probably a non-issue as I assume it'd be stack-allocated -- the JVM is very smart