[rcombs]

Another one caught during libass/libass#874 testing: I'd assumed that our transform quantization would never affect output bounding-box dimensions by more than a couple pixels, but it turns out that with some extreme X/Y rotations, you can get shifts of 4+px, and in some extreme cases, a whopping 100!

The main extreme-rotations file here demonstrates this with a few cases that simply build up a larger-than-usual amount of quantization error, while the perspective-clamp one primarily demonstrates the much larger impact of the FFMAX(v[2], 0.1) in ass_outline.c. VSFilter's corresponding clamp can also create substantial offsets like this, but in different situations than we do, since it does its math a little bit differently (AIUI it would have very little error on this particular sample); we could theoretically match it.

These tests aren't meant as "this is the correct rendering that we should persist indefinitely", but as documentation of our current behavior and sample cases we can revisit if we decide to improve this in the future.

[astiob]

VSFilter's corresponding clamp [is] different [from ours]; we could theoretically match it.

I asked for this when quantization was introduced; IIRC I was told it was not feasible. If it is, then I would very much prefer that it be done.

the first […] samples exhibit […] shifts of about 4.43 pixels […], causing the triangles visible in the output bitmaps to appear longer, with wider bases. These large deviations can likely also cause our output to appear visibly different from VSFilter's.

VSFilter’s triangles are dramatically shorter, essentially truncated, but don’t seem obviously distorted relative to libass’s (although maybe they are if you extend the edges infinitely):
https://slow.pics/c/JAptcIOJ

Is this the same clamp at work again? Can you produce a sample that’s unaffected by the clamp so we can compare to VSFilter directly?

If this is not the clamp’s fault, then this sounds like a clear-cut bug to me, so I’d rather this be an issue on the main repo.

[MrSmile]

There are two separate perspective clamps:

• One in quantization, which can be regulated by MAX_PERSP_SCALE (larger values better). Theoretically it should be possible to eliminate this error if somebody invents a clever quantization scheme that works for extreme Z-range.
• One in final transform, which distorts perspective and not invariant under tesselation. I recall that it was dependent on screen resolution too. It's possible to implement ideal perspective transformation using rational splines and some clever clipping scheme, but VSFilter doesn't do that.

First can be definitely improved, while the second is broken in VSFilter. I'm not sure it's worth (and possible) to exactly emulate its bugs in this case.

[rcombs]

Had Codex do some experimenting and it came up with a mechanism that matches vsfilter's clamp behavior and gets all of these cases down to sub-pixel quantization error. It's still technically possible for some very extreme cases to be outright rejected where vsfilter would draw output, but it also makes that far less likely than on current master. libass/libass@master...rcombs:libass:codex/perspective-quantization

I'm not willing to PR this, since I don't properly understand the math (either the existing version or the codex experiment), so this is just a proof-of-concept.

[MrSmile]

Looks like it's exactly VSFilter bug emulation: tweak final clamping to match VSFilter and then quantize clamped transform. I'm against emulating this bug as it's hard to conceive situation where it'd used meaningfully and resulting code is a mess. I think that trying to emulate ideal perspective transform is better even at the cost of VSFilter divergence.