Skip to content

bugfix(dx8): Use the display format for the fullscreen back buffer when a device reset does not last - #3656

Open
coconutbird wants to merge 2 commits into
TheSuperHackers:mainfrom
coconutbird:fix/fullscreen-device-lost
Open

coconutbird wants to merge 2 commits into
TheSuperHackers:mainfrom
coconutbird:fix/fullscreen-device-lost

Conversation

@coconutbird

@coconutbird coconutbird commented Nov 5, 2026 •

Copy link
Copy Markdown

Needs feedback: the design choice under Alternative, and the untested cases listed under Testing.

Problem

On some systems the exclusive fullscreen game never presents a frame. It resets the device in an endless loop and the screen keeps switching display modes, which makes the whole PC unusable until the game is killed.

Reproduced with weekly-2026-10-30 and current main on Windows 11 10.0.26200, RTX 5070 (driver 32.0.16.1742), two 2560x1440 @ 144 Hz monitors on the same GPU, at 800x600 and at 2560x1440. The retail Steam game.dat also fails to start in fullscreen on this system.

A trace of an instrumented main build at 800x600:

  • Every Reset() returns D3D_OK: 66 resets in about 30 seconds.
  • Every Reset() sends WM_DISPLAYCHANGE, although the display already is in 800x600 @ 144.
  • The next TestCooperativeLevel() returns D3DERR_DEVICENOTRESET before a frame is presented. 0 frames are presented.
  • Waiting 2 seconds between resets and the DISABLEDXMAXIMIZEDWINDOWEDMODE compatibility layer do not change this.

Cause

DX8Wrapper::Find_Color_And_Z_Mode promotes the fullscreen back buffer to D3DFMT_A8R8G8B8, while the display mode is D3DFMT_X8R8G8B8.

In d3d8.dll (SysWOW64, PDB 1680DBE07E5EC781E2FA8676D9ED4B91 age 1, Microsoft public symbols), CSwapChain::Reset checks whether the display already is in the requested mode by comparing the back buffer format as passed in with the current display format:

0x10025695  mov  eax, [ebx+3Ch]    ; BackBufferFormat from D3DPRESENT_PARAMETERS
0x10025698  cmp  eax, [edx+11Ch]   ; current display format, as returned by GetDisplayMode
0x1002569e  jnz  mode_change

The mode change path strips the alpha channel for its mode lookup (CPixel::SuppressAlphaChannel), then calls DdSetModeLH -> D3D8SetMode -> ChangeDisplaySettingsExA and waits up to 3 seconds for DWM. So with an A8R8G8B8 back buffer every fullscreen Reset() changes the display mode, even when the display is already in that mode.

On the reproducing system a display mode change loses the device shortly afterwards, even while the game is in the foreground. A minimal D3D8 test app that only creates a fullscreen device and presents frames shows the difference. Its two runs differ only in the back buffer format:

Back buffer Resets Frames presented in 12 s
D3DFMT_X8R8G8B8 1 42788
D3DFMT_A8R8G8B8 20 (still resetting) 2

With X8R8G8B8 the device is lost once after CreateDevice, and one Reset() without a display mode change restores it. With A8R8G8B8 every Reset() changes the display mode again, which loses the device again.

Change

When the previous fullscreen Reset() did not last until a frame was presented, DX8Wrapper::Reset_Device switches the back buffer to the display format, picks a matching depth stencil format and checks the MSAA mode again. If no depth stencil format matches the display format, it keeps the back buffer format and changes nothing. Reset() then restores the device without a display mode change. Find_Color_And_Z_Mode keeps the display format for the rest of the session, so a later resolution change does not bring the loop back. A presented frame clears the counter, so systems where Reset() works keep the A8R8G8B8 back buffer. The code is shared by Generals and Zero Hour.

Result on the reproducing system: 2 resets at startup, then the game presents frames normally at 800x600 and at 2560x1440.

Tradeoffs

  • After the switch, the shoreline water effect is disabled for the session, as on hardware without destination alpha. renderShoreLinesSorted writes the water depth into the alpha channel of the back buffer, and W3DWater blends the water with D3DBLEND_DESTALPHA. Without it, shallow water at the shore is opaque instead of transparent, and the edge of the water mesh shows above the surf. See the screenshots below.
  • Affected systems still do one display mode changing reset per launch before the switch.
  • A false positive is possible when the device is lost again before the first frame after a reset, for example when alt-tabbing away right after returning. It costs the shoreline effect for that session.
  • Systems without the issue still call ChangeDisplaySettingsExA, and may wait for DWM, on every fullscreen reset, for example when alt-tabbing back into the game. This may be part of the alt-tab delay mentioned in [GEN][ZH] Fix Alt-tabbing crash #846. Not measured.

Screenshots

Zero Hour main menu, windowed at 2560x1440 so that the reset loop does not occur. Both runs use the same build. The right run forces the X8R8G8B8 back buffer that the switch selects, with a local test change that is not part of this PR.

A8R8G8B8 vs X8R8G8B8 back buffer

Shoreline close-up, A8R8G8B8 vs X8R8G8B8 back buffer

Alternative

Never promote the back buffer to A8R8G8B8 in fullscreen. This is deterministic, needs no extra reset, and removes the display mode change from every fullscreen reset for everyone, but it removes the shoreline water transparency in fullscreen for everyone. Feedback on which approach is preferred is welcome.

A follow-up could make the shoreline transparency independent of destination alpha, for example by storing the water depth alpha per map in a texture that the water pass samples. The fullscreen back buffer could then always use the display format without losing the effect.

Testing

Done:

  • Zero Hour Release build (win32 preset with RTS_DEBUG_LOGGING), fullscreen 800x600 and 2560x1440 on the reproducing system: the reset loop is gone and frames are presented.
  • g_generals and z_generals build.

Not done yet:

  • Alt-tab and resolution change after the switch.
  • MSAA enabled.
  • Generals in fullscreen.
  • A system without the issue. The switch is not expected to trigger there.

Relation to #3450

#3450 fixes a different failure, where Reset() fails because resources are still bound. Here Reset() succeeds every time. Both change the Reset() call in DX8Wrapper::Reset_Device, so whichever is merged second needs a small rebase.

AI Disclosure

Claude Opus 5.5 wrote this PR entirely: the code, the investigation and test tooling, and this description. I directed the work.

…en a device reset does not last

In fullscreen the back buffer is promoted to D3DFMT_A8R8G8B8 while the display mode is D3DFMT_X8R8G8B8.
Because the formats differ, every IDirect3DDevice8::Reset() sets the display mode again. On systems where
a display mode change loses the device (seen on Windows 11 with multiple monitors and GPUs), the device is
lost again by its own Reset() before a single frame is presented, so the game resets forever while the
screen keeps switching modes.

When the previous fullscreen Reset() did not last until a frame was presented, Reset_Device now switches the
back buffer to the display format, which lets Reset() restore the device without a display mode change.
Destination alpha effects such as the soft water edge are then unavailable for the rest of the session.
@coderabbitai

coderabbitai Bot commented Nov 5, 2026 •

Copy link
Copy Markdown

Explain Change →

Walkthrough

DX8Wrapper tracks fullscreen resets that occur without a successful present. Before a subsequent fullscreen reset, it can switch the back buffer to the display format and adjust the depth and MSAA settings.

Changes

Fullscreen Reset Handling

Layer / File(s) Summary
Display-format fallback
Core/Libraries/Source/WWVegas/WW3D2/dx8wrapper.h, Core/Libraries/Source/WWVegas/WW3D2/dx8wrapper.cpp
DX8Wrapper adds state and a helper for switching the fullscreen back buffer to the display format. The helper updates the depth format when available and disables MSAA if the selected formats do not support it. Destination-alpha promotion is skipped when this fallback is active.
Reset and present tracking
Core/Libraries/Source/WWVegas/WW3D2/dx8wrapper.cpp
When a fullscreen reset follows an unpresented reset and the back-buffer format differs from the display format, DX8Wrapper selects the fallback before resetting. Successful fullscreen resets increment the counter; successful presents in End_Scene and Flip_To_Primary clear it.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Change: Bug fix · Severity of issue fixed: Medium

Possibly related issues

Possibly related PRs

Suggested labels: Bug, Fix, Stability, Platform, Gen, ZH

Suggested reviewers: xezon

Merge Risk: Minimal · up to da950

The previously reported reset failure path is fixed. No actionable merge-blocking issue is established.

Pre-merge checks: Passed 4

Check name Status Explanation
Title check Passed The title describes the fullscreen back-buffer fallback for a reset that does not last until a frame is presented.
Description check Passed The description explains the reset loop, the back-buffer fallback, its tradeoffs, and the reported tests.
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.

Tip

🎯 Customize your risk assessments

Guide how CodeRabbit assesses merge risk and security architecture risk. Highlight sensitive changes and explain your security boundaries using Merge Risk Instructions and Security Risk Instructions in your review settings. Security architecture risk assessments require the Advanced plan or above.

✅ Automatic approvals, with your rules

Let CodeRabbit approve pull requests once required threads are resolved, pre-merge checks pass, and your risk limits are met. Enable Automatic approvals in review settings, with overrides for teams or file paths.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@greptile-apps

greptile-apps Bot commented Nov 5, 2026 •

Copy link
Copy Markdown

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] The latest change appears safe to merge; a failed depth-format search leaves the original buffer settings intact.

Summary

The latest change checks for a matching depth buffer before changing the fullscreen back buffer.

  • Fullscreen resets switch to the display format when no frame appears.

Reviews (2) · Last reviewed commit: "bugfix(dx8): Keep the back buffer format..." · Reviewed by Greptile

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

Follow-up findings are listed below so you can review them separately from inline comments.

Follow-up (1)

Minor (1)

Minor · If Find_Z_Mode fails, the helper retains the old… · dx8wrapper.cpp:1652-1656

Core/Libraries/Source/WWVegas/WW3D2/dx8wrapper.cpp:1652-1656
Minor · Potential issue · Low value

If Find_Z_Mode fails, the helper retains the old AutoDepthStencilFormat despite already changing BackBufferFormat and committing the fallback flag. Reset is then attempted with an unvalidated depth/back-buffer combination, with no failure propagation or rollback.

Reset can fail repeatedly with the newly committed presentation parameters instead of preserving a usable original configuration or reporting that fallback selection failed.

This can occur when: The original alpha-capable back buffer has a usable depth format, but no tested depth format matches the display-format back buffer on the active device.

Review info

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: f84be70f-7df8-4ede-b06d-9a2850093225
Commits

Reviewing files that changed from the base of the PR and between 116fbcf and 94a12ec.

Files selected for processing: 2

  • Core/Libraries/Source/WWVegas/WW3D2/dx8wrapper.cpp
  • Core/Libraries/Source/WWVegas/WW3D2/dx8wrapper.h

Included review availability: This review used your included allowance. Your plan provides up to 4 included reviews per hour; 3 remain after this review.

@Skyaero42

Copy link
Copy Markdown

This PR lacks AI disclosure

@coconutbird

Copy link
Copy Markdown
Author

This PR lacks AI disclosure

@Skyaero42 Thanks, added an AI Disclosure section to the description.

@coconutbird

Copy link
Copy Markdown
Author

@coderabbitai Fixed in da9500c. Use_Display_Format_For_Fullscreen_Back_Buffer now looks up the depth buffer format first. If no format matches the display format, it keeps the presentation parameters unchanged and does not set the fallback flag.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Additional comments: 1

Core/Libraries/Source/WWVegas/WW3D2/dx8wrapper.cpp: 1

1649-1658: Trivial · Maintainability & Code Quality · Low value

Unverified finding
Verification ran but could not confirm this finding. It is shown for review, not as a verified issue.

Verify that the fallback state stays consistent after a failed Reset() or a later Set_Render_Device.

Use_Display_Format_For_Fullscreen_Back_Buffer sets FullscreenBackBufferUsesDisplayFormat and modifies _PresentParameters before Reset() is called. If Reset() then fails, the flag stays set, and the next Reset_Device call has BackBufferFormat == DisplayFormat, so the helper is not called again. That behavior is acceptable.

A separate case: the flag is never cleared. If the user later switches to windowed mode and back to fullscreen, Find_Color_And_Z_Mode still skips the A8R8G8B8 promotion. This matches the stated "rest of the session" tradeoff.

The Find_Z_Mode call at Line 1649 uses DisplayFormat. In fullscreen, Find_Color_And_Z_Mode sets DisplayFormat from the chosen mode format, so the value is valid. No defect was found. Confirm that this "rest of the session" behavior is intended, and document it in the header.

Review info

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: d05cf658-9767-45ce-aa82-3e1e517f6d11
Commits

Reviewing files that changed from the base of the PR and between 94a12ec and da9500c.

Files selected for processing: 1

  • Core/Libraries/Source/WWVegas/WW3D2/dx8wrapper.cpp

Included review availability: This review used your included allowance. Your plan provides up to 4 included reviews per hour; 2 remain after this review.

@coderabbitai

coderabbitai Bot commented Nov 6, 2026

Copy link
Copy Markdown

Tip

For best results, initiate chat on the files or code changes.

Analysis chain

Script executed:

#!/bin/bash
set -u
file='Core/Libraries/Source/WWVegas/WW3D2/dx8wrapper.cpp'
printf '%s\n' '--- Current revision ---'
git log -1 --format='%h %s'
printf '%s\n' '--- Function outline ---'
ast-grep outline "$file" --match 'Use_Display_Format_For_Fullscreen_Back_Buffer|Find.*Depth' --view expanded
printf '%s\n' '--- Helper and call site context ---'
rg -n -C 45 'Use_Display_Format_For_Fullscreen_Back_Buffer' "$file"
printf '%s\n' '--- Reported commit, targeted diff ---'
git show --format=fuller --no-ext-diff --no-renames da9500c -- "$file"

Length of output: 9463


@coconutbird Thanks! I checked da9500c. If Find_Z_Mode fails, the helper returns before changing the presentation parameters or setting the fallback flag.

I did not run tests.


If you found this review helpful, would you consider giving us a shout-out on X?

Thank you for using CodeRabbit!

You are interacting with an AI system.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants