D3DERR_DEVICENOTRESET on mode/res change but OK on alt-tab??

[OceanMachine]

Hello again!

[Edit: Solved]

Thanks in advance to anyone who takes the time to read this, and anyone who has helped me get this far :) I am pretty new when it comes to C++ and CEGUI, but I have embarked upon a little project to see if I can get a working D3D9 overlay using CEGUI onto an existing game...

OK, so I am pretty much "there" with regard to my project. I am hooking the D3D9 calls from the game, using EndScene() to draw the CEGUI GUI over the top of the existing 3D scene. This is all working fine, including a successful reset of the device and the GUI on alt-tab or when the device is lost. This is all working great.

My problems start when I either attempt to change from Windowed mode to Fullscreen (or vice versa), or change resolution in the game. What happens is I get left with "D3DERR_DEVICENOTRESET" after doing my preD3DReset(), and the device never actually resets. I suspect this is due to resources that have not been freed, but why would this work on alt-tab but not on size or mode change? I am not sure how the *** I can debug the Dll that is injected into another process to find out exactly WHY the reset is failing.

Note: I am never actually calling Reset() on the device myself, I am letting the game do all that kind of stuff - I am just detecting when the device is lost or needing a reset [using TestCooperativeLevel()], and removing created resources using CEGUI::Direct3D9Renderer's preD3DReset() method. I then wait for TestCooperativeLevel() to come back with "D3D_OK" before I call postD3DReset().

For Alt-tab (i.e. device lost) and then Alt-tab back to the game again (i.e. device gets reset by the game and I get D3D_OK again), this is all fine, but not for resolution changes or changes between Fullscreen <-> Windowed mode. In the latter case(s) I just keep getting a result of D3DERR_DEVICENOTRESET over and over again, and I never get the D3D_OK. :(

The device resets fine on both alt-tab and resolution/mode change when I am hooking the D3D calls or not, the only thing that seems to break it is when I render something with CEGUI.

Does anyone have any idea what might be causing the issue?

I am not doing anything particularly fancy, and I am using a stateblock to save the state of the device before calling renderGUI() ( either in EndScene() or Present() ) and then applying it and releasing it straight away, so hopefully this shouldn't be the issue. When I comment out the state block creation call, the same thing happens.

In fact, when I leave everything in my code apart from System::getSingleton().setGUISheet( sheet ) then it works, so it seems to be something I am forgetting to do with CEGUI itself, hence this post. It also works fine if I don't call renderGUI() but still set the GUISheet().

Here is the code I am using:
The "interesting stuff" happens in the first file, where MyGUIRender() [which calls renderGUI() but also initialises the GUI by calling MyGUIInit()] is called from EndScene().
I haven't included absolutely everything, as I have tried to only include things that seem remotely relevant, but let me know if there is something I have not included that you need to know about.

[Edit:] Managed to get a bit of info by finally getting the Debug version of D3D working, and I seem to get the following error on mode change:
[6440] Direct3D9: (ERROR) :All user created D3DPOOL_DEFAULT surfaces must be freed before ResetEx can succeed. ResetEx Fails.
[6440] Direct3D9: (ERROR) :ResetEx failed and ResetEx/TestCooperativeLevel/Release are the only legal APIs to be called subsequently
How can I make sure all surfaces are freed? Is this something CEGUI normally takes care of? Can I tell whether it is CEGUI that is not freeing something (i.e. me not freeing a CEGUI resource)? Do I need to notify CEGUI on WM_DISPLAYCHANGE or something like that?

MyDirect3DDevice9.h:

Code: Select all

#pragma once

class MyDirect3DDevice9 : public IDirect3DDevice9
{
public:
   // We need d3d so that we'd use a pointer to MyDirect3D9 instead of the original IDirect3D9 implementor
   // in functions like GetDirect3D9
   MyDirect3DDevice9(IDirect3D9* d3d, IDirect3DDevice9* device) : m_d3d(d3d), m_device(device), d_lastFrameTime(GetTickCount())
   {
   }

   bool MyGUIInit()
   {
      using namespace CEGUI;
      try
      {
         new CEGUI::DefaultLogger();
         CEGUI::DefaultLogger::getSingleton().setLoggingLevel( CEGUI::Informative );
         CEGUI::DefaultLogger::getSingleton().setLogFilename( "f:/cegui.log" );

         pimpl->d_3DDevice = m_device;
         pimpl->d_D3D = m_d3d;
         pimpl->d_processId = GetCurrentProcessId();

         OldWindowProc = (WNDPROC)SetWindowLong( pimpl->d_window, GWL_WNDPROC, (long)Win32AppHelper::wndProc );

         pimpl->d_renderer = &CEGUI::Direct3D9Renderer::bootstrapSystem( m_device );

         // initialise the required dirs for the DefaultResourceProvider
         CEGUI::DefaultResourceProvider* rp = static_cast<CEGUI::DefaultResourceProvider*>(CEGUI::System::getSingleton().getResourceProvider());
         rp->setResourceGroupDirectory( "datafiles", "f:/datafiles/" );

         // set the default resource groups to be used
         CEGUI::Imageset::setDefaultResourceGroup("datafiles");
         CEGUI::Font::setDefaultResourceGroup("datafiles");
         CEGUI::Scheme::setDefaultResourceGroup("datafiles");
         CEGUI::WidgetLookManager::setDefaultResourceGroup("datafiles");
         CEGUI::WindowManager::setDefaultResourceGroup("datafiles");
         CEGUI::ScriptModule::setDefaultResourceGroup("datafiles");

         // setup default group for validation schemas
         CEGUI::XMLParser* parser = CEGUI::System::getSingleton().getXMLParser();
         if ( parser->isPropertyPresent( "SchemaDefaultResourceGroup" ) )
            parser->setProperty( "SchemaDefaultResourceGroup", "schemas" );

         CEGUI::SchemeManager::getSingleton().create( "VanillaSkin.scheme" );
         CEGUI::WindowManager& winMgr = CEGUI::WindowManager::getSingleton();
         
         CEGUI::Window *sheet;
         sheet = winMgr.loadWindowLayout( "MyApp_test.layout" );
         System::getSingleton().setGUISheet( sheet ); // <===========================## ok when this is commented out ##
      }
      catch (CEGUI::Exception e)
      {
         odprintf( "MyGUIInit() -> bootstrapSystem and Init() FAILED: Error Message: %s", e.what() );
         return false;
      }

      return true;
   }

   bool MyGUIRender()
   {
      static bool WaitingForReset = false;
      static bool MyGUIInitAttempted = false;
      HRESULT coop;

      if ( !MyGUIInitAttempted )
      {
         MyGUIInitAttempted = true;

         if ( !MyGUIInit() )
         {
            MessageBeep( MB_ICONERROR );
            return false;
         }
      }

      CEGUI::System& guiSystem = CEGUI::System::getSingleton(); // #### must be done *after* MyGUIInit() ####
        DWORD thisTime = GetTickCount();
        float elapsed = static_cast<float>(thisTime - d_lastFrameTime);
        d_lastFrameTime = thisTime;

        guiSystem.injectTimePulse( elapsed / 1000.0f );

      coop = pimpl->d_3DDevice->TestCooperativeLevel();
      if( coop == D3DERR_DEVICELOST || coop == D3DERR_DEVICENOTRESET )
      {
         if ( !WaitingForReset )
         {
            WaitingForReset = true;
            pimpl->d_renderer->preD3DReset();
         }
         return true;
      }
      else if ( coop == D3D_OK )
      {
         if ( WaitingForReset )
         {
            WaitingForReset = false;
            pimpl->d_renderer->postD3DReset();
            return true;
         }
      }

      try
      {
         IDirect3DStateBlock9* pStateBlock = NULL;

         m_device->CreateStateBlock( D3DSBT_ALL, &pStateBlock );
         guiSystem.renderGUI();

         if ( pStateBlock )
         {
            pStateBlock->Apply();
            pStateBlock->Release();
            return true;
         }
      }
      catch ( CEGUI::Exception e )
      {
         odprintf( "MyGUIRender() -> renderGUI() FAILED: Error message: %s", e.what() );
      }

      return false;
   }

   STDMETHOD(EndScene)(THIS)
   {
      MyGUIRender();
      return m_device->EndScene();
   }

   /*** IUnknown methods ***/
   STDMETHOD(QueryInterface)(THIS_ REFIID riid, void** ppvObj)
   {
      return m_device->QueryInterface(riid, ppvObj);
   }

   STDMETHOD_(ULONG,AddRef)(THIS)
   {
      return m_device->AddRef();
   }

   STDMETHOD_(ULONG,Release)(THIS)
   {
      ULONG count = m_device->Release();

      if( count == 0)
         delete this;

      return count;
   }

   /*** IDirect3DDevice9 methods ***/
   STDMETHOD(Present)(THIS_ CONST RECT* pSourceRect,CONST RECT* pDestRect,HWND hDestWindowOverride,CONST RGNDATA* pDirtyRegion)
   {
      return m_device->Present( pSourceRect, pDestRect, hDestWindowOverride, pDirtyRegion );
   }

   STDMETHOD(GetDirect3D)(THIS_ IDirect3D9** ppD3D9)
   {
      // Let the device validate the incoming pointer for us
      HRESULT hr = m_device->GetDirect3D(ppD3D9);
      if(SUCCEEDED(hr))
         *ppD3D9 = m_d3d;

      return hr;   
   }

   STDMETHOD(GetCreationParameters)(THIS_ D3DDEVICE_CREATION_PARAMETERS *pParameters)
   {
      pimpl->d_window = 0;

      HRESULT hr = m_device->GetCreationParameters(pParameters);
      if(SUCCEEDED(hr))
         pimpl->d_window = pParameters->hFocusWindow;

      return hr;
   }

   STDMETHOD(Reset)(THIS_ D3DPRESENT_PARAMETERS* pPresentationParameters)
   {
      HRESULT hr = m_device->Reset(pPresentationParameters);

      if(SUCCEEDED(hr))
      {
         odprintf( "Reset() -> Reset OK" );
         odprintf( "Reset() -> pPresentationParameters->AutoDepthStencilFormat: %d", pPresentationParameters->AutoDepthStencilFormat );
         odprintf( "Reset() -> pPresentationParameters->BackBufferCount: %d", pPresentationParameters->BackBufferCount );
         odprintf( "Reset() -> pPresentationParameters->BackBufferFormat: %d", pPresentationParameters->BackBufferFormat );
         odprintf( "Reset() -> pPresentationParameters->BackBufferHeight: %d", pPresentationParameters->BackBufferHeight );
         odprintf( "Reset() -> pPresentationParameters->BackBufferWidth: %d", pPresentationParameters->BackBufferWidth );
         odprintf( "Reset() -> pPresentationParameters->EnableAutoDepthStencil: %d", pPresentationParameters->EnableAutoDepthStencil );
         odprintf( "Reset() -> pPresentationParameters->Flags: %x", pPresentationParameters->Flags );
         odprintf( "Reset() -> pPresentationParameters->FullScreen_RefreshRateInHz: %d", pPresentationParameters->FullScreen_RefreshRateInHz );
         odprintf( "Reset() -> pPresentationParameters->hDeviceWindow: %d", pPresentationParameters->hDeviceWindow );
         odprintf( "Reset() -> pPresentationParameters->MultiSampleQuality: %d", pPresentationParameters->MultiSampleQuality );
         odprintf( "Reset() -> pPresentationParameters->MultiSampleType: %d", pPresentationParameters->MultiSampleType );
         odprintf( "Reset() -> pPresentationParameters->PresentationInterval: %d", pPresentationParameters->PresentationInterval );
         odprintf( "Reset() -> pPresentationParameters->SwapEffect: %d", pPresentationParameters->SwapEffect );
         odprintf( "Reset() -> pPresentationParameters->Windowed: %d", pPresentationParameters->Windowed );
      }
      else
      {
         switch ( hr )
         {
            case D3DERR_INVALIDCALL:
               odprintf( "Reset() -> Failed to Reset, result: %d (D3DERR_INVALIDCALL)", hr );
               break;
            case D3DERR_DEVICELOST:
               odprintf( "Reset() -> Failed to Reset, result: %d (D3DERR_DEVICELOST)", hr );
               break;
            case D3DERR_NOTAVAILABLE:
               odprintf( "Reset() -> Failed to Reset, result: %d (D3DERR_NOTAVAILABLE)", hr );
               break;
            case D3DERR_OUTOFVIDEOMEMORY:
               odprintf( "Reset() -> Failed to Reset, result: %d (D3DERR_NOTAVAILABLE)", hr );
               break;
            default:
               odprintf( "Reset() -> Failed to Reset, result: %d (UNKNOWN ERROR)", hr );
               break;
         }

         odprintf( "Reset() -> pPresentationParameters->AutoDepthStencilFormat: %d", pPresentationParameters->AutoDepthStencilFormat );
         odprintf( "Reset() -> pPresentationParameters->BackBufferCount: %d", pPresentationParameters->BackBufferCount );
         odprintf( "Reset() -> pPresentationParameters->BackBufferFormat: %d", pPresentationParameters->BackBufferFormat );
         odprintf( "Reset() -> pPresentationParameters->BackBufferHeight: %d", pPresentationParameters->BackBufferHeight );
         odprintf( "Reset() -> pPresentationParameters->BackBufferWidth: %d", pPresentationParameters->BackBufferWidth );
         odprintf( "Reset() -> pPresentationParameters->EnableAutoDepthStencil: %d", pPresentationParameters->EnableAutoDepthStencil );
         odprintf( "Reset() -> pPresentationParameters->Flags: %x", pPresentationParameters->Flags );
         odprintf( "Reset() -> pPresentationParameters->FullScreen_RefreshRateInHz: %d", pPresentationParameters->FullScreen_RefreshRateInHz );
         odprintf( "Reset() -> pPresentationParameters->hDeviceWindow: %d", pPresentationParameters->hDeviceWindow );
         odprintf( "Reset() -> pPresentationParameters->MultiSampleQuality: %d", pPresentationParameters->MultiSampleQuality );
         odprintf( "Reset() -> pPresentationParameters->MultiSampleType: %d", pPresentationParameters->MultiSampleType );
         odprintf( "Reset() -> pPresentationParameters->PresentationInterval: %d", pPresentationParameters->PresentationInterval );
         odprintf( "Reset() -> pPresentationParameters->SwapEffect: %d", pPresentationParameters->SwapEffect );
         odprintf( "Reset() -> pPresentationParameters->Windowed: %d", pPresentationParameters->Windowed );

      }

      return hr;
   }

   STDMETHOD_(void, SetCursorPosition)(THIS_ int X,int Y,DWORD Flags)
   {
      m_device->SetCursorPosition(X, Y, Flags);
   }


/* and loads more IDirect3DDevice9 methods ...
   ...
   ...
   ... */

private:
   IDirect3DDevice9* m_device;   
   IDirect3D9* m_d3d;
   DWORD d_lastFrameTime;
};

MyDirect3D9.h:

Code: Select all

#pragma once

#include "MyDirect3DDevice9.h"

class MyDirect3D9 : public IDirect3D9
{
public:
   MyDirect3D9(IDirect3D9* d3d) : m_d3d(d3d)
   {
   }

   /*** IUnknown methods ***/
    HRESULT STDMETHODCALLTYPE QueryInterface(REFIID riid, void** ppvObj)
   {
      return m_d3d->QueryInterface(riid, ppvObj);
   }

    ULONG STDMETHODCALLTYPE AddRef()
   {
      return m_d3d->AddRef();
   }

    ULONG STDMETHODCALLTYPE Release()
   {
      ULONG count = m_d3d->Release();
      if(0 == count)
         delete this;

      return count;
   }

    /*** IDirect3D9 methods ***/
    HRESULT STDMETHODCALLTYPE RegisterSoftwareDevice(void* pInitializeFunction)
   {
      return m_d3d->RegisterSoftwareDevice(pInitializeFunction);
   }

    UINT STDMETHODCALLTYPE GetAdapterCount()
   {
      return m_d3d->GetAdapterCount();
   }

    HRESULT STDMETHODCALLTYPE GetAdapterIdentifier( UINT Adapter,DWORD Flags,D3DADAPTER_IDENTIFIER9* pIdentifier)
   {
      return m_d3d->GetAdapterIdentifier(Adapter, Flags, pIdentifier);
   }

    UINT STDMETHODCALLTYPE GetAdapterModeCount( UINT Adapter,D3DFORMAT Format)
   {
      return m_d3d->GetAdapterModeCount(Adapter, Format);
   }

    HRESULT STDMETHODCALLTYPE EnumAdapterModes( UINT Adapter,D3DFORMAT Format,UINT Mode,D3DDISPLAYMODE* pMode)
   {
      return m_d3d->EnumAdapterModes(Adapter, Format, Mode, pMode);
   }

    HRESULT STDMETHODCALLTYPE GetAdapterDisplayMode( UINT Adapter,D3DDISPLAYMODE* pMode)
   {
      return m_d3d->GetAdapterDisplayMode(Adapter, pMode);
   }

    HRESULT STDMETHODCALLTYPE CheckDeviceType( UINT Adapter,D3DDEVTYPE DevType,D3DFORMAT AdapterFormat,D3DFORMAT BackBufferFormat,BOOL bWindowed)
   {
      return m_d3d->CheckDeviceType(Adapter, DevType, AdapterFormat, BackBufferFormat, bWindowed);
   }

    HRESULT STDMETHODCALLTYPE CheckDeviceFormat( UINT Adapter,D3DDEVTYPE DeviceType,D3DFORMAT AdapterFormat,DWORD Usage,D3DRESOURCETYPE RType,D3DFORMAT CheckFormat)
   {
      return m_d3d->CheckDeviceFormat(Adapter, DeviceType, AdapterFormat, Usage, RType, CheckFormat);
   }

    HRESULT STDMETHODCALLTYPE CheckDeviceMultiSampleType( UINT Adapter,D3DDEVTYPE DeviceType,D3DFORMAT SurfaceFormat,BOOL Windowed,D3DMULTISAMPLE_TYPE MultiSampleType,DWORD* pQualityLevels)
   {
      return m_d3d->CheckDeviceMultiSampleType(Adapter, DeviceType, SurfaceFormat, Windowed, MultiSampleType, pQualityLevels);
   }

    HRESULT STDMETHODCALLTYPE CheckDepthStencilMatch( UINT Adapter,D3DDEVTYPE DeviceType,D3DFORMAT AdapterFormat,D3DFORMAT RenderTargetFormat,D3DFORMAT DepthStencilFormat)
   {
      return m_d3d->CheckDepthStencilMatch(Adapter, DeviceType, AdapterFormat, RenderTargetFormat, DepthStencilFormat);
   }

    HRESULT STDMETHODCALLTYPE CheckDeviceFormatConversion( UINT Adapter,D3DDEVTYPE DeviceType,D3DFORMAT SourceFormat,D3DFORMAT TargetFormat)
   {
      return m_d3d->CheckDeviceFormatConversion(Adapter, DeviceType, SourceFormat, TargetFormat);
   }

    HRESULT STDMETHODCALLTYPE GetDeviceCaps( UINT Adapter,D3DDEVTYPE DeviceType,D3DCAPS9* pCaps)
   {
      return m_d3d->GetDeviceCaps(Adapter, DeviceType, pCaps);
   }

    HMONITOR STDMETHODCALLTYPE GetAdapterMonitor( UINT Adapter)
   {
      return m_d3d->GetAdapterMonitor(Adapter);
   }

    HRESULT STDMETHODCALLTYPE CreateDevice( UINT Adapter,D3DDEVTYPE DeviceType,HWND hFocusWindow,DWORD BehaviorFlags,D3DPRESENT_PARAMETERS* pPresentationParameters,IDirect3DDevice9** ppReturnedDeviceInterface)
   {
      HRESULT hr = m_d3d->CreateDevice(Adapter, DeviceType, hFocusWindow, BehaviorFlags,
         pPresentationParameters, ppReturnedDeviceInterface);
      
      if(SUCCEEDED(hr))
      {
         // Return our device
         *ppReturnedDeviceInterface = new MyDirect3DDevice9(this, *ppReturnedDeviceInterface);
      }

      return hr;
   }

private:
   IDirect3D9* m_d3d;
};

[CrazyEddie]

I've studied this somewhat, and of course you'll appreciate that this is a lot more difficult due to the nature of what you're doing

I can give you one easy fix that should get it working, but is not ideal since it could hurt performance and remove some features of CEGUI. Basically, you can disable CEGUI's caching of windows to texture, which - based on the information given - is where the issue lies. To do this, load the looknfeel xml file into an editor, and change any mention of AutoRenderingSurfaceEnabled from True to False. You will also need to ensure you do not use things such as window rotations, since that will automatically enable the surface for rotated windows.

Ok, to drill down into the issue a bit deeper. First, I will state that in all the testing I have ever done, the reset code relating to surfaces works ok - both for alt-tab and for windowed / fullscreen switching. It seems to me that somewhere, somehow the reference counts on the textures used for the surface(s) are getting additional references added. Looking at my code, it could happen if postD3DReset were to be called 'out of sequence', thereby adding a reference and causing the release in preD3DReset to not actually result in the texture being released. Though this said, looking at your code, this should not be the case - you might want to add some logging just to be doubly sure.

I'm not sure what else to suggest at this stage.

CE.

[OceanMachine]

Thanks for taking a look.

I would suggest that may be right about the postD3DReset() calling order issue - if I comment that out then the reset happens OK on both alt-tab and resolution/mode change [of course the textures don't get restored, but hopefully it still proves the point?].

With a bit more logging, I found that EndScene() is being called multiple times per Present() in my game (which apparently is perfectly valid and quite commonplace). Based on this, I also tried 'locking' my call to renderGUI and pre/postD3DReset to set a flag when called and only reset the flag on Present() to ensure that I was only rendering once per frame (and also only checking device state and calling pre/post3d3Reset once per frame), but this didn't seem to make a difference.

[Edit1:]
So then I added a load mode debug logging, and found that even though EndScene was being called very often, it seems that Reset() was being called directly after Present(), before ANY of the EndScene calls, meaning the device was trying to reset before I had chance to call preD3DReset()! This left it in a state where it seems the D3D API itself was preventing illegal calls to EndScene() [as the device was in a non-cooperative state], meaning I would never be able to call preD3DReset() from within there! This ONLY seemed to happen on res/mode change, not on alt-tab, so I guess it is a timing issue with when the particular game checks and processes requests to change mode or resolution.

I therefore hooked the Reset() call and I setup a call to preD3DReset() in there instead, and then left the call to postD3DReset() in EndScene() to happen the first time it is called with a coop level of D3D_OK, and this fixed the issue!

Lesson to myself an anyone else who ends up having a similar problem when hooking Direct3D calls from another app: It seems you can't reliably check for a non-cooperative device state in EndScene() with the purpose of being able to call your CEGUI preD3DReset() (or other resource-releasing code), because the only valid D3D API calls in this situation are Reset, TestCooperativeLevel, CheckDeviceState or Release, and this is limited by the API itself (according to the DirectX debug DLL output)... You must call preD3DReset() and free other resources from inside one of these (and the most sensible one is inside Reset() right before you call the "normal" device's Reset() function)... This is contrary to what is demonstrated in all of the D3D hook examples that I have seen, so I didn't think anything of it before...

I will try what you suggested with disabling of AutoRenderingSurface and let you know whether this also would have worked, and what the performance hit is like for me.
[Edit2:]
This ALSO fixed my issue, I set AutoRenderingSurface to "False" in the .looknfeel file as you suggested and it worked great. As a bonus, it also didn't have any measurable performance hit - I am getting ~250FPS with the value set to TRUE and ~240FPS with it set to FALSE, but I only have a fairly simple GUI so maybe that is why it works OK for me. Thanks for the alternative solution.

Thanks again CE!! I really appreciate it, and thanks to the rest of the team - keep up the awesome work on the CEGUI system

[clockman]

I got similar problem.
I hooked EndScene in Directx9, and when I resize window looks like game creates new directx device (different pointer), Right now I handle it in that way that I shutdown CEGUI and init it again.
There is some faster way for example just to create new dx9 renderer and set it in code ? I want to save state of windows & scripts when that happends...