Mysterious Desktop Heap Errors

Some of the most challenging software bugs to fix are memory leaks. A single leak can cause different errors all over the system and it can happen at any time. Usually the problem does not get resolved because it only appears after a system has been running for a long time, and it goes away after a system reboot. Below is a discussion about one type of resource leak which involves the desktop heap.

Symptoms

Code or application works under user account but not system account after a period of time

This is usually the first symptom which points to desktop heap errors. If you research the desktop heap, you will learn that every application running under system shares the same desktop heap, so if there is a resource leak on one application, it can affect every application running under system. On a fresh reboot the heap has not yet been exhausted so it works, after the heap has been exhausted the application will continue to work if run under a different user.

"STATUS_DLL_INIT_FAILED (0xC0000142)" error when starting a process

When things start getting really bad, launching any executable under system will start throwing this error. Even a simple C++ console application that simply prints hello world will fail to start. If you have an application that is leaking slowly, you may not encounter this at first but it will start happening eventually.

"A desktop heap allocation failed" error in the windows event log

Sometimes you are lucky enough to get one of these errors in the windows event log. It will be under Windows Logs -> System.

Confirming the problem

If your application is failing when running normally, but works when you start debugging:

1. Make sure your debugging tools are also running under the system user.
   1. Scheduled task method
      This is a quick hack to get something running under system user using a GUI. The first step is to create a scheduled task that will run your application or executable. Use the Task Scheduler included in Windows. Once it's created you should be able to right click on the scheduled task and select Run. Do this to confirm it actually runs as expected. The last step is to edit the scheduled task to run system user. This is done by clicking "Change User or Group"

1. PSexec Tool
   Download SysInternal Tools and find the psexec tool. This has a lot of uses and one of them is to run any command line under system.

>whoami
FOXHOUND udy
>psexec -s cmd.exe
C:\WINDOWS\system32>whoami

NT AUTHORITY\SYSTEM

1. Desktop heap monitor
2. Microsoft has tool that lets your see the desktop heap usage.
   http://www.microsoft.com/en-us/download/details.aspx?id=17782
   Desktop Heap Monitor Version 8.1

Unfortunately, I never actually got this to work on Windows 7. It appears that this tool only supports Windows XP & Server 2003. I did a lot of research trying to get this to work on Windows server 2008 but it looks like it is just not possible.

1. Download the hacked version of the tool
2. Fiddle with command line to download the debug symbols
3. Digitally sign the hacked driver so that it can actually be installed
4. Reboot the machine to start Windows in some special mode so that the tool can be installed (thus getting rid of the desktop heap error anyway)
5. Continue to fail using and installing the tool because Microsoft released a security patch which you have already applied

Fixes

1. Find the offending program and restart it.
   1. Nice way
      Generally programs running under the Windows SYSTEM account are Windows services. These can usually be controlled with the services control manager.

Run -> services.msc. Take your time and stop each service one by one. Stop a service, see if problem keeps happening. If it keeps happening then you can start that service again and move on to the next service.

1. Barbarian style
   Sometimes, restarting the services is not enough. Now we have to go through a similar process but instead of gracefully restarting services, kill every process running under system. Kill a process, test, kill another process, test...

1. Reboot the machine
2. If the above don't work and you really need to get the application working again, simply rebooting the machine solves the problem but only temporarily.

Related Links

http://blogs.msdn.com/b/ntdebugging/archive/2007/01/04/desktop-heap-overview.aspx
http://blog.airesoft.co.uk/2009/10/desktop-heap-monitor-vista-7/
http://support.microsoft.com/kb/947246