Comment Feed for Channel 9 - Analyzing a dump of a process under memory pressure - 11 http://media.ch9.ms/ch9/90c6/0e467261-9396-4562-a224-a376bd0f90c6/NetDebugging11_220.jpg Channel 9 - Analyzing a dump of a process under memory pressure - 11 You may wish to watch the video using one of the high quality links on the right so the text is readable. This episode is the second of three which introduces viewers to the topic of troubleshooting memory pressure issues.  In the previous episode, we took a high level overview of virtual memory as it relates to troubleshooting these common memory pressure issues.  We'll use information from that episode and apply it here as we begin investigation of a dump file from a process exhibiting memory pressure.  Using windbg and some debugging extensions, we'll see how much we can learn about the source of the memory problem from a single dump file.   .NET Debugging for the Production Environment, Part 10 .NET Debugging for the Production Environment, Part 12 en Wed, 19 Jun 2013 23:23:53 GMT Wed, 19 Jun 2013 23:23:53 GMT Rev9 Re: Analyzing a dump of a process under memory pressure - 11 Hi Brad,

 

When I ran "!address summary" on a .net process dump, it displayed output similar to the following for most of the sections

 

*************************************************************************
***                                                                                    ***
***                                                                                    ***
*** Your debugger is not using the correct symbols              ***
***                                                                                    ***
*** In order for this command to work properly, your symbol  ***
*** path must point to .pdb files that have full type information ***
***                                                                                    ***
*** Certain .pdb files (such as the public OS symbols) do not ***
*** contain the required information. Contact the group that   ***
*** provided you with these symbols if you need this command ***
*** to work.                                                                       ***
***                                                                                    ***
*** Type referenced: ntdll!_TEB                                           ***
***                                                                                    ***
*************************************************************************

 

After I run the commands ".sympath+ C:\Working\Symbols" and ".reload", the command "!address summary" resulted in the following

0:000> !address summary

*** WARNING: Unable to verify checksum for PantherDesktop.exe
*** ERROR: Module load completed but symbols could not be loaded for PantherDesktop.exe
*** ERROR: Module load completed but symbols could not be loaded for xpsp2res.dll
*** WARNING: Unable to verify checksum for System.Core.ni.dll
*** ERROR: Module load completed but symbols could not be loaded for System.Core.ni.dll
*** ERROR: Symbol file could not be found. Defaulted to export symbols for TortoiseOverlays.dll -
*** ERROR: Symbol file could not be found. Defaulted to export symbols for TortoiseStub.dll -
*** ERROR: Symbol file could not be found. Defaulted to export symbols for normaliz.dll -
*** ERROR: Symbol file could not be found. Defaulted to export symbols for intl3_tsvn.dll -
*** ERROR: Symbol file could not be found. Defaulted to export symbols for TortoiseSVN.dll -
*** ERROR: Symbol file could not be found. Defaulted to export symbols for PGDetoured.dll -
*** ERROR: Symbol file could not be found. Defaulted to export symbols for wmp.dll -
*** ERROR: Module load completed but symbols could not be loaded for wmploc.dll
*** ERROR: Symbol file could not be found. Defaulted to export symbols for wininet.dll -
*** ERROR: Symbol file could not be found. Defaulted to export symbols for iertutil.dll -
*** ERROR: Symbol file could not be found. Defaulted to export symbols for GdiPlus.dll -
*** WARNING: Unable to verify checksum for SMDiagnostics.ni.dll
*** ERROR: Module load completed but symbols could not be loaded for SMDiagnostics.ni.dll
*** WARNING: Unable to verify checksum for System.IdentityModel.ni.dll
*** ERROR: Module load completed but symbols could not be loaded for System.IdentityModel.ni.dll
*** WARNING: Unable to verify checksum for System.Runtime.DurableInstancing.ni.dll
*** ERROR: Module load completed but symbols could not be loaded for System.Runtime.DurableInstancing.ni.dll
*** WARNING: Unable to verify checksum for System.Runtime.Serialization.ni.dll
*** ERROR: Module load completed but symbols could not be loaded for System.Runtime.Serialization.ni.dll
*** WARNING: Unable to verify checksum for System.ServiceModel.ni.dll
*** ERROR: Module load completed but symbols could not be loaded for System.ServiceModel.ni.dll
*** ERROR: Symbol file could not be found. Defaulted to export symbols for wpfgfx_v0400.dll -
*** ERROR: Symbol file could not be found. Defaulted to export symbols for PresentationNative_v0400.dll -
*** WARNING: Unable to verify checksum for PresentationCore.ni.dll
*** ERROR: Module load completed but symbols could not be loaded for PresentationCore.ni.dll
*** ERROR: Module load completed but symbols could not be loaded for wpftxt_v0400.dll
*** WARNING: Unable to verify checksum for PresentationFramework.ni.dll
*** ERROR: Module load completed but symbols could not be loaded for PresentationFramework.ni.dll
*** WARNING: Unable to verify checksum for PresentationFramework.Aero.ni.dll
*** ERROR: Module load completed but symbols could not be loaded for PresentationFramework.Aero.ni.dll
*** WARNING: Unable to verify checksum for PresentationFramework.Luna.ni.dll
*** ERROR: Module load completed but symbols could not be loaded for PresentationFramework.Luna.ni.dll
*** WARNING: Unable to verify checksum for UIAutomationProvider.ni.dll
*** ERROR: Module load completed but symbols could not be loaded for UIAutomationProvider.ni.dll
*** WARNING: Unable to verify checksum for UIAutomationTypes.ni.dll
*** ERROR: Module load completed but symbols could not be loaded for UIAutomationTypes.ni.dll
*** WARNING: Unable to verify checksum for WindowsBase.ni.dll
*** ERROR: Module load completed but symbols could not be loaded for WindowsBase.ni.dll
*** WARNING: Unable to verify checksum for System.Xaml.ni.dll
*** ERROR: Module load completed but symbols could not be loaded for System.Xaml.ni.dll
*** ERROR: Symbol file could not be found. Defaulted to export symbols for netapi32.dll -
*** WARNING: Unable to verify checksum for Accessibility.ni.dll
*** ERROR: Symbol file could not be found. Defaulted to export symbols for nlssorting.dll -
*** WARNING: Unable to verify checksum for System.ComponentModel.DataAnnotations.ni.dll
*** ERROR: Module load completed but symbols could not be loaded for System.ComponentModel.DataAnnotations.ni.dll
*** WARNING: Unable to verify checksum for System.Configuration.ni.dll
*** ERROR: Module load completed but symbols could not be loaded for System.Configuration.ni.dll
*** ERROR: Symbol file could not be found. Defaulted to export symbols for PGHook.dll -
*** ERROR: Symbol file could not be found. Defaulted to export symbols for System.Data.dll -
*** WARNING: Unable to verify checksum for System.Data.ni.dll
*** ERROR: Module load completed but symbols could not be loaded for System.Data.ni.dll
*** WARNING: Unable to verify checksum for System.Numerics.ni.dll
*** ERROR: Module load completed but symbols could not be loaded for System.Numerics.ni.dll
*** ERROR: Symbol file could not be found. Defaulted to export symbols for System.Transactions.dll -
*** WARNING: Unable to verify checksum for System.Transactions.ni.dll
*** ERROR: Module load completed but symbols could not be loaded for System.Transactions.ni.dll
*** WARNING: Unable to verify checksum for System.Xml.ni.dll
*** ERROR: Module load completed but symbols could not be loaded for System.Xml.ni.dll
*** WARNING: Unable to verify checksum for System.Xml.Linq.ni.dll
*** ERROR: Module load completed but symbols could not be loaded for System.Xml.Linq.ni.dll
*** ERROR: Symbol file could not be found. Defaulted to export symbols for libaprutil_tsvn.dll -
*** ERROR: Symbol file could not be found. Defaulted to export symbols for libapr_tsvn.dll -
*** ERROR: Symbol file could not be found. Defaulted to export symbols for winmm.dll -
*** ERROR: Symbol file could not be found. Defaulted to export symbols for wintrust.dll -
*** ERROR: Symbol file could not be found. Defaulted to export symbols for imagehlp.dll -
*** ERROR: Symbol file could not be found. Defaulted to export symbols for iphlpapi.dll -
*** ERROR: Symbol file could not be found. Defaulted to export symbols for comctl32.dll -
*** ERROR: Symbol file could not be found. Defaulted to export symbols for ole32.dll -
*** ERROR: Symbol file could not be found. Defaulted to export symbols for crypt32.dll -
*** ERROR: Symbol file could not be found. Defaulted to export symbols for urlmon.dll -
*** ERROR: Symbol file could not be found. Defaulted to export symbols for msvcp90.dll -
*** ERROR: Symbol file could not be found. Defaulted to export symbols for msvcr90.dll -
*** ERROR: Symbol file could not be found. Defaulted to export symbols for clr.dll -
*** ERROR: Symbol file could not be found. Defaulted to export symbols for clrjit.dll -
*** WARNING: Unable to verify checksum for mscorlib.ni.dll
*** ERROR: Module load completed but symbols could not be loaded for mscorlib.ni.dll
*** WARNING: Unable to verify checksum for System.ni.dll
*** ERROR: Module load completed but symbols could not be loaded for System.ni.dll
*** WARNING: Unable to verify checksum for System.Drawing.ni.dll
*** ERROR: Module load completed but symbols could not be loaded for System.Drawing.ni.dll
*** WARNING: Unable to verify checksum for System.Windows.Forms.ni.dll
*** ERROR: Module load completed but symbols could not be loaded for System.Windows.Forms.ni.dll
*** ERROR: Symbol file could not be found. Defaulted to export symbols for shell32.dll -


Failed to map Heaps (error 80004005)
Usage: Free
Base Address: 00000000
End Address: 00010000
Region Size: 00010000
Type: 00000000
State: 00010000 MEM_FREE
Protect: 00000001 PAGE_NOACCESS

 

Can you please tell me how can I fix this issue?

 

Regards,

Jehanzeb

posted by Jehanzeb

]]> http://channel9.msdn.com/Series/-NET-Debugging-Stater-Kit-for-the-Production-Environment/Analyzing-a-dump-of-a-process-under-memory-pressure-11#c634828524654915254 Mon, 10 Sep 2012 05:41:05 GMT http://channel9.msdn.com/Series/-NET-Debugging-Stater-Kit-for-the-Production-Environment/Analyzing-a-dump-of-a-process-under-memory-pressure-11#c634828524654915254 Jehanzeb Re: Analyzing a dump of a process under memory pressure - 11 Jehanzeb,

Initially, looks like a symbol issue.

1. What did you use to get this dump?  What tool, and what command?  You need to ensure you have a full user mode dump.  While you can line up symbols correctly with a mini dump (which is typically ~1%-5% of the file size of a full user mode dump), they have limited use when it comes to debugging production application issues.

2. I don't know what your sympath is, but adding c:\windows\symbols isn't likely to help.  Try using our public symbol server:

.sympath srv*c:\symcache*http://msdl.microsoft.com/download/symbols

.reload /f

posted by BradL

]]> http://channel9.msdn.com/Series/-NET-Debugging-Stater-Kit-for-the-Production-Environment/Analyzing-a-dump-of-a-process-under-memory-pressure-11#c634828943715605112 Mon, 10 Sep 2012 17:19:31 GMT http://channel9.msdn.com/Series/-NET-Debugging-Stater-Kit-for-the-Production-Environment/Analyzing-a-dump-of-a-process-under-memory-pressure-11#c634828943715605112 BradL Re: Analyzing a dump of a process under memory pressure - 11 Hi Brad,

 

Thanks for the reply.

 

The dump is generated automatically when the application crashed. It's a full mode user dump. I wrote a tool which uses clrdump to generate the dumps file. I did the following changes in the registry so that Windows run my tool whenever an application is crashed.

 

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\Current Version\AeDebug
Key: Debugger, Value: C:\Dump\ManagedDumpGenerator.exe %d %d %s %d

Key: Auto, Value: 1

 

Regards,

Jehanzeb

posted by Jehanzeb

]]> http://channel9.msdn.com/Series/-NET-Debugging-Stater-Kit-for-the-Production-Environment/Analyzing-a-dump-of-a-process-under-memory-pressure-11#c634832373192966473 Fri, 14 Sep 2012 16:35:19 GMT http://channel9.msdn.com/Series/-NET-Debugging-Stater-Kit-for-the-Production-Environment/Analyzing-a-dump-of-a-process-under-memory-pressure-11#c634832373192966473 Jehanzeb Re: Analyzing a dump of a process under memory pressure - 11 @Jehanzeb: Hmmm.. what arguments do you use when you issue a .dump command.  Also, try the sympath changes I suggested to see if that helps.

posted by BradL

]]> http://channel9.msdn.com/Series/-NET-Debugging-Stater-Kit-for-the-Production-Environment/Analyzing-a-dump-of-a-process-under-memory-pressure-11#c634832543525941164 Fri, 14 Sep 2012 21:19:12 GMT http://channel9.msdn.com/Series/-NET-Debugging-Stater-Kit-for-the-Production-Environment/Analyzing-a-dump-of-a-process-under-memory-pressure-11#c634832543525941164 BradL Re: Analyzing a dump of a process under memory pressure - 11 Brad, the tool I wrote launched clrdump.exe with the following parameters It generates full memory dump.

 

clrdump.exe <pid> <dumpfile> max

 

Just to clarify, I am able to see the managed callback. I found out the root cause of the crash and fixed the issue. But when I try to run "!address -summary", it displays the same output I copied in the first post.

posted by Jehanzeb

]]> http://channel9.msdn.com/Series/-NET-Debugging-Stater-Kit-for-the-Production-Environment/Analyzing-a-dump-of-a-process-under-memory-pressure-11#c634833705218534994 Sun, 16 Sep 2012 05:35:21 GMT http://channel9.msdn.com/Series/-NET-Debugging-Stater-Kit-for-the-Production-Environment/Analyzing-a-dump-of-a-process-under-memory-pressure-11#c634833705218534994 Jehanzeb Re: Analyzing a dump of a process under memory pressure - 11 @Jehanzeb:I've never heard of clrdump before, but from debuginfo.com, it says the default is a mini, which sounds like doing a variation of .dump /m with cdb or the other debuggers from Microsoft's debugging tools package.  Honestly, I never use these mini dumps, as they're very limited in what they provide and essentially useless for tshooting memory issues.  I tried running !address from a dump obtained by .dump /m (see debugger.chm from our Debugging Tools package) and !address wouldn't even run.

In any case, I'd suggest following the sympath chages I suggested above.  Then tell me the results.

posted by BradL

]]> http://channel9.msdn.com/Series/-NET-Debugging-Stater-Kit-for-the-Production-Environment/Analyzing-a-dump-of-a-process-under-memory-pressure-11#c634834460431682169 Mon, 17 Sep 2012 02:34:03 GMT http://channel9.msdn.com/Series/-NET-Debugging-Stater-Kit-for-the-Production-Environment/Analyzing-a-dump-of-a-process-under-memory-pressure-11#c634834460431682169 BradL Re: Analyzing a dump of a process under memory pressure - 11 @BradL: Thanks, the output looks better after the sympath changes you suggested.

 

Just to clarify: I am using max option with clrdump.exe which generates full dump.

 

 

ClrDump.exe supports the following command line parameters:

clrdump <pid> <dumpfile> [min | mid | max]

pid
Process id of the target process (required)

dumpfile
Name of the minidump file (required)

min | mid | max
Size of the minidump file (optional):
min - Smallest possible minidump (enough to recover call stacks of all threads); this option is used by default
mid - Very informative but still compressible minidump
max - Full minidump (full memory contents and all other kinds of information included; usually very large in size and not compressible)

posted by Jehanzeb

]]> http://channel9.msdn.com/Series/-NET-Debugging-Stater-Kit-for-the-Production-Environment/Analyzing-a-dump-of-a-process-under-memory-pressure-11#c634834591003013391 Mon, 17 Sep 2012 06:11:40 GMT http://channel9.msdn.com/Series/-NET-Debugging-Stater-Kit-for-the-Production-Environment/Analyzing-a-dump-of-a-process-under-memory-pressure-11#c634834591003013391 Jehanzeb