Entries:
Comments:
Posts:

Loading User Information from Channel 9

Something went wrong getting user information from Channel 9

Latest Achievement:

Loading User Information from MSDN

Something went wrong getting user information from MSDN

Visual Studio Achievements

Latest Achievement:

Loading Visual Studio Achievements

Something went wrong getting the Visual Studio Achievements

Analyzing a dump of a process under memory pressure - 11

Download

Right click “Save as…”

You may wish to watch the video using one of the high quality links on the right so the text is readable.

This episode is the second of three which introduces viewers to the topic of troubleshooting memory pressure issues.  In the previous episode, we took a high level overview of virtual memory as it relates to troubleshooting these common memory pressure issues.  We'll use information from that episode and apply it here as we begin investigation of a dump file from a process exhibiting memory pressure.  Using windbg and some debugging extensions, we'll see how much we can learn about the source of the memory problem from a single dump file.

 

.NET Debugging for the Production Environment, Part 10

.NET Debugging for the Production Environment, Part 12

Tags:

Follow the Discussion

  • Hi Brad,

     

    When I ran "!address summary" on a .net process dump, it displayed output similar to the following for most of the sections

     

    *************************************************************************
    ***                                                                                    ***
    ***                                                                                    ***
    *** Your debugger is not using the correct symbols              ***
    ***                                                                                    ***
    *** In order for this command to work properly, your symbol  ***
    *** path must point to .pdb files that have full type information ***
    ***                                                                                    ***
    *** Certain .pdb files (such as the public OS symbols) do not ***
    *** contain the required information. Contact the group that   ***
    *** provided you with these symbols if you need this command ***
    *** to work.                                                                       ***
    ***                                                                                    ***
    *** Type referenced: ntdll!_TEB                                           ***
    ***                                                                                    ***
    *************************************************************************

     

    After I run the commands ".sympath+ C:\Working\Symbols" and ".reload", the command "!address summary" resulted in the following

    0:000> !address summary

    *** WARNING: Unable to verify checksum for PantherDesktop.exe
    *** ERROR: Module load completed but symbols could not be loaded for PantherDesktop.exe
    *** ERROR: Module load completed but symbols could not be loaded for xpsp2res.dll
    *** WARNING: Unable to verify checksum for System.Core.ni.dll
    *** ERROR: Module load completed but symbols could not be loaded for System.Core.ni.dll
    *** ERROR: Symbol file could not be found. Defaulted to export symbols for TortoiseOverlays.dll -
    *** ERROR: Symbol file could not be found. Defaulted to export symbols for TortoiseStub.dll -
    *** ERROR: Symbol file could not be found. Defaulted to export symbols for normaliz.dll -
    *** ERROR: Symbol file could not be found. Defaulted to export symbols for intl3_tsvn.dll -
    *** ERROR: Symbol file could not be found. Defaulted to export symbols for TortoiseSVN.dll -
    *** ERROR: Symbol file could not be found. Defaulted to export symbols for PGDetoured.dll -
    *** ERROR: Symbol file could not be found. Defaulted to export symbols for wmp.dll -
    *** ERROR: Module load completed but symbols could not be loaded for wmploc.dll
    *** ERROR: Symbol file could not be found. Defaulted to export symbols for wininet.dll -
    *** ERROR: Symbol file could not be found. Defaulted to export symbols for iertutil.dll -
    *** ERROR: Symbol file could not be found. Defaulted to export symbols for GdiPlus.dll -
    *** WARNING: Unable to verify checksum for SMDiagnostics.ni.dll
    *** ERROR: Module load completed but symbols could not be loaded for SMDiagnostics.ni.dll
    *** WARNING: Unable to verify checksum for System.IdentityModel.ni.dll
    *** ERROR: Module load completed but symbols could not be loaded for System.IdentityModel.ni.dll
    *** WARNING: Unable to verify checksum for System.Runtime.DurableInstancing.ni.dll
    *** ERROR: Module load completed but symbols could not be loaded for System.Runtime.DurableInstancing.ni.dll
    *** WARNING: Unable to verify checksum for System.Runtime.Serialization.ni.dll
    *** ERROR: Module load completed but symbols could not be loaded for System.Runtime.Serialization.ni.dll
    *** WARNING: Unable to verify checksum for System.ServiceModel.ni.dll
    *** ERROR: Module load completed but symbols could not be loaded for System.ServiceModel.ni.dll
    *** ERROR: Symbol file could not be found. Defaulted to export symbols for wpfgfx_v0400.dll -
    *** ERROR: Symbol file could not be found. Defaulted to export symbols for PresentationNative_v0400.dll -
    *** WARNING: Unable to verify checksum for PresentationCore.ni.dll
    *** ERROR: Module load completed but symbols could not be loaded for PresentationCore.ni.dll
    *** ERROR: Module load completed but symbols could not be loaded for wpftxt_v0400.dll
    *** WARNING: Unable to verify checksum for PresentationFramework.ni.dll
    *** ERROR: Module load completed but symbols could not be loaded for PresentationFramework.ni.dll
    *** WARNING: Unable to verify checksum for PresentationFramework.Aero.ni.dll
    *** ERROR: Module load completed but symbols could not be loaded for PresentationFramework.Aero.ni.dll
    *** WARNING: Unable to verify checksum for PresentationFramework.Luna.ni.dll
    *** ERROR: Module load completed but symbols could not be loaded for PresentationFramework.Luna.ni.dll
    *** WARNING: Unable to verify checksum for UIAutomationProvider.ni.dll
    *** ERROR: Module load completed but symbols could not be loaded for UIAutomationProvider.ni.dll
    *** WARNING: Unable to verify checksum for UIAutomationTypes.ni.dll
    *** ERROR: Module load completed but symbols could not be loaded for UIAutomationTypes.ni.dll
    *** WARNING: Unable to verify checksum for WindowsBase.ni.dll
    *** ERROR: Module load completed but symbols could not be loaded for WindowsBase.ni.dll
    *** WARNING: Unable to verify checksum for System.Xaml.ni.dll
    *** ERROR: Module load completed but symbols could not be loaded for System.Xaml.ni.dll
    *** ERROR: Symbol file could not be found. Defaulted to export symbols for netapi32.dll -
    *** WARNING: Unable to verify checksum for Accessibility.ni.dll
    *** ERROR: Symbol file could not be found. Defaulted to export symbols for nlssorting.dll -
    *** WARNING: Unable to verify checksum for System.ComponentModel.DataAnnotations.ni.dll
    *** ERROR: Module load completed but symbols could not be loaded for System.ComponentModel.DataAnnotations.ni.dll
    *** WARNING: Unable to verify checksum for System.Configuration.ni.dll
    *** ERROR: Module load completed but symbols could not be loaded for System.Configuration.ni.dll
    *** ERROR: Symbol file could not be found. Defaulted to export symbols for PGHook.dll -
    *** ERROR: Symbol file could not be found. Defaulted to export symbols for System.Data.dll -
    *** WARNING: Unable to verify checksum for System.Data.ni.dll
    *** ERROR: Module load completed but symbols could not be loaded for System.Data.ni.dll
    *** WARNING: Unable to verify checksum for System.Numerics.ni.dll
    *** ERROR: Module load completed but symbols could not be loaded for System.Numerics.ni.dll
    *** ERROR: Symbol file could not be found. Defaulted to export symbols for System.Transactions.dll -
    *** WARNING: Unable to verify checksum for System.Transactions.ni.dll
    *** ERROR: Module load completed but symbols could not be loaded for System.Transactions.ni.dll
    *** WARNING: Unable to verify checksum for System.Xml.ni.dll
    *** ERROR: Module load completed but symbols could not be loaded for System.Xml.ni.dll
    *** WARNING: Unable to verify checksum for System.Xml.Linq.ni.dll
    *** ERROR: Module load completed but symbols could not be loaded for System.Xml.Linq.ni.dll
    *** ERROR: Symbol file could not be found. Defaulted to export symbols for libaprutil_tsvn.dll -
    *** ERROR: Symbol file could not be found. Defaulted to export symbols for libapr_tsvn.dll -
    *** ERROR: Symbol file could not be found. Defaulted to export symbols for winmm.dll -
    *** ERROR: Symbol file could not be found. Defaulted to export symbols for wintrust.dll -
    *** ERROR: Symbol file could not be found. Defaulted to export symbols for imagehlp.dll -
    *** ERROR: Symbol file could not be found. Defaulted to export symbols for iphlpapi.dll -
    *** ERROR: Symbol file could not be found. Defaulted to export symbols for comctl32.dll -
    *** ERROR: Symbol file could not be found. Defaulted to export symbols for ole32.dll -
    *** ERROR: Symbol file could not be found. Defaulted to export symbols for crypt32.dll -
    *** ERROR: Symbol file could not be found. Defaulted to export symbols for urlmon.dll -
    *** ERROR: Symbol file could not be found. Defaulted to export symbols for msvcp90.dll -
    *** ERROR: Symbol file could not be found. Defaulted to export symbols for msvcr90.dll -
    *** ERROR: Symbol file could not be found. Defaulted to export symbols for clr.dll -
    *** ERROR: Symbol file could not be found. Defaulted to export symbols for clrjit.dll -
    *** WARNING: Unable to verify checksum for mscorlib.ni.dll
    *** ERROR: Module load completed but symbols could not be loaded for mscorlib.ni.dll
    *** WARNING: Unable to verify checksum for System.ni.dll
    *** ERROR: Module load completed but symbols could not be loaded for System.ni.dll
    *** WARNING: Unable to verify checksum for System.Drawing.ni.dll
    *** ERROR: Module load completed but symbols could not be loaded for System.Drawing.ni.dll
    *** WARNING: Unable to verify checksum for System.Windows.Forms.ni.dll
    *** ERROR: Module load completed but symbols could not be loaded for System.Windows.Forms.ni.dll
    *** ERROR: Symbol file could not be found. Defaulted to export symbols for shell32.dll -


    Failed to map Heaps (error 80004005)
    Usage: Free
    Base Address: 00000000
    End Address: 00010000
    Region Size: 00010000
    Type: 00000000
    State: 00010000 MEM_FREE
    Protect: 00000001 PAGE_NOACCESS

     

    Can you please tell me how can I fix this issue?

     

    Regards,

    Jehanzeb

  • Jehanzeb,

    Initially, looks like a symbol issue.

    1. What did you use to get this dump?  What tool, and what command?  You need to ensure you have a full user mode dump.  While you can line up symbols correctly with a mini dump (which is typically ~1%-5% of the file size of a full user mode dump), they have limited use when it comes to debugging production application issues.

    2. I don't know what your sympath is, but adding c:\windows\symbols isn't likely to help.  Try using our public symbol server:

    .sympath srv*c:\symcache*http://msdl.microsoft.com/download/symbols

    .reload /f

  • Hi Brad,

     

    Thanks for the reply.

     

    The dump is generated automatically when the application crashed. It's a full mode user dump. I wrote a tool which uses clrdump to generate the dumps file. I did the following changes in the registry so that Windows run my tool whenever an application is crashed.

     

    HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\Current Version\AeDebug
    Key: Debugger, Value: C:\Dump\ManagedDumpGenerator.exe %d %d %s %d

    Key: Auto, Value: 1

     

    Regards,

    Jehanzeb

  • @Jehanzeb: Hmmm.. what arguments do you use when you issue a .dump command.  Also, try the sympath changes I suggested to see if that helps.

  • Brad, the tool I wrote launched clrdump.exe with the following parameters It generates full memory dump.

     

    clrdump.exe <pid> <dumpfile> max

     

    Just to clarify, I am able to see the managed callback. I found out the root cause of the crash and fixed the issue. But when I try to run "!address -summary", it displays the same output I copied in the first post.

  • @Jehanzeb:I've never heard of clrdump before, but from debuginfo.com, it says the default is a mini, which sounds like doing a variation of .dump /m with cdb or the other debuggers from Microsoft's debugging tools package.  Honestly, I never use these mini dumps, as they're very limited in what they provide and essentially useless for tshooting memory issues.  I tried running !address from a dump obtained by .dump /m (see debugger.chm from our Debugging Tools package) and !address wouldn't even run.

    In any case, I'd suggest following the sympath chages I suggested above.  Then tell me the results.

  • @BradL: Thanks, the output looks better after the sympath changes you suggested.

     

    Just to clarify: I am using max option with clrdump.exe which generates full dump.

     

     

    ClrDump.exe supports the following command line parameters:
    
    clrdump <pid> <dumpfile> [min | mid | max]
    
    pid
    Process id of the target process (required)
    
    dumpfile
    Name of the minidump file (required)
    
    min | mid | max
    Size of the minidump file (optional):
    min - Smallest possible minidump (enough to recover call stacks of all threads); this option is used by default
    mid - Very informative but still compressible minidump
    max - Full minidump (full memory contents and all other kinds of information included; usually very large in size and not compressible)

Remove this comment

Remove this thread

close

Comments Closed

Comments have been closed since this content was published more than 30 days ago, but if you'd like to continue the conversation, please create a new thread in our Forums,
or Contact Us and let us know.