Defrag Tools

Defrag Tools #167 - Debugging User Mode Crash Dumps Redux

Play Defrag Tools #167 - Debugging User Mode Crash Dumps Redux
Sign in to queue

Description

In this episode of Defrag Tools, Andrew Richards and Chad Beeder use Debugging Tools for Windows (WinDbg) to determine the root cause of various application crashes which have occurred on Andrew's computer. We use Sysinternals ProcDump to capture the dumps.

While debugging, we take a side trip into configuring colors for Compressed and Encrypted files in Windows Explorer, and use Sysinternals Process Monitor to determine why the debugger was getting an Access Denied when loading the PDE Debugger Extension.

We did a similar investigation in these two episodes:

We cover how to install the Debugging Tools for Windows in this episode:

Get the Sysinternals tools from http://www.sysinternals.com. We use:

Get the PDE debugger extension from the Defrag Tools OneDrive

Get your Symbol Path to the Microsoft Public Symbol Server:

  • Via Environment Variable
    setx /m _NT_SYMBOL_PATH SRV*C:\My\Sym*https://msdl.microsoft.com/download/symbols
  • In the Debugger
    .sympath SRV*C:\My\Sym*https://msdl.microsoft.com/download/symbols

To collect dumps of crashes on your own machine, install ProcDump as the Postmortem (AeDebugger) debugger:

    md c:\dumps
    procdump.exe -ma -i c:\dumps 

On any dump (user or kernel), you can run automated analysis to view the issue:

    !analyze -v

Debugging Cheat Sheet

  • c0000005 is an Access Violation - use .ecxr & k
  • c000027b is a Stowed Exception (Store Apps) - use !pde.dse
  • e0434352 is a CLR Exception - use !sos.pe
  • e0697282 is a C++ Exception - use .ecxr & k
  • 80000003 is a Breakpoint - use !analyze -v
  • When typing a decimal number, prefix it "0n"
  • When typing a hexadecimal number, prefix it "0x" (the default prefix)

Common Debugger Commands

.exr -1

  • View the Exception Code and the Exception Parameters
  • Number looking like C0xxxxxx and 80xxxxxx are HRESULTs (Error Codes)
  • Number looking like 7FFFxxxxxxxx are usually code (assembler) addresses

!address <number>

  • Display the address information - Commited/Reserved/Free, Image/Mapped/Private
  • Used to determine if a number is code or data.

ln <address>

  • List Nearest address
  • Displays the symbol at or near the address
  • Used to determine if a number is code or data.

.ecxr

  • Change the debugging context to the point of the exception (rather than being at the Windows Error Reporting context)

r

  • View the registers at the current context. (.ecxr produces the same output)

k

  • View the call stack

lmvm <module>

  • View loaded module verbosely with a mask
  • View a module's details, including folder, timestamp, description, copyright, product/file version

|  (Vertical Bar or Pipe character)

  • View the executable's path (e.g. c:\windows\notepad.exe)

!ext.error

  • Get the description of an Error Code. Best at describing System Error Codes.

!pde.err

  • Get the description of an Error Code. Good at describing HRESULTs (80xxxxxx and C0xxxxxx)

!pde.dpx

  • Scrape the current thread for evidence (symbols, structures, strings, etc.)

.formats <number>

  • Displays the number in various formats.
  • Easy way of working out if a number is actually ASCII text, or a date/time

!sos.pe

  • Display a CLR Exception.
  • If there is an Inner Exception, click on the link to view it.

.cordll -u & .cordll -l

  • If SOS isn't loaded, try to do an unload and load of the CLR support.

!peb

  • View the Process Environment Block (Modules, Command Line, Environment Variables, etc.)

!teb

  • View the current Thread's Environment Block (Stack Range, Last Error Code, Last Status Code, etc.)

!gle

  • Get Last Error
  • Display the Last Error Code and Last Status Code of the current thread

.cls

  • Clear the screen.

.reload

  • Force a reload (download) of symbols for the modules on the current stack.

.reload /f

  • Force a full reload (download) of symbols for the modules on the current stack.

 

Store Applications

To view the currently installed Store Applications and their version use:

Registry Editor (regedit.exe)

  • HKEY_CURRENT_USER\SOFTWARE\Classes\ActivatableClasses\Package

PowerShell

 

Tags:

Debugging, sysinternals

Embed

Download

Download this episode

More episodes in this series

Related episodes

rss

The Discussion

  • User profile image
    Ytterbium

    What's is difference between setting these registry key vs procdump?

     

    https://msdn.microsoft.com/en-us/library/windows/desktop/bb787181(v=vs.85).aspx

  • User profile image
    SOAJunK

    Excellent video!
    But in the next one, a little bit slower will be better... ;)

    Thx

  • User profile image
    spgennard

    tip: you can use %* in the d.cmd script eg:

    @dir %*

  • User profile image
    Magic​Andre1981

    What's is difference between setting these registry key vs procdump?

    https://msdn.microsoft.com/en-us/library/windows/desktop/bb787181(v=vs.85).aspx

    With WER you have more control (specify dump options, dump path per app base)

     

    The .dumpdebug is great, but sadly not documented, I found it some months ago in this blog:

    http://sww-it.ru/2016-03-13/1320#more-1320

     

    My symbol cache is over 70GB :P (75 556 358 949 Bytes, 79271 files, 108318 folders).

     

     

  • User profile image
    Luke

    Hello,

    Great show as usual!

    Would you have any advise what else can be check in the debugger that would explain in more detail, why the access violation occurred in Outlook.exe(see windbg output below) on module wwlib.dll? Or would you say that updating wwlib.dll is the best approach?

    0:000> |
    . 0 id: 14094 examine name: C:\Program Files (x86)\Microsoft Office\Office12\OUTLOOK.EXE


    0:000> .exr -1
    ExceptionAddress: 5a0d7dc4 (WWLIB+0x00027dc4)
    ExceptionCode: c0000005 (Access violation)
    ExceptionFlags: 00000000
    NumberParameters: 2
    Parameter[0]: 00000000
    Parameter[1]: 00000304
    Attempt to read from address 00000304


    0:000> .ecxr
    eax=05d00f5e ebx=00000000 ecx=06acc000 edx=06acc000 esi=06acc000 edi=00000000
    eip=5a0d7dc4 esp=0018ed50 ebp=0018ef60 iopl=0 nv up ei pl nz na po nc
    cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00210202
    WWLIB+0x27dc4:
    5a0d7dc4 ffb704030000 push dword ptr [edi+304h] ds:002b:00000304=????????


    0:000> kv
    *** Stack trace for last set context - .thread/.cxr resets it
    # ChildEBP RetAddr Args to Child
    WARNING: Stack unwind information not available. Following frames may be wrong.
    00 0018ef60 5adfd382 00000000 00000001 00000000 WWLIB+0x27dc4
    01 0018ef8c 30238d2e 06bb5064 06a3af64 00060c22 WWLIB+0xd4d382
    02 0018efc8 3023aa63 00000000 06a598c0 00000001 OUTLOOK!FFolderSupportsUnicode+0x2487
    03 0018eff0 3024bc75 00000000 06a598dc 00000000 OUTLOOK!FFolderSupportsUnicode+0x41bc
    04 0018f188 3023fec4 00000000 0fffffff 06aa4800 OUTLOOK!SmoothScroll+0x1098c
    05 0018f19c 3023fd55 06a598dc 0fffffff 6a6a7964 OUTLOOK!SmoothScroll+0x4bdb
    06 0018f800 2f927f6c 00000000 0fffffff 00000002 OUTLOOK!SmoothScroll+0x4a6c
    07 0018f884 3023aaf9 06aa4858 0fffffff 00000000 OUTLOOK!GetAllocCounters+0x2a78d
    08 0018f8a4 3023cfd0 06a3abe0 06a3aba0 00000000 OUTLOOK!FFolderSupportsUnicode+0x4252
    09 0018f910 3023c556 06aa3c30 0018f940 02702970 OUTLOOK!SmoothScroll+0x1ce7
    0a 0018f964 2f976570 06aa3c30 00000000 00000000 OUTLOOK!SmoothScroll+0x126d
    0b 0018f988 30267a8e 06aa3c30 00000000 07150e60 OUTLOOK!GetCentralObject+0x2384
    0c 0018fa40 301a2cfd 00000000 0000001b 0018fa98 OUTLOOK!SmoothScroll+0x2c7a5
    0d 0018fa50 2f987ab1 02702400 3028e990 00000001 OUTLOOK!HrMsgDownloadedNotification+0x2dbc6
    0e 0018fa98 2f9a6ffc 0661bc88 00000000 00000001 OUTLOOK!XGetExplorerStoragePath+0x4adb
    0f 0018fad4 3026e830 00000001 00000000 00000001 OUTLOOK!XGetExplorerStoragePath+0x24026
    10 0018faf8 3026ea50 00000000 00000000 30290b38 OUTLOOK!SmoothScroll+0x33547
    11 0018fb34 2f973397 00000000 00000000 00000000 OUTLOOK!SmoothScroll+0x33767
    12 0018fb5c 2f80ec5f 2f7f0000 00000000 00384c6e OUTLOOK!GetAllocCounters+0x75bb8
    13 0018fbec 757c338a 7efde000 0018fc38 77869902 OUTLOOK+0x1ec5f
    14 0018fbf8 77869902 7efde000 7a58ce6c 00000000 kernel32!BaseThreadInitThunk+0xe (FPO: [Non-Fpo])
    15 0018fc38 778698d5 2f7f3910 7efde000 ffffffff ntdll!__RtlUserThreadStart+0x70 (FPO: [Non-Fpo])
    16 0018fc50 00000000 2f7f3910 7efde000 00000000 ntdll!_RtlUserThreadStart+0x1b (FPO: [Non-Fpo])

    0:000> lmvm WWLIB
    Browse full module list
    start end module name
    5a0b0000 5b1c4000 WWLIB T (no symbols)
    Loaded symbol image file: WWLIB.DLL
    Image path: C:\Program Files (x86)\Microsoft Office\Office12\WWLIB.DLL
    Image name: WWLIB.DLL
    Browse all global symbols functions data
    Timestamp: Thu May 19 04:51:24 2016 (573D383C)
    CheckSum: 01102DB5
    ImageSize: 01114000
    File version: 12.0.6749.5000
    Product version: 12.0.6749.0
    File flags: 0 (Mask 3F)
    File OS: 40004 NT Win32
    File type: 2.0 Dll
    File date: 00000000.00000000
    Translations: 0000.04b0 0000.04e4 0409.04b0 0409.04e4

  • User profile image
    ChadBeeder

    @Luke:Not enough information there to determine anything. If you share the .dmp on a OneDrive we could take a look. If I'm not mistaken, though, that looks like Office 2007, which hasn't gotten any non-security fixes in several years now, so the real answer might be to update to a newer version of Office.

  • User profile image
    Joyce Gammill

    Totally lost; not a clue what y'all are talking about (senior citizen, duh - frustrating); want the update (do I?), but not sure I'll ever get it, i.e., understand what I should be doing. Got anything plain and simple for old gray mares, maybe? FYI: Dell Insipron 1545 laptop, but never used as such - plugged in, no battery, never leaves desk.

  • User profile image
    Timothy Suhr

    I was able to follow the direction the video made. It would be helpful to have more information about this hole topic. Please do not stop here! As a tester I think this may help. More about what would best go into bug reports. Also the use of these tools from a testing perspective would be helpful.

    Thanks again and keep up the great work...

  • User profile image
    pendingio

    Thanks for the Video guys.

    Any Details when this book is coming out, they keep pushing it back a month :)
    https://www.microsoftpressstore.com/store/troubleshooting-with-the-windows-sysinternals-tools-9780735684447

     

  • User profile image
    matt

    if i want to get the most inclusive dump, should i use ProcDump -ma or -mp?

  • User profile image
    matthew grossman

    NM -ma it is

  • User profile image
    siodmy

    how do you get !pe right away? I have to run .cordll -l to make windbg load sos.

  • User profile image
    adolfox
    Just discovered your show and am super excited about it. Thank you for making it!

Add Your 2 Cents