Comment Feed for Channel 9 - Defrag Tools: #29 - WinDbg - ETW Logging http://media.ch9.ms/ch9/3442/c72e017c-3afc-4604-8750-ad09d7a93442/DefragTools29_220.jpg Channel 9 - Defrag Tools: #29 - WinDbg - ETW Logging In this episode of Defrag Tools, Andrew Richards, Chad Beeder and Larry Larsen continue looking at the Debugging Tools for Windows (in particular WinDbg). WinDbg is a debugger that supports user mode debugging of a process, or kernel mode debugging of a computer. This installment goes over the Event Tracing for Windows (ETW) buffers in a kernel mode dump or live session. The ETW buffers can be extracted from the dump and viewed using the Windows Performance Toolkit (WPT). The buffers give you insight in to what has beem happening recently on the computer. We use these commands: !wmitrace.strdump !wmitrace.logsave 0xNN c:\example.etl !wmitrace.eventlogdump 0xNN !wmitrace.help Make sure you watch Defrag Tools Episode #1 and Defrag Tools Episode #23 for instructions on how to get the Debugging Tools for Windows and how to set the required environment variables for symbol and source code resolution. This episode shows how install the Windows Performance Toolkit. Timeline:[00:00] - Event Tracing for Windows (ETW)[02:18] - Windows Performance Toolkit (WPT)[03:48] - !wmitrace.strdump[04:53] - !wmitrace.logsave 0xNN c:\example.etl[05:50] - Windows Performance Analyzer (WPA) & xPerfView[07:57] - _NT_SYMCACHE_PATH[10:24] - !wmitrace.eventlogdump 0xNN[12:16] - Used for logging and performance by many teams[15:35] - Private PDBs are needed to decode some entries[20:00] - Windows Performance Recorder (wprui.exe)[20:35] - Disable Paging Executive[23:40] - WPR adds the NT Kernel Logger [24:19] - 10min run-through of the data collected with the General, CPU and Disk providers en Wed, 22 May 2013 03:33:58 GMT Wed, 22 May 2013 03:33:58 GMT Rev9 Re: Defrag Tools: #29 - WinDbg - ETW Logging Finally you touch my number 1 tool/topic ETW Big Smile

[20:35] - Disable Paging Executive

this is only needed for x64 Windows version to get CallStacks/Stackwalking (the data to walk the call chains is saved outside the stack)

Adding image version data is done later when MERGING the ETL file (user mode ETW events + system data + kernel.etl into the final file).

Without merging, you can't load the symbols on a different machine.

[20:00] - Windows Performance Recorder (wprui.exe)

but with the new UI you loose control on what it traced. It traces too much data and when the source system is under stress it impact the PC too much.

I still use my scripts to run xperf with the flags I need (if I want circular log (with file size), the amount of buffers I want to use).

 

[05:50] - Windows Performance Analyzer (WPA) & xPerfView

that you also use xperfview shows me that all my complains during betatest where right. I said this so many times to Michael when still he was the PM of WPT/XPERF/XPERFVIEW. WPA is a terrible bad UI with blurry graphs and so many scrollbar all over the time. It sucks so much Mad

And new users are lost in to choose which graphs they need or not.

 

btw, UBMP = Unified Background Process Manager

posted by MagicAndre1981

]]> http://channel9.msdn.com/Shows/Defrag-Tools/Defrag-Tools-29-WinDbg-ETW-Logging#c634974839240161321 Tue, 26 Feb 2013 13:58:44 GMT http://channel9.msdn.com/Shows/Defrag-Tools/Defrag-Tools-29-WinDbg-ETW-Logging#c634974839240161321 MagicAndre1981 Re: Defrag Tools: #29 - WinDbg - ETW Logging posted by land

]]> http://channel9.msdn.com/Shows/Defrag-Tools/Defrag-Tools-29-WinDbg-ETW-Logging#c634975333770007714 Wed, 27 Feb 2013 03:42:57 GMT http://channel9.msdn.com/Shows/Defrag-Tools/Defrag-Tools-29-WinDbg-ETW-Logging#c634975333770007714 land Re: Defrag Tools: #29 - WinDbg - ETW Logging @land: We talked about your question in #30, but here are a few tips:

  • Make sure the debugger arch matches the target (use the x86 debugger in this case)
  • Make sure you are debugging the child IE ptocess, not the parent IE process. The parent is 64bit, the children (tabs) are 32bit.

posted by windev

]]> http://channel9.msdn.com/Shows/Defrag-Tools/Defrag-Tools-29-WinDbg-ETW-Logging#c634977617061104746 Fri, 01 Mar 2013 19:08:26 GMT http://channel9.msdn.com/Shows/Defrag-Tools/Defrag-Tools-29-WinDbg-ETW-Logging#c634977617061104746 windev Re: Defrag Tools: #29 - WinDbg - ETW Logging I may have missed something but what kind of dump does it have to be to use these commands, minidump is not supported, correct? so it has to be a full dump? I am sorry if I did not pay enough attention and missed something, the whole episode went by really fast. 

posted by s3curityConsult

]]> http://channel9.msdn.com/Shows/Defrag-Tools/Defrag-Tools-29-WinDbg-ETW-Logging#c634980354558979448 Mon, 04 Mar 2013 23:10:55 GMT http://channel9.msdn.com/Shows/Defrag-Tools/Defrag-Tools-29-WinDbg-ETW-Logging#c634980354558979448 s3curityConsult Re: Defrag Tools: #29 - WinDbg - ETW Logging @s3curityConsult: I don't think you missed it - pretty sure I never pointed that out. The buffers are pool memory in the kernel, so you need a kernel (2) or complete (1) dump. Kernel is the default up to win8, the win8 default is automatic, which is kernel or complete based on pagefile size. You want the c:\windows\memory.dmp file, not the c:\windows\minidump files.

posted by windev

]]> http://channel9.msdn.com/Shows/Defrag-Tools/Defrag-Tools-29-WinDbg-ETW-Logging#c634980516455383794 Tue, 05 Mar 2013 03:40:45 GMT http://channel9.msdn.com/Shows/Defrag-Tools/Defrag-Tools-29-WinDbg-ETW-Logging#c634980516455383794 windev Re: Defrag Tools: #29 - WinDbg - ETW Logging So that we can play with it using xperfview?

Thanks in advance

posted by loverboy

]]> http://channel9.msdn.com/Shows/Defrag-Tools/Defrag-Tools-29-WinDbg-ETW-Logging#c634987152454195186 Tue, 12 Mar 2013 20:00:45 GMT http://channel9.msdn.com/Shows/Defrag-Tools/Defrag-Tools-29-WinDbg-ETW-Logging#c634987152454195186 loverboy Re: Defrag Tools: #29 - WinDbg - ETW Logging @loverboy: This script will capture a lot of cool data. You can get roughly the same data using the default options in WPRUI.exe.

@echo off
echo Press a key when ready to start...
pause
echo .
echo ...Capturing...
echo .
xperf -on PROC_THREAD+LOADER+Base+Diag+Latency+FileIO+DRIVERS+DPC+DISPATCHER -stackwalk Profile+CSwitch+ReadyThread+ThreadCreate -BufferSize 1024 -MinBuffers 256 -MaxBuffers 256 -MaxFile 256 -FileMode Circular
echo Press a key when you want to stop...
pause
echo .
echo ...Stopping...
echo .
xperf -stop -d result.etl

posted by windev

]]> http://channel9.msdn.com/Shows/Defrag-Tools/Defrag-Tools-29-WinDbg-ETW-Logging#c634988884540959528 Thu, 14 Mar 2013 20:07:34 GMT http://channel9.msdn.com/Shows/Defrag-Tools/Defrag-Tools-29-WinDbg-ETW-Logging#c634988884540959528 windev Re: Defrag Tools: #29 - WinDbg - ETW Logging Thanks a lot, that's what I needed ;)

Since I have Windows 7 64bit I don't think I can use WPRUI.exe (Am I right?)
Thanks anyway

posted by loverboy

]]> http://channel9.msdn.com/Shows/Defrag-Tools/Defrag-Tools-29-WinDbg-ETW-Logging#c634988937945490790 Thu, 14 Mar 2013 21:36:34 GMT http://channel9.msdn.com/Shows/Defrag-Tools/Defrag-Tools-29-WinDbg-ETW-Logging#c634988937945490790 loverboy Re: Defrag Tools: #29 - WinDbg - ETW Logging posted by loverboy

]]> http://channel9.msdn.com/Shows/Defrag-Tools/Defrag-Tools-29-WinDbg-ETW-Logging#c634988965503646586 Thu, 14 Mar 2013 22:22:30 GMT http://channel9.msdn.com/Shows/Defrag-Tools/Defrag-Tools-29-WinDbg-ETW-Logging#c634988965503646586 loverboy Re: Defrag Tools: #29 - WinDbg - ETW Logging @loverboy: WPRUI works on Win7 too (not supported, but it works).
@loverboy: kernel.etl is the kernel mode buffers, user.etl (not made here) would be the user mode buffers. The result.etl is the merge of these two, plus, it add the required information to resolve symbols. (The raw buffers just have pointers. The merge adds the module info so that offset can be mapped back to a funcion name via a symbol)

posted by windev

]]> http://channel9.msdn.com/Shows/Defrag-Tools/Defrag-Tools-29-WinDbg-ETW-Logging#c634989006701542497 Thu, 14 Mar 2013 23:31:10 GMT http://channel9.msdn.com/Shows/Defrag-Tools/Defrag-Tools-29-WinDbg-ETW-Logging#c634989006701542497 windev Re: Defrag Tools: #29 - WinDbg - ETW Logging
Talking about WPA(and/or XPerfView) ... when you analyze on a 64bit PC a .etl taken on a 32bit machine, do you have to use the 32bit version (like windbg) or on a 64bit PC you have to use WPA or XPerfView 64bit version anyway?

posted by loverboy

]]> http://channel9.msdn.com/Shows/Defrag-Tools/Defrag-Tools-29-WinDbg-ETW-Logging#c634989771921044440 Fri, 15 Mar 2013 20:46:32 GMT http://channel9.msdn.com/Shows/Defrag-Tools/Defrag-Tools-29-WinDbg-ETW-Logging#c634989771921044440 loverboy Re: Defrag Tools: #29 - WinDbg - ETW Logging @loverboy: I always use the 64bit version for all traces - don't recall ever having an issue. If the stack includes CLR code, you won't get the function names regardless of archectural combination.

posted by windev

]]> http://channel9.msdn.com/Shows/Defrag-Tools/Defrag-Tools-29-WinDbg-ETW-Logging#c634990101819631996 Sat, 16 Mar 2013 05:56:21 GMT http://channel9.msdn.com/Shows/Defrag-Tools/Defrag-Tools-29-WinDbg-ETW-Logging#c634990101819631996 windev Re: Defrag Tools: #29 - WinDbg - ETW Logging
Now it gives an error
C:\Program Files\Windows Performance Toolkit>Recording_Example.bat
Press a key when ready to start...
Premere un tasto per continuare . . .
.
...Capturing...
.
xperf: warning: This system is not fully configured for x64 stack tracing.
Please modify the registry under:

HKLM\System\CurrentControlSet\Control\Session Manager\Memory Management

and set the value:

DisablePagingExecutive (REG_DWORD) = 1

Then reboot before retrying tracing.

Note: Tracing has been enabled, this is just a warning.
xperf: error: NT Kernel Logger: Impossibile creare un file, se il file esiste già. (0xb7).
Press a key when you want to stop...
Premere un tasto per continuare . . .
.
...Stopping...
.
xperf: error: Merge ETL: Percorso specificato non valido. (0xa1).

xperf: error: NT Kernel Logger: Impossibile creare un file, se il file esiste già. (0xb7). means Impossible creating a file if the file already exists (0xb7)

What file is it talking about?

posted by loverboy

]]> http://channel9.msdn.com/Shows/Defrag-Tools/Defrag-Tools-29-WinDbg-ETW-Logging#c634993988230313766 Wed, 20 Mar 2013 17:53:43 GMT http://channel9.msdn.com/Shows/Defrag-Tools/Defrag-Tools-29-WinDbg-ETW-Logging#c634993988230313766 loverboy Re: Defrag Tools: #29 - WinDbg - ETW Logging Problem solved (I think it was just a temporary problem, since there was no result.etl file anywhere)

posted by loverboy

]]> http://channel9.msdn.com/Shows/Defrag-Tools/Defrag-Tools-29-WinDbg-ETW-Logging#c634994004974283644 Wed, 20 Mar 2013 18:21:37 GMT http://channel9.msdn.com/Shows/Defrag-Tools/Defrag-Tools-29-WinDbg-ETW-Logging#c634994004974283644 loverboy Re: Defrag Tools: #29 - WinDbg - ETW Logging Debugging a BSOD due to a bug in Windows 8 64 bit (process MSSE a.k.a. Windows defender during quick scan, driver ndis.sys, error ATTEMPTED_EXECUTE_OF_NOEXECUTE_MEMORY, can reproduce 100% on my system).

Have a complete memory dump created after xperf –on DiagEasy.

!wmitrace.logsave command produces a corrupt ETL, both wpa.exe and xperfview.exe say "Trace C:\Temp\Crashes\DISK.etl could not be successfully opened [0x80070570]. Aborting operation".

Any ideas how to fix?

posted by Const

]]> http://channel9.msdn.com/Shows/Defrag-Tools/Defrag-Tools-29-WinDbg-ETW-Logging#c634999120786027211 Tue, 26 Mar 2013 16:27:58 GMT http://channel9.msdn.com/Shows/Defrag-Tools/Defrag-Tools-29-WinDbg-ETW-Logging#c634999120786027211 Const Re: Defrag Tools: #29 - WinDbg - ETW Logging @loverboy

error 0xb7 occurs when you already run a tool which odes ETW tracing (ResMon, ProcExp):

http://www.msfn.org/board/topic/155479-xperf-error-nt-kernel-logger-cannot-create-a-file-when-that-file-already-exists-0xb7/

posted by MagicAndre1981

]]> http://channel9.msdn.com/Shows/Defrag-Tools/Defrag-Tools-29-WinDbg-ETW-Logging#c635001606961586776 Fri, 29 Mar 2013 13:31:36 GMT http://channel9.msdn.com/Shows/Defrag-Tools/Defrag-Tools-29-WinDbg-ETW-Logging#c635001606961586776 MagicAndre1981 Re: Defrag Tools: #29 - WinDbg - ETW Logging @Const: Email me (defragtools@microsoft.com) to organize a way for you to send me the dump.

posted by windev

]]> http://channel9.msdn.com/Shows/Defrag-Tools/Defrag-Tools-29-WinDbg-ETW-Logging#c635003720336303983 Mon, 01 Apr 2013 00:13:53 GMT http://channel9.msdn.com/Shows/Defrag-Tools/Defrag-Tools-29-WinDbg-ETW-Logging#c635003720336303983 windev