Crash Analysis using WinDbg
I am working on an a complex managed application that has locked up after performing a task. I am recording the steps I take here so I can recall what I need to do the next time. All my short term memory has been uploaded to Google these days!
How to get the dump
There are a bunch of ways to get the dump, we need a dump with memory to be of any use. The simplest way if you are unprepared is to use the Task Manager. On the context menu for any running process is Create Dump File.
There are smarter ways though and a dump can automatically be created if an application meets certain criteria such as becoming unresponsive or hitting a process use peak. SysInternals ProcDump
What tool to use?
Visual Studio can open a mini-dump file but unless the Managed application uses version 4.0 .NET or above you cannot debug a Managed application.
The best tool I have used so far is WinDbg. This is the real man's debugger for Windows applications. You can find downloads for the debugging tools here.
The next question is which WinDbg to run? There are two installed on a x64 OS. Should you run the WinDbg x64? That depends on if you are debugging a 64bit application. If not then run the x86 version of WinDbg.
Opening the crash-dump file
Run the WinDbg and from the file menu Open Crash Dump... <Ctrl+D>. When it opens you will get some information and some complaints about symbols. We will get to symbols later.
First thing you need to do is figure out some details about the application you are debugging.
lm vm appname will list the module information about the appname module. This will include the file version information. If you want to have an easy life ensure that version of the application is installed on your system rather than any other. To successfully debug from a dump file it is essential that the symbols you use match the code being investigated in the dump file or you will get misinformation.
There are two sets of symbols we need. The Microsoft symbols and our own application symbols. The Microsoft symbols we can obtain from the Microsoft symbol server. Out application symbols we can obtain from the PDB files we generated when the application was built. These will be held in our own symbol server. (Doh! - we don't have any of that!)
Our get out of jail for not generating the symbols and storing them during the build process is the wonderful news that JetBrains dotPeek can de-compile our binaries and generate the symbols for our application and can even simulate a symbol server to provide those symbols to the debugger. Phew.
Using JetBrains dotPeek as a Symbol Server
Open dotPeek and add the application installation folder to the paths. dotPeek will de-compile all the binaries in that folder. Enable the SymbolServer and the symbols from that de-compilation will be made available to the debugger.
Setting the symbol path
We want to set the symbol file path to use the Microsoft symbol server, the dotPeek symbol server and to cache the symbols from these servers locally to improve on load times. The command is as follows.
WinDbg Command
- .sympath cache*c:\SymbolCache;srv*http://msdl.microsoft.com/download/symbols;http://localhost:33417/
The symbols for an application are date and time stamped and a checksum is stored. The date and timestamp along with the checksum are used to verify that the symbols loaded match the binary image being debugged. This causes us a problem because we do not use a symbol store as part of our build process so the date and time of any binary we build will not match. We can get extra information on the problems with the loading of symbols using the following commands.
WinDbg Command
- !sym noisy
- .reload -f
It is possible to force WinDbg to load any symbols. This may seem like a get out of jail but there are problems with it. The symbols may not match the code being investigated and this can lead to miss-information about the problem. The only real use of this is when you re-build the application from source at a later date using the label in the source control system.
WinDbg Command
- .symopt+0x40
- .reload -f
Text goes here
Debugging Managed applications
We need some help debugging managed code. There are WinDbg extensions that will assist us.
There is some guidance on this here
There is some guidance on this here
WinDbg Command
- .loadby sos mscorwks
- .reload -f
Text goes here
Extensions to help with managed debugging
SOS
SOS is Microsoft's WinDbg extension for debugging managed applications.
SOSEX
SOSEX is a third party extension which can be found here. Download the 32 bit version and unzip it to the directory where the 32bit version of WinDbg is installed.
WinDbg Command
- !load sosex
Debugging a hang (deadlock)
When debugging a hang in a managed application we may want to do several things but a good start would be to look for threads waiting on the various types of lock.
WinDbg Command
- !sosex.dlk -d
The output of this command will tell us if there are threads that are deadlocked. If there are none then we may want to examine all the call stacks for all the managed threads
WinDbg Command
- ~*e !CLRStack
We may wish to examine all of the Managed locks and which thread holds them
WinDbg Command
- !sosex.mlocks -d
We may wish to see which threads are taking the most time to see which one might be hogging the system
WinDbg Command
- !runaway
We may wish to see the names of the running managed threads
WinDbg Command
- .foreach (address {!dumpheap -short -mt 725e4960}) { .if (poi(${address}+c) != 0) { .printf "%d ",poi(${address}+28); .printf "%mu\r\n", poi(${address}+c)+8 } }
No comments:
Post a Comment