[Solved] Problem debugging P/Invoke calls

I've done plenty of managed/unmanaged development/debugging before but this is the 1st time I ran into this annoying problem. I'm using VS 2010 SP1 on Windows 7 64-bit with all latest OS patches.

Basically, I have a simple struct and a simple unmanaged function declaration like this:

typedef struct
{
    char* Value1;
    int Value2;
}   MyStruct;

unsigned long long Foo(const MyStruct* someStruct, const char* name);

The C# side looks like this:

[StructLayout(LayoutKind.Sequential, CharSet = CharSet.Ansi)]
public struct MyStruct
{
    public string Value1;
    public int Value2;
}

[DllImport("some.dll", SetLastError = true, CharSet = CharSet.Ansi, CallingConvention = CallingConvention.Cdecl)]
public static extern long Foo(ref MyStruct someStruct, string name);

Here is the problem: When I try to step into the unmanaged code when calling it from managed code, I get one of two exceptions: "An unhandled exception of type 'System.Runtime.InteropServices.SEHException' occurred in myApp.exe" or "System.AccessViolationException".

This looks like what you will get from a calling convention mismatch but I tried all 3 and none of them work. The strange thing is that if I step into the unmanaged function (and stop at the opening curly brace), the contents of someStruct is perfectly valid, as well as the value of name. However as soon as I step the next line (before even executing that next line), I get the exception.

The other strange thing is that when I step over the call, it sometimes completes successfully with no problem at all. If I had any declarations wrong it would always fail, not fail sometimes.

Does anyone have an idea what is going on?

BTW, if I disable unmanaged debuggin, everything works fine 100% of the time. This looks like an issue with the debugger.

EDIT: OK now I know this has something to do with the debugger. If I stop at the opening brace of the unmanaged function, and then put a breakpoint on the very next line and press F5, it hits the breakpoint with no problem. If I put another breakpoint anywhere else and hit F5, it stops there too. But as soon as I press F10 (single step), I get the exception.

While I can now at least debug the code by setting breakpoints everywhere, it is annoying since the whole point of this C# project is to be a test harnass for an unmanaged API. Debugging the API without single-step working properly will be a real pain.

Any ideas?

OK turns out that VS can get into some funky state. Resetting all settings back to the defaults solves the problem. See here where someone else ran into the same problem.

@BitFlipper: Curious.  Is there an upgrade involved like in the referenced link?

@davewill:

I'm not aware of any upgrade that solves the problem. Resetting the settings solved it for me in this case though. I then carefully imported just my fonts & colors, shortcuts and a few other select settings and everything is back to normal again.

I know VS has some issues with retaining settings. For instance sometimes it randomly sets some large invalid widths for the combo boxes in the toolbar, forcing everything else off the toolbar. I then have to go in and reset each combo box width to some sane value. It happens about once every week or two.

@BitFlipper: The "upgrade" reference was in regards to how Visual Studio and/or Windows came to be in its current form.  It isn't entirely clear from the referenced thread which.  The indication came from the moderators post suggesting that there may be some history related to VS settings being messed up in upgrade scenarios.

If in your environment an upgrade was involved at some point in it's past then that lends a little to the hint that we should probably keep an eye out in upgrade scenarios.

@davewill:

Ah, got it thanks. However I don't believe it is upgrade related because it just started happening out of the blue. Also the other corruption I mentioned where the combo box sizes get corrupted happens randomly one every week or two. It just seems VS can get its setting corrupted randomly. And this happens on both my desktop and laptop at unrelated times.