the exception that gets thrown when running the C++ app outside of the debugger is 0x40010006, which is DBG_PRINTEXCEPTION_C

Hmm, I have the feeling that Windows won't always like this stack switching business. NT's structured exception handling relies on the stack and if you switch the stack... I'm not quite sure what can happen.

I'm wondering whether the thread switching function is really saving enough registers

It should be enough unless you use SSE. A normal kernel the relise on interrupts to switch threads has no choice but to save all registers because it doesn't know which registers are used. But what you're doing here is more like cooperative multithreading and more importantly, it's all done in C. The C/C++ compiler expects that a function will preserve those 4 registers (ebp, ebx, edi, esi), all other general purpose registers can be modified by the function as needed. Also, the FPU stack is supposed to be empty when a function is called so it doesn't contain anything that needs saving.

Note I need 4 versions, depending on whether it is a static or instance call, and whether it passes the extra data or not

That sounds a bit excesive. Normally there is a single thread start function and this isn't the function you pass to the Thread constructor through a delegate. This start function is a special function that takes care of initializing the thread and then calls the delegate.

Also, the static/instance distinction is the job of the delegate, the thread start function should not be sensitive to this.

Between push eax and push esi you need to add another push for the return address. This can be 0 if you chose never to return from the start function or it can be the address of an "exit function" that calls SwitchContext to get away from the thread.

but EBP is null and so the stack is also messed up

Well, it's null because that's how InitializeContext initializes it. But why do you care about the contents of EBP? The start function doesn't expect a value in this register.

Below is what I have so far

I see you're pushing ESP, why? Again, this will be poped as EBP but you don't need EBP to have any particular value.

So instead, I need to have an additional void* in each thread whose address will be passed in to the SwitchContext call, correct?

Yes, this value contains the current stack pointer for the thread. When you switch away from a thread its current stack pointer gets saved and the stack pointer of the new thread is loaded.

The question is, if this is the very first context switch, what should be stored at the pNewStack value

That's the role of InitializeContext, it sets up the initial stack so it looks similar to the one expected by SwitchContext. That's why the start address gets pushed on the stack, when SwitchContext willreturn to what's stored on the stack, your start address. This might sound strange, the start function is notcalled but returned too, but it works fine as long as the stack is setup correctly. That's also the reason for those 4 push eax, they compensate the fact that SwitchContext pops edi, esi, ebp, ebx.

I assume this should be at some negative offset into that thread's NativeStack

The InitializeContext already computes the correct value but it looks like it doesn't store it correctly. Let's fix things a bit:

void NativePlatform::InitializeThreadContext(RuntimeThread* pThread, VirtualObject** pData) {
    void* stack = pThread->NativeStack;
    void* startAddress = (void*)pThread->GetStartAddress();

    __asm {
        // Save current stack
        mov ecx, esp

        // Switch to pThread's stack
        mov esp, stack

        // Push arguments for the start function
        // It looks like your start function expects pThread and pData so:
        mov eax, pData
        push eax      
        mov eax, pThread
        push eax

        // Push the start function address, SwitchContext will return to this address
        mov eax, startAddress
        push eax

        // Push 4 values to compensate for pop edi, esi, ebx, ebp in SwitchContext
        xor eax, eax
        push eax
        push eax
        push eax
        push eax

        // At this point the stack is similar to what SwitchContext expects, save the stack pointer
        mov stack, esp

        // Switch back to this thread's stack
        mov esp, ecx
    }
    // save the thread's current stack pointer
    pThread->CurrentStackPointer = stack;
}

I added some comments to make it more clear and also added arguments for your start function (my previous example didn't pass any arguments to the start function).

After InitializeContext runs you should be able to switch to the new thread by doing SwitchContext(&currentThread->CurrentStackPointer, &newThread->CurrentStackPointer).

Now there's another fun thing to solve. What happens when the thread exits?

1. you mentioned that it happens after using cout, what happens if you don't use it?
2. How many threads are you starting and from where? All from main thread? Or each thread starts another thread?

Unlikely. It can be wrong only if you changed the default from cdecl to stdcall or fastcall.

Hmm, the code looks correct. The 4 pops after the calls aren't really needed but they don't hurt.

Heap corruption + stack allocated using malloc (I assume)... are you sure the intial native stack pointer is correct?

Interestingly, the compiler gives me this warning:

I'll answer this first because the rest can't be done properly with this in the way. The compiler can generate prolog/epilog code for a function:

SwitchContext:
               push ebp         ; save the previous frame pointer
               mov  ebp, esp    ; setup the frame pointer
               sub  esp, 8      ; space for local variables
               ...              ; function code
               mov  esp, ebp    ; restore the stack pointer
               pop  ebp         ; restore the frame pointer
               ret

Now it should be obvious what's the warning about. The compiler uses ebp for its own purposes and warns you if you change it in inline asm. This compiler generated code can be a problem when you want to implement context switching because it affects the stack layout. You can implement SwitchContext even if such code is present but creating the stack for a new thread will be a problem because you don't know what stack layout SwitchContext expects.

A possible solution is to use __declspec(naked) (which prevents such code from being generated) and __fastcall (which causes the first 2 arguments of the function to be passed in registers ecx and edx).

I'm a bit rusty on my x86 assembly, but what I don't understand is that the code is moving oldStack into eax, then a few lines later, esp is moved into eax. Is my translation of the original code even correct?

Nope. In the original version there were some parathesis which you ignored. Something like mov %esp, (%eax) converts to mov [eax], esp.

Here's an example that uses __declspec(naked) and __fastcall):

__declspec(naked) void __fastcall SwitchContext(VirtualThread* pOld, VirtualThread* pNew)
{
    // N.B. the following code assumes that NativeStack is at offset 4 in VirtualThread. If that's not true
    // then the appropiate offset needs to be used when storing/loading the stack.
    __asm
    {        
        // Save registers
        push ebp
        push ebx
        push esi
        push edi
        // Save old stack
        mov [ecx+4], esp    // store to pOld->NativeStack
        // Load new stack
        mov esp, [edx+4]    // load from pNew->NativeStack
        // Restore registers
        pop edi
        pop esi
        pop ebx
        pop ebp
        ret
    }
}

So what I'm missing right now is a way to set up these stacks for the initial run

To create a new thread you have to allocate memory for its native stack and setup the stack exactly the same as in SwitchContext does. Once the thread stack is properly initialized you can simply call SwitchContext to start the new thread.

In addition, my assumption here is that my VirtualThread class will not only hold the "virtual stack", but also this native stack that will be used to do context switching for each thread.

Yes, if you create thread you'll need to also allocate the native stacks yourself.

void InitializeContext(VirtualThread* pNew, void *startFunction)
{    
    int stackSize = 4096;
    char *stack = reinterpret_cast<char *>(malloc(stackSize)); // allocate some space, malloc is just an example
    stack += stackSize; // go to the top of the stack because the stack grows down
    
    __asm
    {        
        mov ecx, esp // save current stack pointer
        mov esp, stack

        mov eax, startFunction
        push eax    // push the start function address as the return address (of SwitchContext)

        // Save registers
        xor eax, eax // let's always start a thread with zeroed registers
        push eax
        push eax
        push eax
        push eax

        mov stack, esp
        mov esp, ecx // restore the stack pointer
    }
   
    pNew->NativeStack = stack;
}

Closer inspection shows that as far as the test on line IL_0017 goes, it will always end up going to line IL_001d, hence it doesn't even bother.

The test is not the problem, the problem is that the ldc 2 at IL_0019 is nowhere to be found in the generated C++ code. My guess is that the reason for this bug is the following:

I'm using an evaluation stack based approach that accumulates expressions until an instruction comes along that does some real work, like storing

The value (expression) that gets stored here was defined in two different places. Somehow you lost track of one of those definitions and ended up always writing 1. IL_001D can be reached with 2 different stack states but the generated code accounts for only for one.

I'm just curious, what method is used to compile IL to native code?

I don't know exactly what CLR does but what I'd do is convert the IL to some intermediate representation where there's no stack and where all instruction operands are explicit instead of implicit. Such explicit representation are far more easy to reason about.

You may say that you aren't writing a compiler but getting familiar with some compiler techniques would be a good idea. Reading the code generation chapter(s) from the "Dragon Book" (http://dragonbook.stanford.edu/) is advisable.

http://msdn.microsoft.com/en-us/library/ms686206.aspx

If so, I need to get a new temp variable and assigne it the value of the variable, and also replace the previous variable with the temp variable everywhere it shows up on the stack.

Mmm, seems like a cool solution to me.

 to step back through all of the previous instructions (up until the point it got pushed)

Hmm, this is very easy to do when the push and pop are in the same basic block. If they're in different blocks then things can get complicate. One thing that you should note about these temporaries is that they have limited lifetime. Once a temporary is popped it can be reused to store another value.

IMO, you're trying to get things working and optimize the code at the same time. I think it's better to just get things working first.

IL_0041: // ldind.u2
temp42 = ((UInt16)*(UInt16*)pStr);

Since we talk that the stack contains values and not variables, how did pStr got in this statement? It should be temp38 instead.

The IL execution stack contains values. When you do a ldarg (or any other ld) the value stored in the variable is loaded on the stack. After a ld you can modify the variable in any way you like, the stack will still contain the original value of the variable.

Your compiler treats the stack differently, it seems to store references to variables instead of values. This means that whenever you use a stack location that "references" a variable, you get the current value of the variable, not the original one that existed at the time ld was executed.

Let's see this in the code:

dupValue0 = pStr;
pStr = (Char*)(((Int32)dupValue0 + (Int32)2));
local0 = ((((UInt16)*(UInt16*)pStr) == 0) == 0);

Despite it being different, its behavior is exactly the same. The last statement still uses the modified value of pStr instead of using the original one. The correct code should look like this:

oldpStr = pStr;
pStr = (Char*)(((Int32)pStr + (Int32)2));
local0 = ((((UInt16)*(UInt16*)oldpStr) == 0) == 0);

PS: editor problems: don't paste from VS, it seems that it generates html that confuses the post editor. Paste the code to notepad and then copy paste from notepad. This ensures that you paste plain text and not some weird html.

Even if this is allowed I think it's still possible to deal with. If a basic block containing dup is reached with 2 different stack states then this can be treated as if there actually were 2 distinct basic blocks in the first place.

IL_003a: // ldarg.1 - load pStr
IL_003b: // dup - duplicate pStr, now the stack contains the value of pStr twice
...
IL_003e: // add - add one of the pStrs and 2
IL_003f: // starg.s: Char* pStr - store the value on top of the stack to pStr
// !!!the original value of pStr remains on top of the stack!!!
    pStr = (Char*)(((Int32)pStr + (Int32)2));IL_0041: // ldind.u2 - load from the address on top of the stack
// that is, from the original pStr
...
    local0 = ((((UInt16)*(UInt16*)pStr) == 0) == 0); // load from pStr - not the original pStr but the incremented one

The net result is that since you're using the incremented pStr you're one character ahead, this is where the -1 differences comes from.

but it is going to bloat the code a lot (or at least the source files) since I'll need to create a unique structure for each call.

Yes the sources will be a bit bigger but I'd say not by much. Note that you need one struct per method, not per call.

I wonder if it has something to do the the "dup" instructions.

Yep. If you look at the generated code for IL3A-3F you can easily note that dup has no effect. If you remove the dup then your compiler would generate identical code. That's a good indication that something isn't right.

So when I perform the dup operation, the two top items on the stack now contain "pStr".

dup make a copy of the value, it seems to me that your implementation makes a copy of some sort of reference to the value.

A better but more difficult solution: Any structs that have ref type fields also get the same hard-coded functions to manage their pointers just like classes get

That sounds good. And I might have a slightly different idea, treat the stack frame as objects:

• for each method create a struct that contains all the method parameters and locals
• when you call a method allocate such a struct on the stack (by using alloca or other means)
• fill the parameters
• call the method with a pointer to the allocated struct
• you'll also need to store pointers to these stack frames on that virtual stack of yours or simply chain the stack frames into a linked list (pointed to by the thread)

Pros:

• "everything" is an object, you don't need separate cases for "real" objects and stack frames
• this works transparently with value types that contain references
• this might avoid the register problem completly. if a method makes a "alloc" call the compiler won't be able to prove that the stack frame object hasn't been modified and it will have to reload any registers that have been previously read from stack frame

Cons:

• This will very likely disable some compiler optimizations. Usually some of the parameters are passed through registers, this can't happen anymore since the only real parameter will be the pointer to stack frame. Similarly for local variables.

That is certainly not "illegal" code and the compiler can't guard against it. Now if the original ref to MyClass goes out of scope, then it is no longer reachable and it will be GC'd

MyClass can't go out of scope, if one of its methods is running then there must be a stack frame for it and that stack frame must contain the "this" pointer for MyClass.

On a microcontroller it might be more important for the GC to not lock everything up for long periods of times instead of absolute performance.

Sounds like you want generational GC too. Even more complicated .

What I'm suggesting is improving on this process by stripping the fields out of types that were never referenced during all of the code paths.

Sure, you could do that. Not sure if it helps with the List<T> case. I looked at the code and it's indeed possible that CultureInfo gets dragged in by an exception (see the Capacity prop). But it's unlikely that you can eliminate that by doing some static code analysis.

It seems to me that if you don't provide the function type in the template argument list (like in the original code or in your code) then the only alternative is to use std::function or something that does something similar: wraps a callable object. And this wrapping is complex because there are different types of callable objects: pointer to function, pointer to member, functors.

and each of a function's ref type variables and parameters are really a pointer to alocation on this stack like this:

One thing I don't understand is what are you going to do about value types that contain pointers and are stored on the stack. Did I miss something?

that might not work because classes that are not reachable will never be called. If those classes are not rooted but they are still performing some sort of operations (maybe in some long running loop or a timer callback), their class variables will not be updated.

That should never happen, if an object is not reachable then it's garbage and it will be collected. If you pass a GC pointer to native code then you know what you need to do: use "fixed" or GCHandle to prevent the GC from moving/collecting the object.

So probably I will need to call UpdatePointer on each class while walking the object heap.

Watch out that moving objects and updating pointers can be extremly tricky. I'm not talking about finding the roots which can be itself a problem, I'm talking about how can you update pointers in an efficient way. One thing is clear: you can't move one object and then call UpdatePointers on all the other objects in the heap, that has quadratic complexity.

Since evildictait​or mentioned it: have you considered using Boehm GC instead?http://en.wikipedia.org/wiki/Boehm_garbage_collector

And my C++ code suddenly got hugely bloated with all sorts of seemingly unrelated classes like the CultureInfo

Hmm, not sure what CultureInfo has to do with List<>. Maybe the Sort method uses it somehow. Since you wrote that code you should be able to modify it to log some information about why a particular type has been included, right? If I understand correctly now you're including all the methods of an used type, you may consider excluding methods that are never called.

so I think it should be possible to make a VS debugging plugin. Has anyone had experience with this, and how difficult would such a project be?

Absolutely no idea but if you can make all the rest work (compiler, GC etc.) then you should be able to write such a plugin too

Best would be if you could make the optimizer inline , etc..

I don't understand... that's exactly what the compiler did. The code in main is the code from normal_function, 3 calls to the stream insertion operator and nothing else.