Maoni Stephens and Andrew Pardoe: CLR 4 Garbage Collector - Inside Background GC

Fantastic video, really interesting and a great accomplishment the .NET GC is.

I also remember that native vs. managed battle, really very amazing how fast .NET was, in the end the native won but only by writing some lightweight task specific versions of low level citizens, such as allocation, strings and file io..

Yeh, the thing with a lot of native developers (not Raymond Chen, but in general) is they look at GC and think "wow, that's kinda complex, that must take a long time", but they forget that memory management takes quite a lot of time however you do it and malloc/free can be slower than a good generational compacting GC - the only way native gets faster is by writing custom allocators and structuring memory patterns so that large chunks of memory can be free'd in one go. Example: in a 3d engine, all the memory allocated while rendering the frame would be bump-allocated from a single chunk and then at the end it all gets free'd in one go by just resetting the pointer back to the start of the chunk.

Anyway, awesome video. I'm really in awe of the brains behind the CLR, but particularly the GC, because the 'magic' that makes GC work has always been fascinating to me.

I'm currently designing a very simple .NET runtime for a certain embedded platform ( ) using LLVM and trying to get my head around just a single-threaded 'stop-the-world' (what Maoni calls 'blocking') compacting GC is hard enough. I don't even want to think about how complex it gets with threading - just allowing threading of user code, never mind making the GC multi-threaded. This, along with kernel work, really is the rocket science of programming.

thats some powerful voodoo.. great stuff
also fun to see a women in a really technical interview, that doesnt happen often enough imo  wm_in is good but technical it aint

Hahaha, "whiteboarding is fun... unlike waterboarding" lol Charles is awesome.

Great interview, Charles!

Did the GC team talk with MySpace, one of ASP.NET's biggest adopters?  This InfoQ interview (http://www.infoq.com/interviews/MySpace-Architecture-Dan-Farino) indicated that they had performance issues with the .NET garbage collector:

One of our original problems was with the .Net garbage collector. It is just really not designed even on a 64 bit platform to reverse billions of objects and collect and compact without causing at least a noticeable delay. So one of our scaling problems was "Hey, we are going to have to write an unmanaged memory store."

Perhaps MySpace is one of the "external customer" scenarios they test against?

http://blogs.msdn.com/ricom/archive/2005/05/10/performance-quiz-6-chinese-english-dictionary-reader.aspx is Rico Mariani's blog post that began his series of posts where he compared the performance of his managed code solution to Raymond Chen's native code solution.  It also includes a handy set of links to Raymond's posts on the subject.

Great video indeed!

Charles, close to the end Andrew mentioned that SQL team is working on putting more CLR into SQL Server.

Is there a chance to interview anyone on the SQL team on that? Are we to expect T-SQL being (finally) executing on the top of the CLR (after all, what's the point of having two VMs on a database server doing essentially the same type of work)? Are we (finally) to see a decent programming language that can be used to do rich data processing inside the database server (code near) instead of (frankly utterly outdated and feature deprived) T-SQL? Something like F# with LINQ directly to relational engine (bypassing SQL all together) would be a blessing (after all a relational engine does not have to be SQL).

Cheers,

Seva.

Nice work. But I will still hold back on putting most of my business logic in .NET, so far SQL Server managed this job very well. It scales well and does a good job in multithreading, so I usually put most of the complex BL in SQL Server when things get slow, upgrading hardware is cheaper than building your own version of GC, like MySpace did. I know HPC does a good job balancing load, I don't know however how well it integrates with SQL Server. If not then SQL Server Compute Cluster would be nice LOL.

And yes some native F# like language for SQL Server would be great and I mean great. Because T-SQL is clumsy.

In fact when I first demoed LINQ to my team some 2.5 years ago they were not excited at all, all of them one after the other were saying it is Visual FoxPro 10 years ago. Of course FoxPro does not have nice security model, but language integration and dynamism was similar to modern Ruby, F# and LINQ