MSDN Live Meeting - Visual Studio 2010 and .NET 4 Update

Great video! Though I had really hard time listening because of all the `yeah yeah yeah yeah` every few seconds throughout all the video :/

I agree with the previous poster that the "yeah yeah yeah" throughout the entire video was indeed very distracting. As a result you were difficult to follow at times and I had to rewind the video and replay it. In particular, as a presenter and mentor, you need to work even harder on your communication skills. Anything distracting, whether it is the occasional "uhm" or, in this case, the constantly repeated "yeah," hurts your ability to convey your teaching points to others. 

 

Additionally, commitments were made in the demo to post the source code on the presenter's blog on March 24th. As of today, March 26th, this has still not been done. If you commit to doing something before a very wide audience (i.e., Channel 9), make sure you follow through!

 

Now, let's move on to topics more relevant to this technical presentation.

 

The following is not only directed to you, but to all future presenters of anything involving comparisons of performance. The (mis)behaviors described below seem to be rampant, not only in this video, but in many of the presentations on related topics at the last PDC, as well. Without further ado:

1. Do not show performance comparisons of single and multi-threaded code in Debug mode. To make matters worse, not only were the comparisons made using Debug builds, but they were run in the VS debugger, further degrading their performance.
2. No production code should be running as Debug builds and any performance comparisons made using Debug builds are completely vacuous.
3. If you are going to compare the performance of two pieces of code, make sure that you are comparing Release builds with all sensible optimizations turned on.
4. The single-threaded code may have been fast enough, if only the demo had presented it using a Release build, to not even warrant the complexity and overhead of multi-threading.
5. Non-Release build comparisons are very deceptive of the true (relative) relationship between alternative implementations. The overall deception lies in the fact that, by prolonging the run time, in Debug mode, the percentage overhead of such effects in multi-threaded code as lock contention, false sharing, potential context switching due to oversubscription, and other overhead that more than one concurrent thread of execution may introduce, is often greatly diminished. This is unintentionally misleading at best and a downright lie to promote a certain multi-threaded approach, at worst.
6. When measuring the performance of your run, do not move the console window. Do not resize it, do not do anything to the app window other than passively monitor it using a performance tool. Keep your fingers off the mouse and off the keyboard until the run is complete. I can't begin to count how many presentations I've seen,  including this one and many at the PDC, where the presenter moves the console window around during a performance calculation. Do not do this! It impacts the performance of the system and therefore potentially the application being timed / monitored. Even if the impact is subtle, it is neither negligible nor constant in that it can be subtracted out after the fact! Pre-position your console window programatically at the start of the run, before any computations take place. Or, better still, pre-configure its placement before giving the presentation, so that it appears in the correct place on your screen and is of the appropriate dimensions for your demonstration.
7. When presenting the effects of multi-threading, explicitly state what hardware is being used, including whether or not it supports hyperthreading and whether this was enabled in the BIOS for the presentation. The number of physical cores present and their layout on physical CPUs, is also important to mention explicitly, even if it can (sometimes correctly) be deduced from the one or more performance monitors presented.
8. Speaking of performance monitoring, if you are going to use Task Manager or, for that matter, any other performance monitoring tool as part of your performance related discussion, the Kernel Time / User Time breakdown should be displayed and not just the total time. Seeing this breakdown is very important, especially when potential context switching and other kernel mechanisms (e.g., locking) are in play. Not seeing this breakdown, which is trivially enabled (e.g., View/Show Kernel Times in Task Manager), makes the presentation of system load seem superficial and disingenuous. Additionally, showing the total CPU load rather than the load induced by the process or processes being examined, individually, is also somewhat misleading. To truly show the proper system load for multi-threaded tests, PerfMon should be used and such counters as per Thread User / Kernel time, context switches per second, and other counters appropriate to the items being demonstrated should be included.

Let the above points be a lesson to this and all future presenters on this topic, whether on Channel 9, at a PDC, or any other venue.

 

Thanks for reading

 

 

For anyone interested in the scalability of the various approaches presented, here are the results of a proper 64-bit release mode run on a more modern box with 6GB of RAM and a single Intel Xeon W5590 CPU (3.33 GHz / 8MB L3 cache quad-core Nehalem-EP with hyperthreading turned on for a total of 8 logical CPUs). The build was done using VS2010 RC / .NET 4.0 RC and run on the host I just gave the specs for, under the Windows 2008 R2 Standard OS:

 

CalcPrimes
Count of prime numbers: 283146
Elapsed = 1713

 

ThreadingDemo.CalcPrimes
Count of prime numbers: 283146
Elapsed = 1080

 

ParallelFor.CalcPrimes
Count of prime numbers: 283146
Elapsed = 431

 

ParallelForEach.CalcPrimes
Count of prime numbers: 283146
Elapsed = 481

 

ParallelFor.CalcPrimes_TLV
Count of prime numbers: 283146
Elapsed = 430

 

ParallelForEach.CalcPrimes_TLV
Count of prime numbers: 283146
Elapsed = 447

 

LinqDemo.CalcPrimes
Count of prime numbers: 283146
Elapsed = 1740

 

PLinqDemo.CalcPrimes
Count of prime numbers: 283146
Elapsed = 461

 

Summary: It looks like, with a proper release 64-bit build, there are many options available here to gain the scalability, at least with this particular test and the specified number of CPUs.

 

Interestingly, if we make a slight improvement to the algorithm we can improve performance and most likely not even need to consider multithreaded approaches. Also, we can leverage the contents of the container we are forming without worrying about (and paying a concurrency penalty for) locking it, as follows:

        private static void CalcPrimes2()
        {
          List<int> primes = new List<int>() {2};
          
          for (int nr = 3; nr <= 4000000; nr+=2)
          {            
            int upperBound=(int) Math.Sqrt(nr);
            
            foreach (int prime in primes)
            {
              if (prime > upperBound)
              {
                primes.Add(nr);
                break;
              }

              if (nr % prime == 0)
                break;
            }
          }
          Console.WriteLine("Count of prime numbers: " + primes.Count);
        }

Running with these small changes, we get:

 

CalcPrimes2 (Single-Threaded)
Count of prime numbers: 283146
Elapsed = 501

 

Notices how this result is comparable to our best multi-threaded results.

 

For the next test, let's increase the number of numbers being tested for primality to 8 million and see how the various approaches fair:

 

CalcPrimes
Count of prime numbers: 539777
Elapsed = 4585

 

CalcPrimes2 (Single-Threaded)
Count of prime numbers: 539777
Elapsed = 1224

 

ThreadingDemo.CalcPrimes
Count of prime numbers: 539777
Elapsed = 2870

 

ParallelFor.CalcPrimes
Count of prime numbers: 539777
Elapsed = 1146

 

ParallelForEach.CalcPrimes
Count of prime numbers: 539777
Elapsed = 1210

 

ParallelFor.CalcPrimes_TLV
Count of prime numbers: 539777
Elapsed = 1144

 

ParallelForEach.CalcPrimes_TLV
Count of prime numbers: 539777
Elapsed = 1178

 

LinqDemo.CalcPrimes
Count of prime numbers: 539777
Elapsed = 4623

 

PLinqDemo.CalcPrimes
Count of prime numbers: 539777
Elapsed = 1183

 

Now, the slightly improved single threaded function (CalcPrimes2) is almost in line with out best parallel attempt, using the previous algorithm, despite the fact that this is a decent multi-processor host (i.e., 4 physical, 8 logical CPU cores) which allows for a lot of potential parallelism to the various decent parallel approaches presented in this session. This example (i.e., extending the domain of numbers being tested to 8 million) and how it scales as the workload increases, underlines the point about taking a little time to improve the algorithm first, before jumping with both feet into a multi-threaded solution.

 

If you are still not convinced, let's look at what happens when the domain is doubled again to the first 16 million numbers:

 

CalcPrimes
Count of prime numbers: 1031130
Elapsed = 12248

 

CalcPrimes2 (Single-Threaded)
Count of prime numbers: 1031130
Elapsed = 3017

 

ThreadingDemo.CalcPrimes
Count of prime numbers: 1031130
Elapsed = 7682

 

ParallelFor.CalcPrimes
Count of prime numbers: 1031130
Elapsed = 3054

 

ParallelForEach.CalcPrimes
Count of prime numbers: 1031130
Elapsed = 3119

 

ParallelFor.CalcPrimes_TLV
Count of prime numbers: 1031130
Elapsed = 3052

 

ParallelForEach.CalcPrimes_TLV
Count of prime numbers: 1031130
Elapsed = 3103

 

LinqDemo.CalcPrimes
Count of prime numbers: 1031130
Elapsed = 12318

 

PLinqDemo.CalcPrimes
Count of prime numbers: 1031130
Elapsed = 3106

 

In this run, the (improved) single-threaded solution (marginally) beats our best parallel attempts with the weaker algorithm.

 

Thanks for listening