Entries tagged with kernel - Channel 9

Mark Russinovich: Inside Windows 7 Redux

Charles — Thu, 22 Oct 2009 16:30:00 GMT

Windows 7 is here, available to all for purchase and ships today with new PCs! To celebrate this momentous occasion for Windows and Microsoft, Technical Fellow Mark Russinovich joins me in a discussion that extends the great conversation we had last year on Windows 7 internals. In his previous C9 interview, Mark told us about many of the new additions to the Windows kernel which enable Windows 7 (and Windows Server R2) to scale to large numbers of processors. Well, removing the kernel dispatcher lock is not all that the great Arun Kishan did. He also developed a new scheduling mechanism known as Distributed Fair Share Scheduling (DFSS). Mark describes what this is and how it works.

We also discuss NUMA, non-uniform memory access, (and Mark explains NUMA to us while showing a demo or two on a 256 processor machine!)

Moving on to Windows memory management, the domain of the great engineer Landy Wang, Mark discusses the new additions to the Windows Memory Manager and explains why they matter to those of us who spend all of our time and in user mode.

Learn about all of this and much more as Mark digs into the insides of Windows 7, way deep down in the system (the culmative effects of which help to make Windows 7 Microsoft's most reliable, scalable and efficient general purpose operating system to date). As usual, Mark explains very complex mechanisms and concepts in a readily understandable way. This is a very conversational piece and we cover a lot of ground in a relatively short period of time. We also learn exactly why Mark is so passionate about operating systems and what the spark was that set off his passion and curiosity of how things work internally.

Mark will be presenting at PDC09 in the Technical Leaders track and the free Windows 7 Developer Boot Camp. His talks will be very deep and will explore all aspects of the new, improved Windows 7 kernel. I highly recommend that you attend both of these talks if you are going to PDC (you're going, right?!).

Check out the Windows area on 9 for more great Windows 7 content, all rolled up into a nice experience!

Enjoy!

Note: Check out all the 9 Guys Mark has. :) Also, you should subscribe to his incredible blog.

Silviu Calinoiu: Inside Windows 7 - Fault Tolerant Heap

Charles — Tue, 25 Aug 2009 05:35:00 GMT

The Fault Tolerant Heap (FTH) is a subsystem of Windows 7 responsible for monitoring application crashes and autonomously applying mitigations to prevent future crashes on a per application basis. For the vast majority of users, FTH will function with no need for intervention or change on their part.

Principal Development Lead and rock star developer Silviu Calinoiu is the mastermind behind FTH. Here, we go deep into how FTH works and why it's designed the way it is.

The Fault Tolerant Heap is another example of the low level efficiency built into the system: FTH automatically corrects memory faults that cause applications to crash which has the pleasant side effect of preventing future crashes. How does FTH work, exactly? What types of memory problems does it address, specifically? How do developers monitor FTH events and can they override FTH's behavior? What does this all mean to the average user?

FTH, as an autonomous monitoring and correction system, represents a step in the right direction for the evolution of a more homeostatic general purpose operating system. Simply, Windows is getting smarter in the sense that it's increasingly becoming better at self-regulation and self-healing. Yes, there's a very long way to go, but we're making real progress.

You will continue to learn about recoverability in Windows over the coming months here on C9.

Tune in.

Arun Kishan: Inside Windows 7 - Farewell to the Windows Kernel Dispatcher Lock

Charles — Thu, 06 Aug 2009 17:28:00 GMT

You've learned about many of the new features of the latest version of the Windows kernel in the Mark Russinovich Inside Windows 7 conversation here on Channel 9. One of Mark’s favorite kernel innovations is the way the Windows 7 kernel manages scheduling of threads and the underlying synchronization primitives that embody kernel thread management.

Prior to Windows 7 (and therefore Windows Server 2008 R2) the Windows kernel dispatcher employed a single lock, the dispatcher lock, which worked well for a relatively small numbers of processors (like 64). However, now that we find ourselves in the midst of the ManyCore era, well, 64 processors aren’t that many... A new strategy was required to scale Windows to large numbers of processors since a single lock is limited in capability, by design: The masterful David Cutler, one of the world's greatest software engineers, wrote the NT scheduler in a time when the notion of affordable 256-processor machines was more science fiction than probable.

As we learned in the Mark Russinovich video, Windows 7 can now scale to 256 processors thanks to the great engineering of Arun Kishan, a kernel architect you've met on C9 back in the Vista days. In order to promote further scalability of the NT kernel, Arun completely eliminated the dispatcher lock and replaced it with a much finer grained set of synchronization primitives. Gone are the days of contention for a single spinlock. How did Arun pull this off, exactly, you ask? Who is this genius? Well, tune in. Lots of answers await…

Arun's work directly benefits the overall performance of Windows running on many processors and means, simply, Windows can now really scale. Thank you, Arun!

Spinlocks are synchronization primitives that cause a processor to busy-wait until the state of the lock’s memory location changes.

As the name implies, the dispatcher lock is the fundamental lock associated with the kernel dispatcher, or the scheduler.

Michael Fortin: Windows 7 Efficiency

Charles — Wed, 22 Jul 2009 20:42:00 GMT

The Windows 7 project involved very efficient software engineering planning and execution. It is no surprise that an equivalent level of efficiency exists throughout the OS (efficiency in how the OS deals with faults, threads, memory management, power management, process management, window management, graphics, audio, local search, diagnostics, and on and on - truly excellent, and efficient, engineering).

Michael Fortin is a Distinguished Engineer in the Windows Core Operating System Division. His team builds the technologies that help make Windows 7 reliable, stable and performant, which are core ingredients in any highly efficient general purpose operating system. You'll hear us talk about Windows 7 as a very efficient general purpose operating system quite a bit over the coming months. In fact, if I had to sum up Windows 7 in one word it would be Efficient.

Michael's team also builds the troubleshooting and diagnostics systems in Windows, including the internal mechanisms that construct fault data packages and sends them to cloud-based components which receive data from millions of clients running Windows 7. Michael's team is a global team - engineers are located in multiple places around the world including a stellar team of engineers located in Beijing, China (you'll meet them in the future right here on C9).

You may remember Michael from his last interview on Channel 9 that covered his work on Vista's SuperFetch and ReadyBoost technologies. Yep, these great technologies are alive and well in Windows 7 and have evolved to meet the needs of the evolving system and help add to the overall efficiency of Windows. (There, I wrote "efficiency" again...)

Over the past year or so, Michael's team has received, analyzed and acted upon a very large amount of data sent from Windows 7 Beta and RC running on a variety of PCs with a variety of hardware and software configurations in place. This data was used to construct new system features, like the Fault Tolerant Heap, and to engineer updates to existing mechanisms to make them more robust or performant or reliable or stable... You will meet some of his team here on C9 in the future and we will dig into many of the mechanisms Michael touched on in this conversation (Fault Tolerant Heap, Troubleshooting and Diagnostics, etc).

Here, Michael and I chat about the work his team has done, the engineering philosophy that has driven efficiency into Windows at all levels (from the kernel to the shell), the knowledge his team has gained about how Windows is used in the wild, what the most common problems have been and the solutions that are based on this important telemetry data. So, for all of you out there who chose to send fault data from your PC to Microsoft - THANK YOU. You truly have helped, in a fundamental way, to make Windows 7 the most efficient general purpose operating system from Microsoft to date. Yeah. True story.

Enjoy.

Mark Russinovich and David Solomon: Windows Internals 5 Released

Charles — Fri, 19 Jun 2009 16:01:00 GMT

Windows kernel expert and kernel "professor" David Solomon and Windows Kernel Technical Fellow Mark Russinovich have written another great book covering, in great detail, the internal composition of Windows Vista and Windows Server 2008. A third author,Alex Ionescu, joined the fray this time around. Together, they've just released Windows Internals Fifth Edition. My order has been submitted! If you want to really understand the mechanics of Windows' latest generation general purpose kernel, then go get this book.

David was in Redmond recently conducting deep training on the Windows 7 kernel, which contains 95% of the same ingredients as the Windows Vista kernel. So, of course we had to get the dynamic duo together to talk about their book, the Windows kernel, their history (they've been working together for a long time) and their future.

Enjoy! Happy reading.

Mark Russinovich: Pushing the Limits of Windows - Paged and Nonpaged Pool

Charles — Thu, 02 Apr 2009 22:39:00 GMT

Paged and Nonpaged Pool.

Mark's posts are stellar. The detail, the clarity, the usefulness. Definitely one of my favorite technical Windows-related blogs. The post I linked to specifically is incredibly interesting given that it details a rare, but very painful problem and in process teaches us about some fundamental properties of the Windows kernel (see his other Pushing the Limits posts on Physical and Virtual memory).

C

Dave Probert: Inside Windows 7 - User Mode Scheduler (UMS)

Charles — Mon, 02 Feb 2009 20:11:00 GMT

Here, we continue our exploration of the morphology of Windows 7 on Going Deep with windows kernel architect Dave Probert. You may remember him from an early four part episode of Going Deep where he teaches us about general purpose operating system architectures and history: Part 1, Part 2, Part 3, Part 4

That was a great conversation from a few years ago and it's been way too long since we returned to Windows kernel world to converse with and learn from Dr. Probert. Not surprisingly, Dave has been busy innovating the Windows core.

Dave and team, working very closely with the Parallel Computing Platform People, have created a very compelling new user mode thread scheduling/management system in Windows 7. In a nutshell, the User Mode Scheduler provides a new model for high-performance applications to control the execution of threads by allowing applications to schedule, throttle and control the overhead due to blocking system calls. In other words, applications can switch user threads completely in user mode without going through the kernel level scheduler. This frees up the kernel thread scheduler from having to block unnecessarily, which is a very good thing as we move into the age of Many-Core... Speaking of Many-Core, remember the piece we did on the Concurrency Runtime (ConcRT)? ConcRT is built on top of UMS and is the best way to most effectively utilize this new user mode thread scheduling model in Windows 7.

Make yourself comfortable and spend some time watching and listening to Dave make all of this crystal clear.

This is another great conversation with a fantastic OS architect and Windows kernel professor. Lots to learn here. Enjoy.

Chittur Subbaraman: Inside Windows 7 - Service Controller and Background Processing

Charles — Fri, 23 Jan 2009 19:39:00 GMT

At any given point, Windows is executing a lot of code. Some of this code runs in the background as services. In pre Windows 7 operating systems some services were set to auto run even though the configuration of the system (installed apps, hardware, etc) did not warrant them running. This had a four-fold potential effect:

1) Windows might start up slower as the service fires up
2) Users may experience sluggish performance as the service runs taking up processing and memory resources
3) Windows might take a long time to shut down as the service shuts down, unwinding itself and cleaning up it's resources
4) The surface area for code-level security breaches is larger(though, since Vista, most services run in a restricted security context)

Chittur Subbaraman, Windows kernel developer extraordinaire, and team spent a great deal of time thinking about and rectifying these problems by re-architecting the Windows 7 Service Controller. They also identified services that don't need to auto run (like a TabletPC Pen service that need not ever run on a desktop (non-Tablet) machine by default). But they went much further than simply figuring out which services can be set to manual start-up state in Windows 7. They added a new feature for service developers based on the trigger pattern: services can be started and shut down via triggers - this means developers are able to specify programmatically when a service needs to start or stop. This allows Wndows to control services in a much more dynamic way so less code has to run in any given user session. The Service Controller monitors and reacts to trigger events as opposed to just running services marked as auto when the system starts. Less code running in the background on Windows means more resources available for foregrond processing, faster start up of sessions and faster shut down.

The great work in the Windows 7 service controller by Chittur and team has a direct impact on the performance of Windows 7. Tune in to learn about the details and history of the service controller (and Task Manager).

Here are some great resources for you to read to get the details behind all of this great engineering in the background processing mechanisms deep inside Windows 7.

·White paper on Designing Efficient Background Processes.

·PDC talk on Designing Efficient Background Processes.

Mark Russinovich: Inside Windows 7

Charles — Wed, 14 Jan 2009 19:39:00 GMT

How has Windows evolved, as a general purpose operating system and at the lowest levels, in Windows 7? Who better to talk to than Technical Fellow and Windows Kernel guru Mark Russinovich? Here, Mark enlightens us on the new kernel constructs in Windows 7 (and, yeah, we do wander up into user mode, but only briefly). One very important change in the Windows 7 kernel is the dismantling of the dispatcher spin lock and redesign and implementation of its functionality. This great work was done by Arun Kishan (you've met him here on C9 last year). EDIT: You can learn exactly what Arun did in eliminating the dispatcher lock and replacing it with a set of synchronization primitives and a new "pre-wait" thread state, here. The direct result of the reworking of the dispatcher lock is that Windows 7 can scale to 256 processors. Further, this enabled the great Landy Wang to tune the Windows Memory Manager to be even more efficient than it already is. Mark also explains (again) what MinWin really is (heck, even I was confused. Not anymore...). MinWin is present in Windows 7. Native support for VHD (boot from VHD anyone?) is another very cool addition to our next general purpose OS. Yes, and there's more!

Tune in. This is a great conversation (if you're into operating systems). It's always great to chat with Mark.

Richard Ward: Windows Architecture - Past, Present and Future

Charles — Fri, 07 Nov 2008 17:30:00 GMT

Artists embody a diverse set of talents and interests. This gives them a rich array of experiences to draw upon in their work. In the same way, Richard Ward brings an eclectic background and collection of interests to his efforts at Microsoft. He brings this diversity to his own art of programming within the Windows Core Architecture team.

While most Windows developers are focusing on the next version of Windows, Richard and his team are already hard at work on the version which will follow. In this episode of Behind The Code, we will discover what experiences Richard finds most helpful, as he focuses on building out the core infrastructure components of what will, one day, be running on the vast majority of the world’s computers.

Read Richard Ward’s { End Bracket } column in MSDN Magazine.

Mark Russinovich: On Working at Microsoft, Windows Server 2008 Kernel, MinWin vs ServerCore, HyperV,

Charles — Fri, 14 Dec 2007 19:01:00 GMT

I recently sat down with Technical Fellow and SysInternals founder Mark Russinovich to dig a bit into what's new in the Windows Server 2008 kernel. Of course, we talk about many things including HyperV, application virtualization, kernel architecture (not everybody defines an OS kernel in the same way - tune in to understand why this is the case. Mark has his own definition that may not be the same as yours....).

Recently, the MinWin project was in the press after a university video lecture by a Microsoft Windows architect was released on the net. Most people confuse MinWin with Windows Server 2008's ServerCore technology - the confusion stems from the incorrect assumption that ServerCore is a byproduct of the MinWin work. In fact, they are not at all related. Mark explains the differences and hopefully this will end the confusion...

Of course, Mark spends time on the whiteboard in this interview, drawing out the kernel architecture, explaining HyperV, touching on application virtualization (running client applications without having to install them locally - tune in to understand what I mean...).

Channel 9 is and has always been about showcasing the humans behind our technologies in addition to drilling into how we make our products, and of course why we do what we do (in a technical sense). Mark is a huge addition to the Windows family and his technical leadership is already being felt throughout buildings 26 and 43. Mark tells me about how life is going inside the Mothership, what a Techincal Fellow is (it's the highest level of engineering career stage at Microsoft), individual contribution versus management, and more.

As always, it's an honor and pleasure to spend time talking with Mark. He's one of our brightest technical minds and Windows architecture is in very good hands.

Enjoy.

Mark Russinovich: From Winternals to Microsoft, On Windows Security, Windows CoreArch

Charles — Fri, 23 Mar 2007 22:21:00 GMT

If you write code on Windows or like to know what goes on under the hood in Windows, then you've no doubt heard of Mark Russinovich. He's an OS kernel expert and a co-founder of Winternals; a company that produced must-have operating system and development utilities for Windows (Winternals is now a Microsoft subsidiary as we purchased them in July, 2006. Yay!).

Mark is now a Technical Fellow in Windows and is a member of the Windows Core Architecture team (you met some of the here.

Singularity III: Revenge of the SIP

Adam Kinney — Fri, 18 Aug 2006 21:16:17 GMT

It's been a while since we checked in with the Singularity folks over in MSR Redmond. You know, the usual suspects like Jim Larus and Galen Hunt. So, Charles went and visited them recently to see where they are with Singularity, see what they're up to, what's new... As you may expect, they are doing some really interesting work with, among other novel software constructs, SIPs (Software Isolated Processes). Learn all about them and how they are architected into the Singularity system. Yes, we get Galen on the whiteboard again. This time, you will also meet the newest member of the Singularity team, Mark Aiken, Software Developer, who's been working on some interesting hardware protection stuff in Singularity. It's always great to spend time with the Singularity folks. Oh yeah, this time we get http://blogs.msdn.com/doronh/

KMDF homepage: http://www.microsoft.com/whdc/driver/wdf/KMDF.mspx

KMDF bits (v1.1 right now): http://www.microsoft.com/whdc/driver/wdf/KMDF_pkg.mspx

WDF: (UMDF, verification tools): http://www.microsoft.com/whdc/driver/wdf/default.mspx

Going Deep: Richard Ward - Engineering security into Windows Vista

scobleizer — Fri, 14 Apr 2006 22:07:06 GMT

Richard is an architect on the kernel team on Windows Vista. In other words, everything in Windows builds on top of his team's work. Here Charles Torre has a 52 minute conversation with him about the kinds of things that are being done deep inside Windows from a security perspective. But you'll learn more about the innards of Windows than you might expect from this conversation.

Virtualization

Charles — Tue, 14 Feb 2006 22:07:46 GMT

Ever wonder how virtualization works? Where is it used? What role could virtualization play in the future of Windows, for example? How will it evolve over time?

We recently met with some of the Microsoft Virtualization people to find some answers to these questions and learn a whole lot about virtualization, hypervisor, and the future of this super cool and important technology.

You can get more info on virtualization products and demos here and here.

IT Heroes Episode XII: Dave Probert Windows Kernel Architect

mjmurphy — Mon, 13 Feb 2006 18:13:11 GMT

Dave Probert Windows Kernel Achitect

Until recently Dave managed kernel development for Windows, including Windows 2000, XP, Server 2003, and early phases of XPSP2 and Vista. Dave is currently working on a project to release kernel sources to universities and developing ProjectOZ, an experimental environment based on the SPACE project at UC Santa Barbara, where Dave earned his PhD in Electrical & Computer Engineering.

Dave is a co-author/consultant of a Chinese textbook on operating system principles illustrated with Windows, and for the last several years has taught a short course with Prof. Kei Hiraki at University of Tokyo on Windows Internals, and spoken widely at universities on the architecture of the Windows kernel. Prior to Microsoft, Dave's experience was primarily focused on UNIX kernels, including several years as Vice President of Software Engineering at Culler Scientific Systems. Dave's career began in the late 1970s at Burroughs, where he was a computer architect designing hardware and writing microcode for the B1900.

In this interview we discuss the history of modern operating systems and how we got to where we are today. Dave also passionately evangelizes Windows Curriculum Resource Kit project that releases over 500,000 lines of source code of the NTOS Kernel (Windows 2003 Kernel) to Academia for education and research, which is currently available for download from the Windows Academic Alliance website.

IT Heroes: Stories from the Trenches; these are the real stories of men and women in IT making a difference everyday. We talk with authors, innovators and implementers about emerging technologies, troubleshooting and remediation of common infrastructure issues and charitable contributions in the community. We seek to educate and provide a forum for open discussion of the many uses for and specific ways in which people are everyday exploiting technology to create opportunities for themselves and others.

Stream the Dave Probert interview now!

Originally posted Friday, February 03, 2006 1:57 AM by MJMurphy_TechNet at http://blogs.technet.com/mjmurphy [6]

Eugene Lin and Jason Cobb - Windows Plug and Play

Charles — Wed, 18 Jan 2006 19:56:24 GMT

Ever wonder what's really going on when you attach a peripheral device to your computer (running Windows XP or Vista)? Well, we wanted to find out the low level truths about Plug and Play in Windows. Who else better to talk to than Eugene Lin and Jason Cobb, the Plug and Play Guys ( Eugene is a Program Manager and Jason is a dev lead, both are on the Device Management and Installation team ). Join us ( Charles Torre and special guest Channel 9 correspondent/Technical Evangelist Jeremy Mazner ) as we discuss how PnP works and what's new for Vista in this most interesting and oft overlooked killer feature of Windows.

Rob Short (and kernel team) - Going deep inside Windows Vista's kernel architecture

scobleizer — Fri, 23 Dec 2005 22:47:40 GMT

Rob Short is the corporate vice president in charge of the team that architects the foundation of Windows Vista. This is a fascinating conversation with the kernel architecture team. It's our Christmas present to all of the Niners out there who've stuck with us day after day.

This is a very candid interview. We even ask "do you ever wish the registry had never been developed?" Charles Torre does this "going deep" interview. Out of all the interviews we've done this one is the most interesting because this team has such a deep impact on how reliable, scalable, secure, etc Windows Vista actually will be.

In this video you'll meet:

Darryl Havens, Architect.
Richard B. Ward, Architect.
Rich Neves, Architect.

Enjoy! And, don't be suprised if your brain hurt the way ours hurt after having this conversation.

ARCast - Transactional File System and Registry

Charles — Wed, 14 Dec 2005 04:04:09 GMT

Transactions have long been the domain of the database but what if you could transact virtually any resource in your application? How would that change the architecture and design of the systems you are building? Well get ready because Windows Vista includes a powerful new transaction capability for the file system, registry and virtually any kind of kernel object. Listen as Ron gets the scoop on the new transaction features of Vista from Dana Groff.

Dana is the Program Manager in Core File Services focusing on transaction technologies. As the PM for Transactional NTFS, the Kernel Transaction Manager, and Common Log File System he coordinated building this new kernel transaction infrastructure for Longhorn with Microsoft’s existing user-mode transaction infrastructure. Dana is a veteran of the software industry having worked for DEC, IBM, Sybase and much smaller companies as well over 21 years in industry developing OLTP and Database middleware.

Links:
Malcolm Smith's Blog
Jim Johnson's Blog
File System Team's Blog for Developers
File System's Blog for IT Pro's
Florin Lazar's Blog

Surendra Verma: Vista Transactional File System

Charles — Fri, 02 Dec 2005 19:51:41 GMT

Surendra Verma, Development Manager on the Vista Kernel team, digs into Vista's new Transactional File System whith Charles Torre. TxF, as it is referred to internally, is a new kernel construct that is part of an updated Vista NTFS. Surendra provides a high level overview of TxF in this video. We will continue to look under the hood in future Going Deep episodes with interviews with developers working on the Kernel Transaction Manager, a key component of TxF, which Surendra touches on during the white board session, which is a Going Deep requirement.

Enjoy.