Neal Christiansen - Inside File System Filter, part I

Sign in to queue

Description

File System Filters are kernel-mode non-device drivers that monitor inbound and outbound FileSystem IO.

A prime example of an FSM is anti-virus software (the primary function of an AV app is to monitor IO stream content looking for virus patterns, after all).

Anyway, we were introduced to Neal by Dana Epp (he's working with the filter driver team to build a new security system and helped us during this interview) and we were impressed with Neal.

Why? Well, he's built two operating systems himself. More on that later, but hope you enjoy the first part of this, second part to come Monday.

Here, he takes you on a tour of the depths of Windows. Inside the kernel and the world of so-called kernel-mode drivers.

Embed

Download

Download this episode

The Discussion

  • User profile image
    Sven Groot
    Pretty interesting stuff. We need to have more videos like this, digging deep into Windows (or other MS technologies).

    It might be a good idea to have a video with someone from the kernel team explaining how the NT kernel works. Not everybody here has read books on that, and I'm sure many people would be interested. Even better, get someone who can really tell us why certain things are designed the way they are, and what the benefits or disadvantages of that design have turned out to be since the conception of NT. This video already does some of that, but it focuses on one area of the kernel. Not that that's really a bad thing, but I just think it'd be a good idea to try and paint the big picture too.
  • User profile image
    koorb

    Excellent video!
    Some of the videos don't have much useful content, but this is very educational.

    Neal mentions the Crash Analysis reports. Is there any chance of Channel 9 finding-out what happens to those automated reports, and why sometimes stuff is uploaded and other times not?

  • User profile image
    Charles
    Sven Groot wrote:
    Pretty interesting stuff. We need to have more videos like this, digging deep into Windows (or other MS technologies).

    It might be a good idea to have a video with someone from the kernel team explaining how the NT kernel works. Not everybody here has read books on that, and I'm sure many people would be interested. Even better, get someone who can really tell us why certain things are designed the way they are, and what the benefits or disadvantages of that design have turned out to be since the conception of NT. This video already does some of that, but it focuses on one area of the kernel. Not that that's really a bad thing, but I just think it'd be a good idea to try and paint the big picture too.


    Funny you should mention this, Sven. Smiley

    I am going to introduce a new series on Channel 9 in the relatively near future which I'm calling Deep Windows (but that may change). This video is in fact a precursor and your reply gives me even more incentive to make Deep Windows a reality. Also, I will be starting a more speculative and theoretical "interview" series that will include roundtable discussion among very big thinkers here at Microsoft. 

    It's going to be a deep year on Channel 9.

    Going deep,

    Charles 


    EDIT: I'm leaning towards calling the series Going Deep. Yes. That's it.
  • User profile image
    Charles
    koorb wrote:

    Excellent video!
    Some of the videos don't have much useful content, but this is very educational.

    Neal mentions the Crash Analysis reports. Is there any chance of Channel 9 finding-out what happens to those automated reports, and why sometimes stuff is uploaded and other times not?



    Glad you liked the video. There are more Neal vids coming soon.

    Perhaps we should interview some of the CA people. Wink
  • User profile image
    Jarod_24
    Great video, Just one question.

    If these Filters can now be unloaded at any time, what prevents someone from writing a virus that will unload the antivirus filter?

  • User profile image
    JamesC
    One big vote for Deep Windows here, as well as the roundtable.  There are things that I don't know to ask but affect me every day - let's hear it!  Keep it up and thanks for the C9 Guy Wink

    PS - Will we ever see Bill here?
  • User profile image
    Jerrold
    Compliments to Neal Christiansen on a well communicated overview of file system filters.
  • User profile image
    Fonze
    I would be extremely interested in seeing some interviews with people who will talk about the inner workings of windows. I'm focusing on OS's as part of my CS major, and the one thing they don't talk enough about are the inner workings of windows, we study mostly linux =/
  • User profile image
    Sven Groot
    Jerrold wrote:
    Compliments to Neal Christiansen on a well communicated overview of file system filters.

    I'd like to second that. I wish my professors were as clear. Wink
  • User profile image
    Gandalf
    This is a very good development Smiley I actually registered at this site anticipating this moment.

    By the way, who works on the kernel team at Microsoft? Is Dave Cutler still doing work on the NT kernel?

  • User profile image
    staceyw
    Very cool - thanks guys.

    Question One:
    Neal said they could not monitor all locks and such that a Mini-Filter may have, so the FM can not undu all that - makes sense.  However, could you try/catch around the callback for each MiniFilter and if an exception or bad return code, then unload that filter, post an Event log, and keep going?  Or would that still leave a bunch of locks and memory leaks out there?

    Question Two:
    Speaking theoretically.  Someone mentioned C#.  How might they allow something like a MF to be written in C#?  Thinking outside the box now.  I realize Kernel mode and no clr in Kernel mode.  But could a compiler and a special IO library be written, such that a c# program would compile into something that would run in kernel mode?  Thinking about computers getting faster.  Maybe some day, you could have a special Kernel Level-CLR that would allow a special version of the framework to be used to develop Kernel drivers.  Then drivers could be run in Kernel Managed code (KMC).

    Question Three:
    No mention of Dave Cutler on NT design.  Is he still around?  What is he working on now?

    Cheers and hats off to Neal and C9!

    --
    William Stacey [MVP]
  • User profile image
    The Channel 9 Team
    Dave Cutler is definitely still around. In fact, Neal mentions him on part II of this interview.
  • User profile image
    littleguru
    Could you also do some videos with Raymond Chen (blog: http://weblogs.asp.net/oldnewthing) he really knows a lot about Windows. It would be cool seeing him talk a bit about his experiences.

    His blog btw. is wonderful to read.
  • User profile image
    Dr. Shim
    Charles wrote:

    Funny you should mention this, Sven. Smiley

    I am going to introduce a new series on Channel 9 in the relatively near future which I'm calling Deep Windows (but that may change). This video is in fact a precursor and your reply gives me even more incentive to make Deep Windows a reality...


    Damn, that sounds nice. This is very surprising news indeed!
  • User profile image
    nealch

    You have to have administrator privilege to unload a minifilter. 

    The developers of minifilters can decide if they want to support unload (we encourage it due to JimAl's "no reboot" initiative).  They can also do additional authentication themselves to make sure a minifilter is being unloaded by someone appropriate.

  • User profile image
    nealch
    It is really not practical to try and caputue failures in minifilters and unload them.  It simply masks bugs in drivers and can lead to other strange things.  FOr example if someone had an encryption filter that crashed and was automatically unloaded you as a user might wonder why you can no longer access your encrypted data.  There is no way to handle all of this generically

    It is better to provide tools such that 3rd party developers can create quality drivers that don't have crashing bugs.  One of the things we are working on for longhorn is a comprehensive driver verifier for minifilters like we have for other drivers in the system.

    As far as C# goes in the kernel, you should talk to the device driver guys; they are thinking about this for the future.
  • User profile image
    Charles
    littleguru wrote:
    Could you also do some videos with Raymond Chen (blog: http://weblogs.asp.net/oldnewthing) he really knows a lot about Windows. It would be cool seeing him talk a bit about his experiences.

    His blog btw. is wonderful to read.


    Raymond does not want to be interviewed on camera and we respect that. Sorry. We tried.

    Charles
  • User profile image
    Charles
    Gandalf wrote:

    By the way, who works on the kernel team at Microsoft?


    Several people work on the kernel team (Neal is one of them) and you are going to meet more KernelPeople in the near future. Stay tuned.

    Charles
  • User profile image
    nektar
    If SP4 is the final service pack for Windows 2000 meaning that no other widespread updates will be issue for that os, how come you are going to update it with the latest file filter technology you have mentioned?
  • User profile image
    nektar
    Although your area of expertize is file filters, I would like to ask you why does not Windows support more file systems? At least for reading only. I mean other operating systems can successfully read and write to many file systems, not only NTFS and the legacy fat. Ok, not as reliably to all of them but still they have more interoperability support. If Windows has a better i/o architecture why isn't it more interoperable as well? Supporting more file systems like Unix/Linux ones, would enable us to access data that we have created in these oses, like let's say a diskette from a Linux system.
    Also, I read somewhere that the SDK for writing new file system driver or for directly working with NTFS costs $1000. Is that correct? And if yes why?
  • User profile image
    rhm
    You can code filesystems using the DDK, which MSDN subscribers can download. Non-subscribers can order it for the cost of the media.

    If you want to access a foreign filesystem just for light use (such as reading ext2 formatted floppies) it would be easier and safer to run the filesystem code as a library is usermode and interface it to the NT filesystem using re-parse points (roughly equivalent to the loopback device in Linux). I'm sure there's code out there that does this already, or at least there's code for running ext2 in user-mode so it wouldn't take long to put together.
  • User profile image
    staceyw
    Diskettes?  People still use those? Smiley  Have not touched one in about a year.  I think things like NFS, made it so you really don't need another driver.  Things like VMWare probably also reduce the need anymore.  I bet you could find one however.  I had thought most *nixes these days offer a DOS diskette ability - maybe not.

    --
    wjs 
  • User profile image
    Gandalf
    The Channel 9 Team wrote:
    Dave Cutler is definitely still around. In fact, Neal mentions him on part II of this interview.


    Brilliant Smiley

    Is there going to be an interview or a video with him?

  • User profile image
    littleguru
    Charles wrote:

    Raymond does not want to be interviewed on camera and we respect that. Sorry. We tried.

    Charles


    That's terrible. Thank you for the try.
  • User profile image
    Charles
    Gandalf wrote:
    The Channel 9 Team wrote: Dave Cutler is definitely still around. In fact, Neal mentions him on part II of this interview.


    Brilliant Smiley

    Is there going to be an interview or a video with him?



    Probably not, sorry. We tried... However, there WILL be some other kernel heavyweights coming to theatre near you Wink

    Please stay tuned.

    Charles
  • User profile image
    Andre Da Costa
    I still use a diskette for tranferring files since the XP CD Burning Wizard is so unreliable.
  • User profile image
    Sven Groot
    One word: USB Memory Stick

    ...

    Okay, two words and an acronym. Wink
  • User profile image
    androidi
    Charles wrote:

    Probably not, sorry. We tried... However, there WILL be some other kernel heavyweights coming to theatre near you Wink


    Uh, now I do not get it. Wasn't channel9 pretty much bumping to people around the places without asking ahead? Just go to where the big shots are and run into them. They won't get the chance to say no =)
  • User profile image
    Jaz
    how about a podcast with those who don't wish to be seen on video
  • User profile image
    eddwo
    I hope Longhorn at least gets a writeable UDF file system driver. It would be useful for random access archival storage systems like DVD-RAM and Iomega Rev.
  • User profile image
    Charles
    Jaz wrote:
    how about a podcast with those who don't wish to be seen on video


    It's more a matter of not wanting to be interviewed than a matter of not wanting to be on video. We respect people's right to not take part in this. After all, not everybody likes to be asked questions and have their answers shared with the world, be it on streaming video or audio-only. Maybe they are shy or simply just don't want to do it. It really doesn't matter what the reason is. Channel 9 is all about respect.

    Note that we seldom if ever just tape random people, though it does happen rarely, especially when we tour around product teams. For example, we interviewed Herb Sutter recently (VC ++ Architect and ISO C++ luminary) and ran into somebody in the hall that told us all about the experimental compiler framework called Phoenix.

    We always set up interviews ahead of time. 

    Charles 
  • User profile image
    Jaz
    of course i fully respect the wishes of raymond and everyone else, but it just sounded like he didn't wish to be videowed than interviewed.  His blog is pretty excellent though anyway.
  • User profile image
    eddwo
    I've just started reading "Showstopper!" and it appears Cutler always wanted to stay out of the limelight.

    "At Digital, he had given no interviews, and he insisted Microsoft never asked him to speak with the press. He even warned Gates: "If you bring the press in to see me, I'll do something that will make you never bring them in again." "

    Still its a fascinating book, I am glad I was able to get a copy second hand. 

    I've also just got a copy of "Windows Internals 4th Edition" which has a short piece by Cutler as an introduction, including a photograph of him and the authors.
  • User profile image
    Cd-MaN
    Who do you contact with regards to the Filter Fest?

    Also, beware that Yahoo! filters that registration e-mail as junk and keep up the good job.
  • User profile image
    Prasanna K

    Dear All,

     

    I am trying to see the video, but guess there is some problem could please suggest were i could get access to it now.

     

    Thank You.

     

     

    Prasanna.K

Add Your 2 Cents