Windows, NT Cache Manager - Molly Brown

Molly Brown, Going Deep...
Why am I getting flashbacks of Debbie Reynolds singing "Belly up to the Bar, Boys"...

+1 Vote for changing cacheing to favour applications over large file transfers.

I record programs off digital TV, they come in at about 1.4Gb an hour. Just transfering one of these to my networked storage means a lot of my applications page themselves out in favour of the file cache. I don't care if it takes an extra 5 minutes to transfer the file, I'd prefer to keep the programs responsive.

See this discussed before on Larrys blog.
http://blogs.msdn.com/larryosterman/archive/2005/03/25/402314.aspx#403814

To me this is one of the major causes of the occasional unresponsive feel of windows, the other is the freeze ups when browsing for network resources with explorer.

Does watching these videos count as doing revision for my Operating Systems exam next month?

I'm glad Molly mentions improving CopyFile() in Longhorn. A good improvement would be for it not to stomp all over the standby pages when copying a file that's never actually going to be read again. Just using a fixed number of pages when copying files (there has to be some size past which it makes no improvement to the copy speed) or restricting it to just use memory that is actually free unless free memory is too small would be a good idea. The current behaviour is where you have a bunch of applications minimized and Windows has pre-emptively shadowed them to the pagefile and then put most of them on the standby page list, then you initiate the transfer of a multi-gigabyte file from disk to disk and it kicks all the standby pages out of memory so the next time you try and use one of those minimised applications it can take forever to become active again (much longer than the application took to load in the first place and you can't even kill a swapped app on Windows).

Actually that behaviour of swapping out apps to grow the filecache on XP is wrong in desktop mode IMO. A page allocated to a foreground app should be regarded as 5 to 10 times more important than disk cache pages. XP is way too happy about paging applications. I have 1Gb of RAM and never use more than about 5-600Mb and yet applications are getting swapped. I can only imagine the cache manager is wanting to use more than what's left but how much is a 10-20% increase in cache size going to improve performance compared to waiting a minute for outlook to page back in when I switch back to it?

Another excellent interview here where we get to understand where windows meets computer science. I'd love to work on the kernel team. Charles is right when he says the improvements made in the kernel bubble up. More of these. If you need a hand to make them i have a camera and a lot of free time!

PS isn't it about time that we summise how all these various teams are building longhorn...it isn't just Avalon, the kernel team seems fixated. In the virtual server disks they were coy about their whiteboards and longhorn..leads me to suspect that some of their work is going into longhorn too...undo disks, or something along that architectural path.

Wow long rant

Drex

Indeed not... there was a musical called "The Unsinkable Molly Brown", made into a movie starring Debbie Reynolds.  It tells the true story of a woman named Molly Brown who grew up poor, struck it rich, and survived the sinking of the Titanic (hence, Unsinkable, Going Deep)

The author of the musical - Meredith Willson - also wrote another musical called "The Music Man", which was made into a movie starring Shirley Jones and little "Ronnie Howard" (as he was then credited)

Excelent series of videos! I've always been looking forward to learn more about Windows internals.

BTW, Does anyone knows where to find more about the Longhorn kernel improvements like the one Molly mentioned?

Raptor

My guess is that you will find info about kernel improvements in Longhorn over at http://www.sysinternals.com. Mark Russinovich and David Solomon wrote a summary article of the kernel improvements in Windows XP (http://msdn.microsoft.com/msdnmag/issues/01/12/XPKernel/default.aspx) earlier. And of course they have written the definitive guide to the Windows Kernel: "Inside Windows 2000"(http://www.sysinternals.com/insidew2k.shtml).

But it seems that Charles have got something more about the future of the Windows kernel. Looking forward to that

You can see what's going on in the cache currently using the kernel debugger's !filecache command. This seems to work OK with local kernel debugging on XP.

You will always see that 'current size' reported by !filecache is smaller than Task Manager's "System Cache" counter. That's because the System Cache counter includes, IIRC, the complete size of all standby and modified lists in the system, regardless of which working set they belonged to. It is true that, in effect, this is cached data - it takes only a single page fault and setting the PTE in the page fault handler to add it back to the working set, rather than having to go out to the disk (a soft fault rather than a hard fault). Again, IIRC, System Cache is actually the system working set size and also includes the current physical size of pageable system and driver code, and paged pool (system- and driver-allocated data that is not needed at high IRQLs).

I believe the standby and modified lists are actually double-counted, being also counted in the Available counter in Task Manager. That might explain why, on this 1GB machine, I apparently have 527,600KB Available and 584,640KB System Cache. !filecache reports a current size of 116,228KB.

What's puzzling me is why NTFS's transaction log ($LogFile) is being reported as having 28,064KB on Standby or Modified lists, making it the third-biggest consumer of file cache (after the master file table, $Mft, and the Software registry hive). Surely NTFS isn't going to need to re-read log data once it's committed to disk?

(It's understandable that $Mft is the largest user of cache once you realise that small files and directories are kept as resident attributes in their Master File Table record. This effectively hides those small files from the cache manager, which is a good thing, as the cache manager can only map multiples of 256KB blocks.)

(It's also understandable that the Software hive is the second largest user, as it's being accessed all the time by Explorer and by applications. XP and 2003, IIRC, use the cache directly - there is no other copy of the hive data in memory.)

Not a system-level developer, just interested! This information is from my recall of Windows Internals, 4th Edition (the updated version of "Inside Windows 2000").

I don't know if anyone else is interested, but I ripped the audio of this video (with Robert's kind permission) and posted it on OurMedia.  You can find it at http://www.ourmedia.org/node/9980

Charles wrote:

Maurits wrote: Molly Brown, Going Deep... Why am I getting flashbacks of Debbie Reynolds singing "Belly up to the Bar, Boys"...

I don't understand this. My hope is that it is not offensive.


C

A pun I believe:

http://www.debbiereynoldsonline.com/mollybrown.htm

hi,
how could a filesystem handle if the user opens one file twice and modify data, and will save the data at the same time? does fileobject 2 not override the content of fileobject 1 if you close it at the same time? maybe its a good idea to save the sector of the FAT into the RAM if the user opens a file...?

Will Molly Brown be attending PDC 2008? If so, I'd really like to get some guidance from her on how an application is supposed to handle error 1450 - ERROR_NO_SYSTEM_RESOURCES from ReadFile et. al.

e.g. http://blogs.msdn.com/psssql/archive/2008/07/10/sql-server-reports-operating-system-error-1450-or-1452-or-665-retries.aspx

We need a complete list of the API that can get this error, and a pattern for dealing with them.