Comment Feed for Landy Wang - Windows Memory Manager (Going Deep on Channel 9)

Re: Landy Wang - Windows Memory Manager

Geoff Joy — Sun, 17 May 2009 19:17:29 GMT

So if the VMM is so smart, when will Internet Explorer NOT leak memory? One of the most aggravating things about IE is that it steadily expands to fill all available memory, degrading the system over time. This is true for IE7 and IE8 and is clearly observable in Task Manager. Exiting all instances of IE doesn't free it all and eventually the only solution is to reboot.

Re: Going Deep: Landy Wang - Windows Memory Manager

bosko — Sat, 08 Jul 2006 16:34:39 GMT

Beer28 wrote:

...
I can't help but notice that even though swap is enabled on linux, it's hardly ever used because I have alot of ram. It's not turned off. But linux will not use the disk swap partition space unless it's necessary. I can have 1 gig of ram on windows and have notepad open and it will still use the swap file if I have it enabled. It obviously doesn't need it, why is that?

Why is swap space on windows used when you have enough physical ram to handle the processes? As opposed to it not being used until it's necessary on linux?

This is actually really crappy behavior on the part of Linux. In my experience, in real load scenarios, the behavior of swapping out entire processes only when under heavy load tends to lead to trashing. The system spends most of its time swapping out entire processes only to have to struggle to swap them back in so they can get their share of CPU time, to only then swap them back out again, etc.

Lots of systems, not just Windows, will page out unused pages ahead of time. In certain cases (like heap pages which tend to be backed by swap), this paging out is pushed to swap. So it's normal to see swap in use even though there are lots of free pages hanging around (i.e., lots of free RAM) -- on FreeBSD, or similar, this is what happens.

-B

Re: Landy Wang - Windows Memory Manager

fparis — Sun, 21 May 2006 09:51:09 GMT

Hi there:

I´m a end user from Venezuela in the multimedia field, (using 3dmax, premiere, AfterFX, photoshop and so on), with a basic survival knowledge of windows set up. Mi system is a custom ordered Pentium D on Asus MB with 2 gig of ram, and lots of disk space. As you may know, all the applications I use are memory-cannibal, so I don´t know if I can disable the swap memory without affecting the performance of the programs. It´s there some sort of a program that "cleans" the swap memory after exiting an application without having to restart windows? I ask this because some times after a heavy use of some of this programs, the performance of the computer comes down to a crawl, which it´s noticeable in the update of the widows in the win explorer, for example.

Thanks in advance for any advice on this issue.

Re: Going Deep: Landy Wang - Windows Memory Manager

Animadei — Tue, 03 Jan 2006 11:38:55 GMT

androidi wrote:
I haven't tested whether there's a real perf benefit for this, but have always thought that it's optimal to have one drive/array for system and apps and second for pagefile and backups/infrequently accessed data. I am talking about what is the optimal 2 hdd setup for both performance and some redundancy - special arrangement like this or some special RAID that gives both perf + redundancy on just 2 drives?

Haha you're fanny! Just add more memory and avoid page files altogether. There's a product like SuperCache that does Disk I/O level caching. All of course at the expense of reserving more memory for these operations. Personally I gave up page files after disk thrashing really put a burden on performance, especially when we're talking about compiling and linking 800MB worth of source codes every 15 minutes. The only reason you get "more" performance boost by putting the paging file on the second drive is that whenever you access your system the first disk is busy and the second drive is free to work. If you really think about this, to really get the benefits, at least for IDE "technology", you needed to put each device on its own separate channel. Basically one disk per ribbon. But then again, reality hits you again. We're talking about many orders of magnitudes of slowness because the disk is just so slow. You should look into borrowing memory from other systems over the network, if you're kinda cheap, since accessing them is still much faster than your local disks.

Re: Going Deep: Landy Wang - Windows Memory Manager

Animadei — Tue, 03 Jan 2006 11:24:57 GMT

Beer28 wrote:
Does the memory manager work with the kernel scheduler?

If a process releases it's timeslice as soon as it's started because it's a service or never accesses a good portion of memory or is in a sleep wait state, is it's heap more likely to be paged out?

What about non-paged driver memory? How does the memory manager handle that? ISR's can't use paged memory, are they still handled by the memory manager on windows TM?

Heap is always most likely to paged out. That's why we have working-sets to keep processes happy.

ISRs cannot use paged memory as a general rule, and a good one if you care about performance. No fun waiting for the file system to respond in at least 3 orders of magnitude delays. With that said, non-paged memory is simple: Don't use it if you don't absolutely need it. These are non-paged and thusly subtracted from your total free pool. When you unload your driver, all the allocated memory goes back to the system.

The memory manager only cares about allotting free pages and reclaiming pages that are committed, or pages from working-sets. People here are confusing paging for swapping and vice versa. Swapping is swapping a good portion of memory to disk to free memory. Paging is swapping out a "page" of memory, in i386 world that's typically 4KB.

By the kernel scheduler, you mean thread scheduler? These are two different concepts. The VMM probably implements LRU or something of that sort or use a clock algorithm. Who knows? :) VMM typically has nothing to do with kernel scheduling. Should we entertain this idea, the prediction of when memory resources should be freed for other proesses is probably left to some kind of cache policy manager. That's a big research task! For now the working-set is our best bet for temporal and local cache prediction without requiring us to think too hard about when to actually free memory with a scheduler. We can easily reclaim memory from the LRU working-set pages from any process, or randomly if we want pseudo-LRU characteristics.

Re: Going Deep: Landy Wang - Windows Memory Manager

sonypappan — Fri, 25 Nov 2005 10:53:08 GMT

I have a basic doubt. Whether memory to memory transfer using system DMA controller is possible in windows?
If not why?
Please reply back me on http://sourcefrog.net/weblog/software/linux-kernel/swap.html

Re: Going Deep: Landy Wang - Windows Memory Manager

msemack — Fri, 01 Jul 2005 20:30:12 GMT

AndyC wrote:
The best place is the only partition on a dedicated drive on a dedicated controller. Anything else is a compromise of some sort.

I agree. And that is fully compatible with what I said. It's the most-used partition (the only one) of the least-used drive. :-)

AndyC wrote:
And even that's assuming all the drives are similar speeds. Which may not be the case.

This is true. However, in a system with a sufficient amount of RAM, the pagefile will be hit so infrequently that this all becomes academic.

Re: Going Deep: Landy Wang - Windows Memory Manager

AndyC — Fri, 01 Jul 2005 20:24:22 GMT

msemack wrote:

Assuming you have multiple drives, the best place for the pagefile is the middle of the most-used partition on the least-used drive.

The best place is the only partition on a dedicated drive on a dedicated controller. Anything else is a compromise of some sort.

And even that's assuming all the drives are similar speeds. Which may not be the case.

Re: Going Deep: Landy Wang - Windows Memory Manager

supersonic — Fri, 01 Jul 2005 20:17:27 GMT

well the pic isn't offline , its still there

if u can see it from the direct link , go to that site
http://www.winsupersite.com/showcase/longhorn_preview_2005.asp

scroll down a bit und u will see the last thunbnailpic on the right sight

Re: Going Deep: Landy Wang - Windows Memory Manager

msemack — Fri, 01 Jul 2005 20:02:02 GMT

androidi wrote:
I haven't tested whether there's a real perf benefit for this, but have always thought that it's optimal to have one drive/array for system and apps and second for pagefile and backups/infrequently accessed data.

Assuming you have multiple drives, the best place for the pagefile is the middle of the most-used partition on the least-used drive.

Re: Going Deep: Landy Wang - Windows Memory Manager

androidi — Fri, 01 Jul 2005 19:48:31 GMT

I haven't tested whether there's a real perf benefit for this, but have always thought that it's optimal to have one drive/array for system and apps and second for pagefile and backups/infrequently accessed data. I am talking about what is the optimal 2 hdd setup for both performance and some redundancy - special arrangement like this or some special RAID that gives both perf + redundancy on just 2 drives?

Re: Going Deep: Landy Wang - Windows Memory Manager

msemack — Fri, 01 Jul 2005 19:44:28 GMT

Beer28 wrote:
Why is swap space on windows used when you have enough physical ram to handle the processes? As opposed to it not being used until it's necessary on linux?

There is no such thing as "swap space" in Windows (NT anyway). Swapping and paging are not the same. Don't confuse them.

Windows "backs" pages of memory in the pagefile. I should note that it doesn't back every memory page in the pagefile. Things like EXE and DLL code are already backed in their on-disk files.

Notice that I used the word "backs". The pagefile is (loosely) mirroring the contents of RAM. This means that when an app does need to get paged to disk, it's already in the pagefile. The RAM can just be marked as free.

If this wasn't done, when an app needed to be paged out, Windows would have to write the modified pages to disk right then and there. This would increase the disk thrashing when a system gets maxed out.

Yes Windows is using the pagefile. Is it actually running programs from the pagefile? Probably not. Unfortunately, a lot of people don't understand this.

I feel that Task Manager is to blame for some of this. The way that it reports pagefile usage is misleading. If you want to get an idea for how much of your pagefile is really being used, run Perfmon.

FYI, You are incorrect when you say that Linux doesn't use the swap space with lots of physcial memory. It does something very similar to what I have described above.

Re: Going Deep: Landy Wang - Windows Memory Manager

msemack — Fri, 01 Jul 2005 19:30:36 GMT

Beer28 wrote:
Is there any major differences between the page file and swap?

There is a difference between paging and swapping. You can't use the two terms interchangeably.

Beer28 wrote:
Why not have a privilaged swap partition instead of a file?

Pagefile accesses are largely limited by seek times. This is precisely why Windows tries to put the pagefile in the middle of your partition.

Remember, pagefile access is happening in parallel with other file activity. The futher the disk heads have to move, the slower your system will run.

If you put your pagefile on a dedicated partition (or use a swap partition like Linux does), then the disk heads have to go out of their way on each pagefile read/write.

The optimal place for a pagefile is in the middle of the most heavily used parition of the hard drive.

Re: Going Deep: Landy Wang - Windows Memory Manager

Christian Liensberger — Fri, 01 Jul 2005 12:43:13 GMT

Orbit86 wrote:
beer, looks like you want to see Windows code...

We don't need to see any code. There is much scalability between talking about a pagefile that is used to extend memory and the real code.

As beer said he should have talked a little more about the algorithms and how they work.

If you attend a lecture on OS at the university they tell you more then was mentioned in the video. And they simply talk about concepts and not about the actual code.

Re: Going Deep: Landy Wang - Windows Memory Manager

AndyC — Fri, 01 Jul 2005 12:42:04 GMT

Beer28 wrote:
Are there any major differences between the way windows handles memory pages and the way unix or linux handles pages?

Yes, lots. I guess, since this is a Going Deep video, that it is time for the obligatory Windows Internals link. :)

Beer28 wrote:

Is there any major differences between the page file and swap?
Why not have a privilaged swap partition instead of a file?

NTFS stores everything in files, even the MFT. That way it can move sections of them around if there are corrupt sectors. It also means they can be dynamically resized as and when is necessary.

Again the Window Internals book goes into a lot of detail about this.

Re: Going Deep: Landy Wang - Windows Memory Manager

GRiNSER — Thu, 30 Jun 2005 19:48:24 GMT

supersonic wrote:

looks a lot like that

http://www.winsupersite.com/images/review/lh5060_logon.jpg

yes it looks exactly like this. [although the picture is offline now, but i saw it when it was there] - i was asking my question because i remembered this picture ;)

Re: Going Deep: Landy Wang - Windows Memory Manager

Charles — Thu, 30 Jun 2005 18:45:33 GMT

We'll probably try and go a tad deeper with Landy sometime in the future, addressing the Longhorn Memory Manager, specifically. I realize this wasn't as deep as the other Going Deeps, but the MM is pretty complex and requires more than 30 minutes...

C

Re: Going Deep: Landy Wang - Windows Memory Manager

Christian Liensberger — Thu, 30 Jun 2005 11:34:30 GMT

I'm a little bid sad that they did not go a little bit deeper. Most of the talk was already known from books or the university...

Re: Going Deep: Landy Wang - Windows Memory Manager

supersonic — Wed, 29 Jun 2005 20:49:07 GMT

Beer28 wrote:
Orbit86 wrote:could be Longhorn because It was a login screen that I didnt recognize...its linux

it's not linux, there's no halt, reboot, ect... bar at the bottom.
It looks like a lockout screen more than a login screen.

looks a lot like that

http://www.winsupersite.com/images/review/lh5060_logon.jpg

Re: Going Deep: Landy Wang - Windows Memory Manager

Charles — Wed, 29 Jun 2005 18:15:03 GMT

GRiNSER wrote:
was there longhorn running on the pc in the video?

That's a good question. What OS was it, I wonder... :)

Re: Going Deep: Landy Wang - Windows Memory Manager

GRiNSER — Wed, 29 Jun 2005 17:46:27 GMT

was there longhorn running on the pc in the video?

Re: Going Deep: Landy Wang - Windows Memory Manager

DuNuNuBatman — Wed, 29 Jun 2005 16:35:55 GMT

All this talk of food is making my hugry...

Re: Going Deep: Landy Wang - Windows Memory Manager

koorb — Wed, 29 Jun 2005 14:12:15 GMT

Manip wrote:

Three things wrong with that

1. What does it have to do with this topic?
2. Is Windows unstable?
3. If your cheap hardware is making Windows unstable how would managed code help that?

Like the who video was about memory management. And managed code has its own garbage collector.

Re: Going Deep: Landy Wang - Windows Memory Manager

LarryOsterman — Wed, 29 Jun 2005 13:51:51 GMT

koorb wrote:
Interesting and something tells me that Windows would be a lot more stable if everybody moved to managed code.

And where would the managed code get it's memory?

Also, you can still leak memory in 100% managed code.