Entries:
Comments:
Discussions:

Loading user information from Channel 9

Something went wrong getting user information from Channel 9

Latest Achievement:

Loading user information from MSDN

Something went wrong getting user information from MSDN

Visual Studio Achievements

Latest Achievement:

Loading Visual Studio Achievements

Something went wrong getting the Visual Studio Achievements

Memory Compression in Windows 10 RTM

38 minutes, 3 seconds

Download

Right click “Save as…”

I had the privilege of spending some more time with the Windows 10 Kernel folks!! I love learning about the OS with the folks that actually work in it day in and day out. In this video Mehmet Iyigun spent some time discussing why the System process in Windows 10 is taking a bit more memory and why it's a good thing. A process taking more memory sounds like a bad thing - that is until I understood more about memory management, paging, and hard / soft page faults. Turns out that that the OS is doing some clever optimizations that allow your processes to trim some of the memory but not necessarily page it out to disk. Not only is the memory preserved in RAM, but it is also compressed - making hard page faults a more rare occurrence. The results should make for a snappier experience. The best part is that this feature is not an experimental feature: it has been available since the RTM bits of Windows 10 were released.

Enjoy!

 

Tags:

Follow the discussion

  • Oops, something didn't work.

    Getting subscription
    Subscribe to this conversation
    Unsubscribing
    Subscribing
  • Johan Peterssondodavinkeln Developer in LA

    Love these videos. I remember all the videos from the Vista era... Please more of these! :) Would love to more from higher level as well, maybe about the GUI and the audio stack. Keep em comin!

  • Jan Kratochv√≠lJan​Kratochvil Jan Kratochvil

    This was awesome, thanks for making it happen Seth!

    I'd love to see more videos like this with similar content depth, keep them coming.

  • Very interesting and enjoyable. Like the previous posters I too wold love to see more videos at this depth.

    Fantastic idea of using a separate store for each universal app so you can page the whole thing out as one block when it is tombstoned/suspended.

    One question - aside from the increased complexity, why is the memory compression not done on the modified list pages stage rather than the PF? It seems to me that all that is being achieved is you are using memory to make a compressed page file in memory, however this just adds to the memory pressure on the system which means more pages have to be moved to the PF.

     

  • Mehmet IyigunMehmet Iyigun

    JockStrap,

    We don't want to aggressively compress pages that are on the modified list because:
    1. Many of those pages are only temporarily on the modified list and will be soft-faulted back into the working set "soon". Having to compress and decompress these pages would be wasteful.
    2. If the system is not under memory pressure, the memory manager may keep pages on the modified list for a *long* time which increases the chances of #1 happening. These pages may never need to be written out because e.g. the process that owns them may be closed.
    3. The system has the necessary policy to decide whether modified pages should be written out (based on available memory and dirty page generation rate, etc), so if we're remotely close to experiencing memory pressure on the machine the modified pages will be sent to the pagefile at which point they will be compressed and memory pressure will be alleviated.

    The bottom-line is that, while compression/decompression is pretty efficient, we don't want to overdo it.

  • That all is cool, however I observe system commit unbounded growth on several PCs with Win 10 RTM. I have 16 GB of RAM, and launched programs consume at most 10 gigs. And I observed that system commit growed up to about 30 Gigs over couple of days of system uptime. And closing all user processes doesn't solve the problem - system continues to consume 10-15 Gigs. If you limit the page file in size sooner or later system starts reporting low memory! 

    Look at these Task Manager screenshots: http://1drv.ms/1OycsB0

    How come all these processes consume 22 gigs of page file??

    In Process Explorer I can't see any process (including system) which consumes so much. And the most interesting part is that after logging out and logging in system commit drops to couple of gigs! So guys, you have some big trouble with memory management, so you should better investigate this ASAP before users start noticing something is wrong.

  • And here is how my memory looks after closing most user processes: http://1drv.ms/1KrKcOO

    How come my system still consumes almost 14 gigs of page file memory with almost no apps running?? And almost 8 Gigs of physical memory is in use not including caching! What for??

    The System process, which is where this memory compression consumption is displayed in TM, consumes not that much, as you can see. And I don't observe this cache-like behavior you describe for low-memory conditions, when compressed pages are written to page file, freeing the physical memory.

    To be clear: I would have bothered with these numbers (assuming this all is due to memory compression or whatever), BUT I do observe system slowdown, low memory warnings and apps crashes while there is no obvious reason to observe this, since my apps don't consume so much, and I have lots of physical memory installed.

  • andriysavin,
     
    I did not touch on the relationship between compression and commit usage in this talk because I thought it might be too complicated (I continue to be impressed by the technical depth of people on Channel9). So, let me make a few points:
     
    1. Memory compression does not charge any additional commit for the compressed store(s); it "borrows" commit from the process owning the uncompressed pages, so no additional commit is necessary. You can see this by going to the "Details" tab in Task Manager and adding the "Commit Size" column (or the "Private Bytes" column in Process Explorer).
     
    2. Furthermore, on platforms like Mobile (which are extremely commit-constrained), we actually *decrease* system-wide commit usage by decommitting the delta between a process' commit charge and its compressed store size when that process gets suspended. This is only done for "modern" applications that are resource-managed and it typically eliminates 60-75% of the commit charge for these processes. I won't get into why we don't do that on Desktop SKUs, but there are good reasons.
     
    3. There are roughly three types of commit: private, shared and "other". Private commit is charged for private allocations (VirtualAlloc, etc) from the process and generally accounts for the vast majority of commit associated with a process. Private commit usage is displayed in the details tab of Task Manager in the "Commit Size" column, so a leak of this category is easy to identify. Shared commit, on the other hand, is charged for shared allocations that are pagefile-backed and is much more difficult to associate with a process using publicly available tools (also because it's potentially shared between processes, so there may not be a single "owner"). The "other" category of commit is allocated for various kernel-mode purposes including driver physical memory allocations. A kernel debugger is currently the tool that will give you most detail about commit breakdown. For more information, Bing "debugger shared commit" and click on the first two links that show up (I can't paste the links here because then Channel9 thinks my post appears to be spam).
     
  • Thumb up for more videos like this.

    @mehmeti: Now it is clear what this pagefile #2 is. I have one question though:
    You said that modern apps have their own store, but I can oversee how System Store size grows when working set of some app is trimmed. Is there some nuances?

    http://andreybazhan.com/img/Store1.PNG

    http://andreybazhan.com/img/Store2.PNG

     

  •  That's not the system store size. You're looking at what we call the "virtual pagefile", represented by all the compressed stores (system store as well as per-app stores). The 31MB increase in size is the uncompressed data size for SearchUI.exe private pages (the rest of its private working set may be zero-filled pages which are discarded, etc). There's no easy way (beyond a kernel debugger) using public tools to see the store associated with SearchUI.exe.

  • @mehmeti: Now it's clear, thanks!!!

  • I read somewhere that memory compression was changed in TH2 (breaking chrome for a while) was that something in addition to this stuff? would be fun to hear more details about that :)

  • Memory compression (just like paging and virtual memory) is completely transparent to applications, so it can't really break Chrome or any other application. 

    With every release of Windows, we continue tuning and improving our memory management policies and mechanisms. The TH2 update is no exception, but there are no fundamental changes to how memory compression works.

     

  • Thank you for nice episode.

    I want to see the memory compression work by Performance Monitor, etc.
    Can I know which performance monitor object related to the memory compression.

  • Thanks for the video. Very nice and clear explanation!

    I am currently wondering how this feature will work with ASP.NET's App Suspend feature.

    Will the suspended application pools be compressed into the store, following your explanation, or is App Suspend it's own custom implementation that bypasses the memory manager's workflow?

  • CharlesCharles Welcome Change

    Perfect timing! Was just discussing this with a colleague and wondering what's actually going on when System process "eats" significant amounts of memory (as seen in taskmgr...).

    Great job and thanks for the information, Mehmet!

    Keep these coming, Seth! :)

    C

  • ArturoArturo

    This is great stuff. Two questions:
    - Memory pressure is higher on Windows Mobile. Will memory compression be available in Win10 Mobile RTM?
    - Will there by any APIs to (1) get the compressed data (2) ask the OS to compress something because?

    Thanks.

  • @mehmeti:That is what they claimed anyway, (that something in windows insider builds related to memory broke some feature in chrome) but i dont know if that was actually the case or not.

    The reason i asked is because there was an article on insider announcements for TH2 about memory compression and i thought since they brought it up is was a new feature

     

  • thanks for this great video!

    what's your opinion about memory deduplication? Wouldn't that be great on hyper-v servers?

  • Memory compression is definitely enabled on Windows 10 Mobile.

    I don't expect that we'll expose APIs that interface with memory compression because we want to keep it as something that is entirely transparent to applications. That's also why we don't have performance counters for it either.

    Memory de-duplication is something we've been doing pretty aggressively on Windows client SKUs since Windows 8. On Server 2012, you can enable this functionality via powershell: https://msdn.microsoft.com/en-us/dn567639

    Hyper-V is a different story and we'll be sharing more about that in the coming months.

  • Ali DemirozAli Demiroz

    Thanks for the great video.

    May I ask which algorithm you are using for memory compression? Is there a single one or set of them that you chose from, after some kind of analysis on data to be compressed?

    Reducing the size of actual data to %30-%40 is great achievement I think.

    It would be even better if you can point some paper/material about it.

  • john wernekenjohn werneken

    Win 10 Pro 64 Build 10547 memory


    have decided new 10547 memory management model is not ready for prime time. perfectly

    ordinary activity maxes at 96% slows things to a crawl especially built-on-fly stuff

    like file search, add favorite, context menu. Also rmapp.exe from MS SysInternals

    does not work, at all, with this build.

  • Hate video. I can read at least 100 times faster than I can listen. Also, my native language being English, aural is flexible but ambiguous and context requires extra effort to determine. Text is flexible unambiguous mostly and context is obvious. Instead of 30 seconds, 40 minutes. grr,

  • I think churn is what I am seeing, as there may be 30 independent automated processes running, any one of which is always doing something. Finally, I hate mobile and small form factor and small memory machines. 8.1 spring base load 14 % win 10 pro 64 until 10547 about 21 %. win 10 pro 64 10547 47%. GRRR.

  • hard to tell as rmapp.exe broken by 10547.

  • Ya couldn't have cleaned the whiteboard first?   At first I thought it was stainless steel :-.

  • Fabulous FadzFabulous_​Fadz One does not simply walk into Mordor

    This was a great video Seth and Mehmet, and seeing as I came to the comments late, pretty much all my follow up questions have been addressed. All I can say is that I now understand why the System was taking up so much memory yet my PC wasn't struggling. Before this I couldn't make head or tail of it.

    Like @dodavinkeln said, this also reminds me of the Vista era videos with the dev team. Keep these in-depth videos coming. :)

     

  • Excellent video and explanations! Keep these coming and thanks for taking the time to make this.

  • Jason FossenJason Fossen

    Great video! Looking forward to how this plays out on phones/tablets using Continuum in PC Mode, Xbox, HoloLens, Raspberry Pi's, in-memory data deduplication, etc. Maybe have the option to encrypt the data too for sensitive apps, like banking?

  • Great video.  Thanks for taking the time to produce it.  :D

  • 1990s technology1990s technology

    took them long enough to implement 1990s technology, Quarterdeck Magnaram 97 feature

  • WinInsiderWinInsider Mike, MCAD

    Computers with 8GB or 16GB of RAM... Mehmet do you recommend disabling page file?

  • DeepakDeepak

    BTW, good taste in scotch.
    Try 14 years Balvanie, its even better :-)

  • Plenty of memory? I have 16G and only 24% of it is used. In spite of that, I see that the "system and compressed memory" service is using 14.9M (not a concern) and it is continually creating .1M of disk activity. You say that the purpose of this is to decrease disk activity, yet this is continually hitting on my disk even though it has nothing to do.

    Will the activity cease over time? I just finished the Win10 upgrade this morning and my computer was asleep probably from noon until 6pm when I arrived home. It is now 8pm and task manager has shown .1 to .3M of disk activity continually over the past two hours although memory(<25%) and CPU (7-12%) are low.

  • AndrewAndrew

    THANKS, MICROSOFT! I have 99% memory usage and the system freezes for 10 minutes or so with no mouse movement or keyboard input + IO lamp ON. I'm using the Microsoft Wireless Mouse 5000 and Keyboard 3000 v2.0 via a USB stick. I had to "power reset" 8 times while attempting to de-install SQL Server. De-installing Visual Studio freezes too. THIS IS RIDICULOUS! YOU CALL IT A FEATURE? Poolmon identified smNp i.e. rdyboost.sys (10.0.10586.0) as the culprit. You didn't TEST this "feature" at all!!!!! Further comments of mine in Windows Feedback and Microsoft Community.

  • How about fixing memory management instead?! I disable pagefile and use cleanmem, it works like a charm.

  • WaldemarWaldemar

    Very good video.

  • JIt Hoong LimJIt Hoong Lim

    Thank you for this video! this is so awesome. I am finally getting a clear understanding of this memory management in Windows. :)))

  • ZachZach

    This feature is bugged in the latest update. You cant disable it. Its an idiotic feature when you are CPU bound in the app in question and you have stupid overkill amounts and speed DDR4.

Remove this comment

Remove this thread

Close

Comments closed

Comments have been closed since this content was published more than 30 days ago, but if you'd like to continue the conversation, please create a new thread in our Forums, or Contact Us and let us know.