Memory Compression in Windows 10 RTM

Download this episode

Download Video

Description

I had the privilege of spending some more time with the Windows 10 Kernel folks!! I love learning about the OS with the folks that actually work in it day in and day out. In this video Mehmet Iyigun spent some time discussing why the System process in Windows 10 is taking a bit more memory and why it's a good thing. A process taking more memory sounds like a bad thing - that is until I understood more about memory management, paging, and hard / soft page faults. Turns out that that the OS is doing some clever optimizations that allow your processes to trim some of the memory but not necessarily page it out to disk. Not only is the memory preserved in RAM, but it is also compressed - making hard page faults a more rare occurrence. The results should make for a snappier experience. The best part is that this feature is not an experimental feature: it has been available since the RTM bits of Windows 10 were released.

Enjoy!

 

Tags:

Kernel, Windows

Embed

Format

Available formats for this video:

Actual format may change based on video formats available and browser capability.

    The Discussion

    • User profile image
      dodavinkeln

      Love these videos. I remember all the videos from the Vista era... Please more of these! :) Would love to more from higher level as well, maybe about the GUI and the audio stack. Keep em comin!

    • User profile image
      Jan​Kratochvil

      This was awesome, thanks for making it happen Seth!

      I'd love to see more videos like this with similar content depth, keep them coming.

    • User profile image
      JockStrap

      Very interesting and enjoyable. Like the previous posters I too wold love to see more videos at this depth.

      Fantastic idea of using a separate store for each universal app so you can page the whole thing out as one block when it is tombstoned/suspended.

      One question - aside from the increased complexity, why is the memory compression not done on the modified list pages stage rather than the PF? It seems to me that all that is being achieved is you are using memory to make a compressed page file in memory, however this just adds to the memory pressure on the system which means more pages have to be moved to the PF.

       

    • User profile image
      Mehmet Iyigun

      JockStrap,

      We don't want to aggressively compress pages that are on the modified list because:
      1. Many of those pages are only temporarily on the modified list and will be soft-faulted back into the working set "soon". Having to compress and decompress these pages would be wasteful.
      2. If the system is not under memory pressure, the memory manager may keep pages on the modified list for a *long* time which increases the chances of #1 happening. These pages may never need to be written out because e.g. the process that owns them may be closed.
      3. The system has the necessary policy to decide whether modified pages should be written out (based on available memory and dirty page generation rate, etc), so if we're remotely close to experiencing memory pressure on the machine the modified pages will be sent to the pagefile at which point they will be compressed and memory pressure will be alleviated.

      The bottom-line is that, while compression/decompression is pretty efficient, we don't want to overdo it.

    • User profile image
      andriysavin

      That all is cool, however I observe system commit unbounded growth on several PCs with Win 10 RTM. I have 16 GB of RAM, and launched programs consume at most 10 gigs. And I observed that system commit growed up to about 30 Gigs over couple of days of system uptime. And closing all user processes doesn't solve the problem - system continues to consume 10-15 Gigs. If you limit the page file in size sooner or later system starts reporting low memory! 

      Look at these Task Manager screenshots: http://1drv.ms/1OycsB0

      How come all these processes consume 22 gigs of page file??

      In Process Explorer I can't see any process (including system) which consumes so much. And the most interesting part is that after logging out and logging in system commit drops to couple of gigs! So guys, you have some big trouble with memory management, so you should better investigate this ASAP before users start noticing something is wrong.

    • User profile image
      andriysavin

      And here is how my memory looks after closing most user processes: http://1drv.ms/1KrKcOO

      How come my system still consumes almost 14 gigs of page file memory with almost no apps running?? And almost 8 Gigs of physical memory is in use not including caching! What for??

      The System process, which is where this memory compression consumption is displayed in TM, consumes not that much, as you can see. And I don't observe this cache-like behavior you describe for low-memory conditions, when compressed pages are written to page file, freeing the physical memory.

      To be clear: I would have bothered with these numbers (assuming this all is due to memory compression or whatever), BUT I do observe system slowdown, low memory warnings and apps crashes while there is no obvious reason to observe this, since my apps don't consume so much, and I have lots of physical memory installed.

    • User profile image
      mehmeti
      andriysavin,
       
      I did not touch on the relationship between compression and commit usage in this talk because I thought it might be too complicated (I continue to be impressed by the technical depth of people on Channel9). So, let me make a few points:
       
      1. Memory compression does not charge any additional commit for the compressed store(s); it "borrows" commit from the process owning the uncompressed pages, so no additional commit is necessary. You can see this by going to the "Details" tab in Task Manager and adding the "Commit Size" column (or the "Private Bytes" column in Process Explorer).
       
      2. Furthermore, on platforms like Mobile (which are extremely commit-constrained), we actually *decrease* system-wide commit usage by decommitting the delta between a process' commit charge and its compressed store size when that process gets suspended. This is only done for "modern" applications that are resource-managed and it typically eliminates 60-75% of the commit charge for these processes. I won't get into why we don't do that on Desktop SKUs, but there are good reasons.
       
      3. There are roughly three types of commit: private, shared and "other". Private commit is charged for private allocations (VirtualAlloc, etc) from the process and generally accounts for the vast majority of commit associated with a process. Private commit usage is displayed in the details tab of Task Manager in the "Commit Size" column, so a leak of this category is easy to identify. Shared commit, on the other hand, is charged for shared allocations that are pagefile-backed and is much more difficult to associate with a process using publicly available tools (also because it's potentially shared between processes, so there may not be a single "owner"). The "other" category of commit is allocated for various kernel-mode purposes including driver physical memory allocations. A kernel debugger is currently the tool that will give you most detail about commit breakdown. For more information, Bing "debugger shared commit" and click on the first two links that show up (I can't paste the links here because then Channel9 thinks my post appears to be spam).
       
    • User profile image
      AndreyBazhan

      Thumb up for more videos like this.

      @mehmeti: Now it is clear what this pagefile #2 is. I have one question though:
      You said that modern apps have their own store, but I can oversee how System Store size grows when working set of some app is trimmed. Is there some nuances?

      http://andreybazhan.com/img/Store1.PNG

      http://andreybazhan.com/img/Store2.PNG

       

    • User profile image
      mehmeti

       That's not the system store size. You're looking at what we call the "virtual pagefile", represented by all the compressed stores (system store as well as per-app stores). The 31MB increase in size is the uncompressed data size for SearchUI.exe private pages (the rest of its private working set may be zero-filled pages which are discarded, etc). There's no easy way (beyond a kernel debugger) using public tools to see the store associated with SearchUI.exe.

    • User profile image
      AndreyBazhan

      @mehmeti: Now it's clear, thanks!!!

    • User profile image
      aL3891

      I read somewhere that memory compression was changed in TH2 (breaking chrome for a while) was that something in addition to this stuff? would be fun to hear more details about that :)

    • User profile image
      mehmeti

      Memory compression (just like paging and virtual memory) is completely transparent to applications, so it can't really break Chrome or any other application. 

      With every release of Windows, we continue tuning and improving our memory management policies and mechanisms. The TH2 update is no exception, but there are no fundamental changes to how memory compression works.

       

    • User profile image
      Yoshihro​Kawabata

      Thank you for nice episode.

      I want to see the memory compression work by Performance Monitor, etc.
      Can I know which performance monitor object related to the memory compression.

    • User profile image
      csrakowski

      Thanks for the video. Very nice and clear explanation!

      I am currently wondering how this feature will work with ASP.NET's App Suspend feature.

      Will the suspended application pools be compressed into the store, following your explanation, or is App Suspend it's own custom implementation that bypasses the memory manager's workflow?

    • User profile image
      Charles

      Perfect timing! Was just discussing this with a colleague and wondering what's actually going on when System process "eats" significant amounts of memory (as seen in taskmgr...).

      Great job and thanks for the information, Mehmet!

      Keep these coming, Seth! :)

      C

    • User profile image
      Arturo

      This is great stuff. Two questions:
      - Memory pressure is higher on Windows Mobile. Will memory compression be available in Win10 Mobile RTM?
      - Will there by any APIs to (1) get the compressed data (2) ask the OS to compress something because?

      Thanks.

    • User profile image
      aL3891

      @mehmeti:That is what they claimed anyway, (that something in windows insider builds related to memory broke some feature in chrome) but i dont know if that was actually the case or not.

      The reason i asked is because there was an article on insider announcements for TH2 about memory compression and i thought since they brought it up is was a new feature

       

    • User profile image
      thecYrus

      thanks for this great video!

      what's your opinion about memory deduplication? Wouldn't that be great on hyper-v servers?

    • User profile image
      mehmeti

      Memory compression is definitely enabled on Windows 10 Mobile.

      I don't expect that we'll expose APIs that interface with memory compression because we want to keep it as something that is entirely transparent to applications. That's also why we don't have performance counters for it either.

      Memory de-duplication is something we've been doing pretty aggressively on Windows client SKUs since Windows 8. On Server 2012, you can enable this functionality via powershell: https://msdn.microsoft.com/en-us/dn567639

      Hyper-V is a different story and we'll be sharing more about that in the coming months.

    • User profile image
      Ali Demiroz

      Thanks for the great video.

      May I ask which algorithm you are using for memory compression? Is there a single one or set of them that you chose from, after some kind of analysis on data to be compressed?

      Reducing the size of actual data to %30-%40 is great achievement I think.

      It would be even better if you can point some paper/material about it.

    • User profile image
      john werneken

      Win 10 Pro 64 Build 10547 memory


      have decided new 10547 memory management model is not ready for prime time. perfectly

      ordinary activity maxes at 96% slows things to a crawl especially built-on-fly stuff

      like file search, add favorite, context menu. Also rmapp.exe from MS SysInternals

      does not work, at all, with this build.

    • User profile image
      johnwerneken

      Hate video. I can read at least 100 times faster than I can listen. Also, my native language being English, aural is flexible but ambiguous and context requires extra effort to determine. Text is flexible unambiguous mostly and context is obvious. Instead of 30 seconds, 40 minutes. grr,

    • User profile image
      johnwerneken

      I think churn is what I am seeing, as there may be 30 independent automated processes running, any one of which is always doing something. Finally, I hate mobile and small form factor and small memory machines. 8.1 spring base load 14 % win 10 pro 64 until 10547 about 21 %. win 10 pro 64 10547 47%. GRRR.

    • User profile image
      johnwerneken

      hard to tell as rmapp.exe broken by 10547.

    • User profile image
      kjoyner

      Ya couldn't have cleaned the whiteboard first?   At first I thought it was stainless steel :-.

    • User profile image
      Fabulous_​Fadz

      This was a great video Seth and Mehmet, and seeing as I came to the comments late, pretty much all my follow up questions have been addressed. All I can say is that I now understand why the System was taking up so much memory yet my PC wasn't struggling. Before this I couldn't make head or tail of it.

      Like @dodavinkeln said, this also reminds me of the Vista era videos with the dev team. Keep these in-depth videos coming. :)

       

    • User profile image
      cppForMe

      Excellent video and explanations! Keep these coming and thanks for taking the time to make this.

    • User profile image
      Jason Fossen

      Great video! Looking forward to how this plays out on phones/tablets using Continuum in PC Mode, Xbox, HoloLens, Raspberry Pi's, in-memory data deduplication, etc. Maybe have the option to encrypt the data too for sensitive apps, like banking?

    • User profile image
      C9FiservFan

      Great video.  Thanks for taking the time to produce it.  :D

    • User profile image
      1990s technology

      took them long enough to implement 1990s technology, Quarterdeck Magnaram 97 feature

    • User profile image
      WinInsider

      Computers with 8GB or 16GB of RAM... Mehmet do you recommend disabling page file?

    • User profile image
      Deepak

      BTW, good taste in scotch.
      Try 14 years Balvanie, its even better :-)

    • User profile image
      slink9

      Plenty of memory? I have 16G and only 24% of it is used. In spite of that, I see that the "system and compressed memory" service is using 14.9M (not a concern) and it is continually creating .1M of disk activity. You say that the purpose of this is to decrease disk activity, yet this is continually hitting on my disk even though it has nothing to do.

      Will the activity cease over time? I just finished the Win10 upgrade this morning and my computer was asleep probably from noon until 6pm when I arrived home. It is now 8pm and task manager has shown .1 to .3M of disk activity continually over the past two hours although memory(<25%) and CPU (7-12%) are low.

    • User profile image
      Andrew

      THANKS, MICROSOFT! I have 99% memory usage and the system freezes for 10 minutes or so with no mouse movement or keyboard input + IO lamp ON. I'm using the Microsoft Wireless Mouse 5000 and Keyboard 3000 v2.0 via a USB stick. I had to "power reset" 8 times while attempting to de-install SQL Server. De-installing Visual Studio freezes too. THIS IS RIDICULOUS! YOU CALL IT A FEATURE? Poolmon identified smNp i.e. rdyboost.sys (10.0.10586.0) as the culprit. You didn't TEST this "feature" at all!!!!! Further comments of mine in Windows Feedback and Microsoft Community.

    • User profile image
      TairikuOkami

      How about fixing memory management instead?! I disable pagefile and use cleanmem, it works like a charm.

    • User profile image
      Waldemar

      Very good video.

    • User profile image
      JIt Hoong Lim

      Thank you for this video! this is so awesome. I am finally getting a clear understanding of this memory management in Windows. :)))

    • User profile image
      Zach

      This feature is bugged in the latest update. You cant disable it. Its an idiotic feature when you are CPU bound in the app in question and you have stupid overkill amounts and speed DDR4.

    Comments closed

    Comments have been closed since this content was published more than 30 days ago, but if you'd like to continue the conversation, please create a new thread in our Forums, or Contact Us and let us know.