Tech Off Thread

43 posts

Microsoft's XP RAM Disk Driver

Back to Forum: Tech Off
  • User profile image
    VIad

    AndyC, I am not savvy in Windows internals, in how it caches and manages file O/I. I came across this old article mentioning an anecdotal evidence of how people were able to significantly increase their Photoshop productivity with ramdisk http://www.pcworld.idg.com.au/article/73385/boost_performance_win_xp_pro_ramdisk/

    Can you please elaborate on your point why "it won't ever be faster"? Please explain how is HDD I/O not holding up applications under no circumstances in WinXP for instance? I understand there is RAM caching of all file I/O in any modern OS, but is there no such use case when ramdisk really helps? I'm just confused by categorical "it won't ever be faster" and the article I referred to above.

    Anyone who understands subject matter please share your knowledge.

  • User profile image
    AndyC

    @VIad: If you have a RAMdisk, some of the RAM is not usable by Windows, ergo swapping happens more often. Except that in this case you're needlessly copying data from one bit of RAM (the RAMdisk) into another bit of RAM (the memory left for Windows). In effect you're storing duplicate copies of lots of things in memory and occasionally copying them around for no good reason. This hurts performance in comparison with simply using the RAM as RAM.

    If you put more memory in a PC, then allocate a large chunk of that to a RAMdisk, as per the article you linked to, you will quite possibly see some performance increase. It's just that it will be less of a performance increase than simply putting the same amount of memory in and letting Windows allocate it as needed.

  • User profile image
    evildictait​or

    , AndyC wrote

    @VIad: If you have a RAMdisk, some of the RAM is not usable by Windows, ergo swapping happens more often. Except that in this case you're needlessly copying data from one bit of RAM (the RAMdisk) into another bit of RAM (the memory left for Windows). In effect you're storing duplicate copies of lots of things in memory and occasionally copying them around for no good reason. This hurts performance in comparison with simply using the RAM as RAM.

    If you put more memory in a PC, then allocate a large chunk of that to a RAMdisk, as per the article you linked to, you will quite possibly see some performance increase. It's just that it will be less of a performance increase than simply putting the same amount of memory in and letting Windows allocate it as needed.

    Whilst that may be true for some scenarios (like if you do something stupid like allocate space for a pagefile in the RAM disk), for others it will have a beneficial effect. Storing something on a RAM disk means that it never touches the real disk, so a RAM disk might be ideal for storing lots of intermediate files which for some reason must be written out to somewhere - for example if you're taking a photo and passing it through lots of small programs each of which does a transform to the file and then writes it back to disk, then a RAM disk would speed things up just because you avoid all of the file flushes going out to disk.

    Similarly, playing a hi-def movie from a RAM disk would effectively guarantee no skipping of the movie compared with running from a hard-disk or CD-ROM because the content doesn't need to be loaded or streamed into memory - it's already there.

    I think the important point here, though, is that it all depends on how you use the RAM disk. If you're careful you might get a speed up compared with just letting Windows do it's thing (because you know what you're going to want to do more than Windows can guess what you're going to want to do), but in practise you'd probably get a bigger bang for your buck by just giving all of the memory to Windows and reducing the strain on the pagefile.

  • User profile image
    AndyC

    , evildictaitor wrote

    *snip*

    Whilst that may be true for some scenarios (like if you do something stupid like allocate space for a pagefile in the RAM disk), for others it will have a beneficial effect. Storing something on a RAM disk means that it never touches the real disk, so a RAM disk might be ideal for storing lots of intermediate files which for some reason must be written out to somewhere - for example if you're taking a photo and passing it through lots of small programs each of which does a transform to the file and then writes it back to disk, then a RAM disk would speed things up just because you avoid all of the file flushes going out to disk.

    If you've got lots of spare RAM, chances are the file will exist in the disk cache and won't actually be committed to the physical disk anyway. Windows already does a far better job at caching than you'll get from a RAMdisk.

  • User profile image
    evildictait​or

    , AndyC wrote

    *snip*

    If you've got lots of spare RAM, chances are the file will exist in the disk cache and won't actually be committed to the physical disk anyway. Windows already does a far better job at caching than you'll get from a RAMdisk.

    When you flush a file to a RAM disk, it is definitely in memory and possibly on disk. When you flush a file to a real disk, it lives definitely on disk and possibly in memory.

    If you think you're smarter than Windows (and that's not unlikely because Windows doesn't have a crystal ball to predict what you're going to want to do next), it's possible you might be able to use this to your advantage, because you have temporary files that live only possibly on disk, instead of definitely on disk, and thus avoid the IO-overhead of doing the flushes to disk.

    I don't disagree with you that for most people the default of letting Windows get on with its job is probably your best course of action, I just disagree with your suggestion that RAM disks always slow down what you're trying to achieve.

  • User profile image
    AndyC

    , evildictaitor wrote

    *snip*

    I don't disagree with you that for most people the default of letting Windows get on with its job is probably your best course of action, I just disagree with your suggestion that RAM disks always slow down what you're trying to achieve.

    At best, it'll be the same - letting Windows handle it will cache it or it'll be in the RAMdisk. At worse it'll be a lot slower, aside from the needless copying there is also the possibilty that the increased RAM pressure will mean that Windows will actually page out parts of your file and will therefore be swapping to and from physical storage to access it despite the fact you have a copy sat around in memory doing nothing, along with a whole bunch of additional free memory acting as RAMdisk free space.

    If you push really hard you might find a extremely obscure edge case in which a RAMdisk beats the cache, but it's exceptionally hard to do. It exists purely for compatibility with diskless systems and it no longer makes sense to use it otherwise.

  • User profile image
    evildictait​or

    , AndyC wrote

    At best, it'll be the same

    Nope. Here's a program that works markedly better on a RAM disk:

    FILE* fp = fopen("Z:\\foo.txt");
    BYTE buf[1024] = { 0 }; 

    for(int i = 0; i < 100*1000; i++)
    {
      fwrite(fp, &buf, sizeof(buf)); 
      fflush(fp); // this forces disk-io if you're on disk, and doesn't if you're on a ram-disk - it can't be buffered in the filesystem cache because fflush is an IO-commit event.
    }

    fclose(fp);

    Another example, perhaps more useful example is

    robocopy Z:\files Q:\files   will run much faster if Z and Q are RAM disks compared to real disk, because no IO is involved - only memory accesses.

    grep Z:\mysourcetree\ strcpy will also find vulnerable functions in a large source base much faster too, although I'd question who would be brave enough to have their source tree on a RAMDisk instead of something that will survive a power-cycle.

    Don't get me wrong - RAMDisks were deprecated for a reason; but they were invented for a reason too. For all sorts of operations they will be faster than the file-system cache - it's just that for an average user using a RAM-disk is probably worse than not using it.

    The inexcusable myth, however, is putting the pagefile on a RAM disk will always have a strictly negative performance impact on your system.

  • User profile image
    cheong

    Yup. I think most "diff" kind of programs works better if you could just put the file on ramdisk.

    Most applications that expects large files will do memory mapped I/O even if your system has plenty of memory left unused. Moving the whole file to different region on RAM gives better performance.

    I remember copying one of my VB6 project (the project is large, the built EXE file is about 12-15 MB in size, and build time is painful) to memory before compile will dramatically shorten the build time too.

    Recent Achievement unlocked: Code Avenger Tier 4/6: You see dead program. A lot!
    Last modified
  • User profile image
    AndyC

    , evildictaitor wrote

    *snip*

    Nope. Here's a program that works markedly better on a RAM disk:

    FILE* fp = fopen("Z:\\foo.txt");
    BYTE buf[1024] = { 0 }; 

    for(int i = 0; i < 100*1000; i++)
    {
      fwrite(fp, &buf, sizeof(buf)); 
      fflush(fp); // this forces disk-io if you're on disk, and doesn't if you're on a ram-disk - it can't be buffered in the filesystem cache because fflush is an IO-commit event.
    }

    fclose(fp);

    That would be one of those pathological cases of unrealistic behaviour that probably favours the RAM disk. And I say probably because you could theoretically still hit the scenario in which memory is tight enough that writing to foo.txt causes page faults in the RAM disk scenario (and thus read/writes to disk) but doesn't in the standard disk scenario which only ever needs to write updates to disk. You can find benchmark type tests that prove pretty much anything, they're rarely worthy of consideration.

    Another example, perhaps more useful example is

    robocopy Z:\files Q:\files   will run much faster if Z and Q are RAM disks compared to real disk, because no IO is involved - only memory accesses.

    If the best example possible is copying files, I believe my point has been proven as it again is a very edge case example - you don't normally copy files for no purpose (backing up might constitute the purpose if they weren't RAM drives).

    I'll discount the grep example too because, as you say, it'd be borderline stupid to keep something like source code on a RAM disk. And before someone suggests it, if you've copied it there from a fixed disk first you've already paid the price that getting it into the filesystem cache would require.

  • User profile image
    cheong

    , AndyC wrote

    I'll discount the grep example too because, as you say, it'd be borderline stupid to keep something like source code on a RAM disk. And before someone suggests it, if you've copied it there from a fixed disk first you've already paid the price that getting it into the filesystem cache would require.

    Actually, I'd not be sure.

    Application compilation is often done in multi-pass way. If the project folder is big enough, it would be very likely to go beyond the filesystem cache size.(That's why I see compile time improvement by copying the project folder to a ramdisk first)

    Also, filesystem cache manager handles read operation with "sequential read" hint and "random access" hint differently. (See Raymond Chen's blog two weeks ago) Random access opeations do not enjoy prefetch in the current implementation.

    Recent Achievement unlocked: Code Avenger Tier 4/6: You see dead program. A lot!
    Last modified
  • User profile image
    evildictait​or

    , AndyC wrote

    That would be one of those pathological cases of unrealistic behaviour

    Its not as unrealistic as you think. Databases for example do pretty much exactly this. If you don't need the database to survive a reboot (which is often the case with search indexers for example) then this would work better with the database on a ram-disk to avoid the random-access writes to disk after every database access.

    If the best example possible is copying files, I believe my point has been proven as it again is a very edge case example - you don't normally copy files for no purpose (backing up might constitute the purpose if they weren't RAM drives).

    If you don't like that example, take the example of storing Temporary Internet Files or the system TEMP directory on a ramdisk.

    Or if you don't like that, how about disassembling a movie into lots of bitmaps to push to a program that modifies the bitmap, only to recompile the bitmaps back into a movie at the end?

    Or Video streaming when the disk is in-use/old/slow/unreliable

    Or a place to download torrents to and from avoid lots of seeking during the random-access read/writes

    Or as a destination to unzip temporary files during an installation

    Or to store your OBJ files during compilation

    Or for disk-less computers where all data is stored on network drives

  • User profile image
    hachi

    sorry that i'm a total noob in this matter, but I too was googling about this because I have a very old system and wanted to see what I can do to gain some performance. I have read and realize the points made here, but what if i have this scenario? will i gain any performance boost?

    I am running Windows XP 32bit, but I have always had 8GB of physical RAM installed on the MB, however, my XP only recognizes 3.25 GB at max, in other words, I have 4.75 of unused and unmanaged memory by the OS (assuming this knowledge of mine is correct?) I read up in another article that there are RAM DISK programs, that will enable you to allocate the unused portion of this physical RAM into a virtual drive under XP.

    now the question is. If i then disable all pagefile on my regular HDD and allocate the paging file entirely onto the RAMDISK (that uses the unused physical RAM), will i then gain a performance increase?

     

  • User profile image
    evildictait​or

    I am running Windows XP 32bit, but I have always had 8GB of physical RAM installed on the MB, however, my XP only recognizes 3.25 GB at max, in other words, I have 4.75 of unused and unmanaged memory by the OS (assuming this knowledge of mine is correct?) I read up in another article that there are RAM DISK programs, that will enable you to allocate the unused portion of this physical RAM into a virtual drive under XP.

    Sadly it doesn't work like that - the limit of 3.25 GB is the total amount of memory that Windows XP 32-bit can see, regardless of whether that memory is being used by Windows or by a RAM-disk driver. If you give 1GB to a ramdisk driver, then Windows will only have 2.25GB available.

    If you want to make use of more of your physical RAM, you need to upgrade to a 64-bit OS

    (Windows XP-64 will do, but newer versions of Windows are probably better)

    , hachi wrote

    now the question is. If i then disable all pagefile on my regular HDD and allocate the paging file entirely onto the RAMDISK (that uses the unused physical RAM), will i then gain a performance increase? 

    .

    No. Never move the pagefile to a RAM disk. Doing so will always damage performance.

  • User profile image
    hachi

    the thing is, here's what i found, look at under the 'Extended Memory Access' feature, it states clearly it can break beyond the 3.25 Limit by 32bit XP... that is why i was questioning this. Because it seems logical if it indeed can use this unmanaged RAM so it won't actually eat into the original 3.25 used by the OS itself, i already understood that traditionally, RAM DISKS don't function that way, but this software made me 2nd guess that

  • User profile image
    hachi
  • User profile image
    evildictait​or

    , hachi wrote

    the thing is, here's what i found, look at under the 'Extended Memory Access' feature, it states clearly it can break beyond the 3.25 Limit by 32bit XP... that is why i was questioning this. Because it seems logical if it indeed can use this unmanaged RAM so it won't actually eat into the original 3.25 used by the OS itself, i already understood that traditionally, RAM DISKS don't function that way, but this software made me 2nd guess that

    The problem isn't the OS having a hard-coded limit that a driver can just ignore. The problem is that the CPU can't address that much memory - so if that company says they can break it they're either:

    a) Lying.

    b) Using device-specific or x64 features to access beyond that limit, but this slows the machine down more than just running your OS permanently in x64-mode - i.e. a x64 OS would be faster.

    If you don't mind me saying, it seems like you're approaching this from the wrong direction. Why are you using Windows-XP 32-bit? Any x64 OS doesn't have these limits right from the start, and x64 Windows will seamlessly run your 32-bit apps from previous versions of Windows, so your upgrade path should be pretty easy.

    Even if you're one of the people who really hates upgrading beyond XP, you could just use XP-64 with all of its blue-windowed goodness and and without the 3.25GB physical RAM limitation.

  • User profile image
    hachi

    i just wanted mimimal change to the system, reinstalling everything takes forever... 

  • User profile image
    evildictait​or

    , hachi wrote

    i just wanted mimimal change to the system, reinstalling everything takes forever... 

    Not really. You should be able to upgrade to Windows-XP 64 without reinstalling many of your programs, so long as they don't do too many registry shenanigans.

    It's like saying "I want my car to have more space in the trunk, so I've tied this barrel to the top of my 20-year old car for additional storage space because buying a new car takes forever". I can see why you're doing it, but it's the wrong solution for the problem at hand. The answer is to bite the bullet and just get a new one.

Comments closed

Comments have been closed since this content was published more than 30 days ago, but if you'd like to continue the conversation, please create a new thread in our Forums, or Contact Us and let us know.