Entries:
Comments:
Posts:

Loading User Information from Channel 9

Something went wrong getting user information from Channel 9

Latest Achievement:

Loading User Information from MSDN

Something went wrong getting user information from MSDN

Visual Studio Achievements

Latest Achievement:

Loading Visual Studio Achievements

Something went wrong getting the Visual Studio Achievements

Discussions

Bass Bass Knows the way the wind is flowing.
  • It is time... Move the filesystem off of disks

    Dexter said:
    Bass said:
    *snip*

    Bad analogy. A hashtable data structure moves data around because there's nothing better it can do. A database can do many things to get the best performance and it does just that.

    A modern DB (eg: Drizzle) is going to delegate as much responsibility to the kernel as possible. This makes the code simpler, and this easier to optimize. That's just how it is.

  • It is time... Move the filesystem off of disks

    CreamFilling512 said:

    Hints are like, I am going to read sequentially, or I am going to read randomly.  It's not like, here's a hint describing this complex internal data structure that you could write a 10000 page book about.  If the OS was capable of such a level of heuristic it would be super slow, its just optimized for general use.

    Listen, once it's in memory (the only thing a DB can actually do on it's own) it's not automagically optimized either. You are going to have to structure your data structures in such a way that they make optimial use of the hardware's cache as well. You can not avoid this. 

     

    A lot of those really complex data structures (eg: the judy array) are so complicated is because they are designed around being cached. They can not cache themselves, because x86 does not allow this. So they must structure their data to be cached by the system implicitly.

  • It is time... Move the filesystem off of disks

    Dexter said:
    Bass said:
    *snip*

    Seriously, do you really want/expect a database system to move gigabytes or terrabytes of data around just to keep the kernel happy?

    Why not? Isn't that exactly what happens when you invoke a database optimization?

     

    Sometimes you have to move gigabytes or terrabytes of data to have optimal performance. Look up the hash table data structure.

  • It is time... Move the filesystem off of disks

    Dexter said:
    Bass said:
    *snip*

    Any application (big enough, complex enough to worth the effort of doing it) can do better than the kernel at caching because an application will always know better than the kernel what data it needs. The kernel can at best obeserve the reads and the writes and some hints passed through the system calls and do some guesswork based on that. The kernel does not have a time machine to look into the future but the application might just have one.

     

    So I don't really agree. I think a program can provide enough hints to a kernel to let the kernel do all the real work. IE: Moving commonly read data to a certain part of a file. Again, similar to x86 optimization.

  • It is time... Move the filesystem off of disks

    CreamFilling512 said:
    Bass said:
    *snip*

    No way man, the OS has no knowledge of the internal structure of the database.  You can certainly get better performance by doing more work.  Database servers optimize their layout on the physical disk.

    You can optimize the internal structure of the database such that the OS will optimize reads to the fullest.

     

    It's really no different then optimizing instructions, you have no control over branch prediction and cache usage on an x86 processor. But you can still optimize code for branch prediction and cache, by modifying the structure of your program.

  • It is time... Move the filesystem off of disks

    Dexter said:
    Bass said:
    *snip*

    Indeed, transactional databases require non cached writes (and it's not only about filesystem caching but also about hardware caching).

     

    As for read caching: of course they cache reads, it would be insane not to do it. The filesystem has no magic orb to tell it what exactly to cache, read ahead, discard from cache etc.

    It doesn't but an OS can accomplish a lot of the intelligent read caching by a paging algorithm, which keeps track of the most commonly read parts of a file. This is easy to implement with the information the kernel gets from mmap and subsequent use of the mmaped space. Most of the performence heavy lifting can be done by the kernel, which knows more about the characters of the persistent store (eg: location of the R/W head, etc).

     

    It would be downright stupid for a DB to take all of this in it's own hands, but I am sure there is some legacy DBs out there who still do all of this, because they were written during a time where MS-DOS was considered advanced. Smiley

  • It is time... Move the filesystem off of disks

    CreamFilling512 said:

    From what I understand from Windows internals stuff.  Any time you open a file regardless of how you do it, its implemented internally using memory-mapped files. File gets mapped to some pages in virtual memory, then it gets brought in by on-demand paging and some heuristics to do basic read-ahead, I imagine the paging file and on-demand paging of EXE/DLLs use the same mechanism.  But any kind of I/O APIs, like C's fread() or whatever, never issues I/O requests, you'll just get I/O requests if it touches some memory-mapped file and gets a page fault.

    Yeah Windows has a similar call to mmap, and I assume also does paging. Smiley

  • Zune for Mac, YEESSS

    dahat said:
    magicalclick said:
    *snip*

    Agreed... but at what cost?

     

    Last I heard... Macs (desktops and laptops) comprise only about 5-6% of the PC market... let's assume for the moment that that same % when it comes to the PC of iPhone/iPod users.

     

    While it's true that that still represents millions and millions of prospective users... who is more likely to switch to a Zune/Windows Mobile based device over an iPod/iPhone device? A PC user... or a Mac user?

     

    Your average user on either side likely has some degree of investment in one side or the other through other purchased software (productivity & games)... as well as a familiarity and comfort that is no minor thing.

     

    But lets say you are the guy making this decision... you've decided you are going to offer a similar/identical experience for the Zune on the Mac platform... and soon after your lieutenants come to you with a simple question “how are you going to pay for it?” Are you going to cut features from the larger experience by moving existing developers to the Mac stuff? Are you going to raise the overall development costs (and possible retail costs) by hiring even more people? Or are you going to delay shipment of your product even longer to get everything you want done... all the while risking falling behind of the competition? Keeping in mind the whole time that this is all being done in the hopes of being able to take some part out of a 5-6% bit of the market (and pay for the development costs)... where the competition is already heavily entrenched.

     

    So which will it be? Cut features, increase costs, or slip the schedule?

     

    Instead I'd argue that all of that time and those resources would be better spent making the Zune on Windows experience even better than the iTunes on Windows, or even iTunes on Mac experience so as to secure your base even better than the competition has over their own... then and only then... can/should you venture off and try to take some of their territory from them.

    You don't have to port Zune software for Zune to work with Mac. You just have to implement the MTP specification. You know, the same specification you guys actually wrote?

     

    I can't even make this up. Microsoft writes a specification that it hopes all DAPs follow. Almost all DAPs follow (except for Apple, but they implemented USB Mass Storage so w/e).

     

    Then Microsoft releases it's own DAP which does not implement MTP, and in fact, is explictly designed through all sorts of DRM-like shenanigans to only work on Windows with the Zune software. LOLWUT?

     

    The lack of Mac/Linux support an explictly designed feature you guys spent time and money designing. That's right, your company "design features / increased costs" for the purpose of explicitly denying Mac users from using the Zune.

     

    So your excuse is lame and bogus. Sorry bro.

  • It is time... Move the filesystem off of disks

    CreamFilling512 said:
    Bass said:
    *snip*

    Well I'm talking about the commercial database servers where scaling is necessary.  Like Microsoft SQL Server running Hotmail or something.  And if you've ever run MSSQL you know that it will consume all memory on the machine with the out-of-box configuration, because its doing its own disk caching.

    Perhaps, but commercial databases tend to guarantee some kind of data integrity, which is impossible unless they persist the contents of a transaction. I don't think they would use write caching by default, as it is fundamentally dangerous to this objective.

     

    I don't think a well designed DB would do extensive read caching either. It's something that is more readily done by a kernel. As everyone has been saying, that's read caching is what most FS do for you (including NTFS) for free.

     

    A DB can provide detailed information about their file I/O requirements through an mmap call anyway.

  • It is time... Move the filesystem off of disks

    CreamFilling512 said:
    Bass said:
    *snip*

    That's bizarre, I mean usually database severs want to do their own caching, and control disk flushing since they know better than the OS.

    I don't think most actually do, or if they do, they don't do a particularly good job of it. Really with DBs you expect that if you do an INSERT it will actually happen. So a lot of databases (at least Postgres and SQLite) call fsync after the completion of a simple write operation. Which on barriers enabled FS, tends to block until the data is actually written out to disk. Which is of course, slow.

     

    Without barriers the kernel decides when it feels it is appropriate to actually write the data to disc.This means a lot of file operations (both read and write) are all happening entirely in RAM, and only when the disk is available and it doesn't hamper performence will the kernel persist the contents to disk. This can be as long as 60 seconds after the fsync request was made (or longer?).


    You can fine tweak the performance vs data security, but the more data security you want, the less performance you are going to get (and vise-versa). Just a fact of life I guess. Smiley