Channel 9 Forums - Tech Off - SQL Server 2008 CTP, FILESTREAM & NTFS limitations http://mschnlnine.vo.llnwd.net/d1/Dev/App_Themes/C9/images/feedimage.png Channel 9 Forums - Tech Off - SQL Server 2008 CTP, FILESTREAM & NTFS limitations http://channel9.msdn.com/Forums Channel 9 keeps you up to date with the latest news and behind the scenes info from Microsoft that developers love to keep up with. From LINQ to SilverLight – Watch videos and hear about all the cool technologies coming and the people behind them. http://channel9.msdn.com/Forums en Sun, 19 May 2013 00:54:53 GMT Sun, 19 May 2013 00:54:53 GMT Rev9 5 -5 -1 Tech Off - SQL Server 2008 CTP, FILESTREAM & NTFS limitations I'm just wondering what's up with this.

I just set up a DB in 2008 CTP to store a massive amount of images. I'm using the FILESTREAM feature to store the binary data in the file system. I was wondering why my migration process came to a slow crawl after about 900,000 images and i was baffled as all the system and SQL server metrics seems to be just fine. after doing a little research i noticed people complaining somewhere about NTFS not playing that well with directories of that size and it seems to be the thing that is slowing my inserts down.

I also see that you cannot span the files into multiple folders since SQL server manages these files.

Why would SQL limit you in this way if NTFS cannot support such huge folders well?

If i'm missing something or there is a workaround for this somewhere i'd love to know cause i'm just shocked that this would be the case in software designed to support gigantic table.

Any thoughts, Ideas?

]]> http://channel9.msdn.com/Forums/TechOff/407930-SQL-Server-2008-CTP-FILESTREAM--NTFS-limitations/407930#407930 Thu, 05 Jun 2008 21:09:10 GMT http://channel9.msdn.com/Forums/TechOff/407930-SQL-Server-2008-CTP-FILESTREAM--NTFS-limitations/407930#407930 Jimi Schacht 5 http://channel9.msdn.com/Niners/Jimi2Cool/Discussions/RSS Tech Off - SQL Server 2008 CTP, FILESTREAM & NTFS limitations Well I have known for a long time that folders that have a large number of files can lead to perf issues on all file and folder operations.

I first saw this on Sun / Solaris around 1997
An ISP I worked for had users email on a disk array and saw massive slowdowns as the number of users went up.

also there were / perhaps still are IIS plugins for servers that want to host lots of free web sites.

the general answer is to create a tree in place of a flat directory.

example: 

Mailboxes/
a/
b/
c/
...
z/

then in each of the folders a..z create folders a..z and in each of them repeate.

so for example if my pop3 user name was "Denny"
the file would be found in the folder/path
Mailboxes/d/e/n/denny/*

with the result that each folder has a limited number of files and no one folder has a lot of files.
assuming of course that names are "spread out"
sometimes a hash is used to create folder names and map real names to the hash.

I would take the 900000 plus files and make a tree.

I suspect you could see the same access problem doing a dir-search or a set of file open functions in a program.

]]> http://channel9.msdn.com/Forums/TechOff/407930-SQL-Server-2008-CTP-FILESTREAM--NTFS-limitations/9d5a30f69b3148dd880e9dfa00916953#9d5a30f69b3148dd880e9dfa00916953 Thu, 05 Jun 2008 22:24:09 GMT http://channel9.msdn.com/Forums/TechOff/407930-SQL-Server-2008-CTP-FILESTREAM--NTFS-limitations/9d5a30f69b3148dd880e9dfa00916953#9d5a30f69b3148dd880e9dfa00916953 figuerres 5 http://channel9.msdn.com/Niners/figuerres/Discussions/RSS Tech Off - SQL Server 2008 CTP, FILESTREAM & NTFS limitations It's not NTFS, per se, it's the mechanism used to generate legacy 8.3 filenames that slows things down. If you know you're going to have folders containing more than about 300,000 files (and you aren't relying on 8.3 filenames) then it's recommended to turn off 8.3 filename generation(see: http://technet.microsoft.com/en-us/library/bb457112.aspx - scroll down to Optimizing NTFS Performance).

You can turn off short filename generation by using the command: fsutil behavior set disable8dot3

]]> http://channel9.msdn.com/Forums/TechOff/407930-SQL-Server-2008-CTP-FILESTREAM--NTFS-limitations/87affb1deac54e9ab7da9dfa009169c9#87affb1deac54e9ab7da9dfa009169c9 Fri, 06 Jun 2008 09:24:25 GMT http://channel9.msdn.com/Forums/TechOff/407930-SQL-Server-2008-CTP-FILESTREAM--NTFS-limitations/87affb1deac54e9ab7da9dfa009169c9#87affb1deac54e9ab7da9dfa009169c9 AndyC 5 http://channel9.msdn.com/Niners/AndyC/Discussions/RSS Tech Off - SQL Server 2008 CTP, FILESTREAM & NTFS limitations
AndyC said:
It's not NTFS, per se, it's the mechanism used to generate legacy 8.3 filenames that slows things down. If you know you're going to have folders containing more than about 300,000 files (and you aren't relying on 8.3 filenames) then it's recommended to turn off 8.3 filename generation(see: http://technet.microsoft.com/en-us/library/bb457112.aspx - scroll down to Optimizing NTFS Performance).

You can turn off short filename generation by using the command: fsutil behavior set disable8dot3
FANTASTIC! thanks so much Andy! that was exactly what the issue was. i was able to execute that command and my apps performance returned to exactly where it had been when it started (about 100X faster than where it had dropped to by the way).

]]> http://channel9.msdn.com/Forums/TechOff/407930-SQL-Server-2008-CTP-FILESTREAM--NTFS-limitations/5aa35ab99331423da6a49dfa00916a1c#5aa35ab99331423da6a49dfa00916a1c Fri, 06 Jun 2008 15:53:10 GMT http://channel9.msdn.com/Forums/TechOff/407930-SQL-Server-2008-CTP-FILESTREAM--NTFS-limitations/5aa35ab99331423da6a49dfa00916a1c#5aa35ab99331423da6a49dfa00916a1c Jimi Schacht 5 http://channel9.msdn.com/Niners/Jimi2Cool/Discussions/RSS Tech Off - SQL Server 2008 CTP, FILESTREAM & NTFS limitations
AndyC said:
It's not NTFS, per se, it's the mechanism used to generate legacy 8.3 filenames that slows things down. If you know you're going to have folders containing more than about 300,000 files (and you aren't relying on 8.3 filenames) then it's recommended to turn off 8.3 filename generation(see: http://technet.microsoft.com/en-us/library/bb457112.aspx - scroll down to Optimizing NTFS Performance).

You can turn off short filename generation by using the command: fsutil behavior set disable8dot3
You just saved me a lot of headaches! Thank you.

]]> http://channel9.msdn.com/Forums/TechOff/407930-SQL-Server-2008-CTP-FILESTREAM--NTFS-limitations/5d3ea329e24a449fb93f9dfa00916a94#5d3ea329e24a449fb93f9dfa00916a94 Sun, 08 Jun 2008 05:52:14 GMT http://channel9.msdn.com/Forums/TechOff/407930-SQL-Server-2008-CTP-FILESTREAM--NTFS-limitations/5d3ea329e24a449fb93f9dfa00916a94#5d3ea329e24a449fb93f9dfa00916a94 Turrican 5 http://channel9.msdn.com/Niners/turrican/Discussions/RSS