Coffeehouse Thread

61 posts

Forum Read Only

This forum has been made read only by the site admins. No new threads or comments can be added.

Does Parallel.ForEach only return when ALL iterations have completed?

Back to Forum: Coffeehouse
  • User profile image
    BitFlipper

    @davewill: I've added your code to my test code and will let you know the next time I get the "The directory is not empty" exception. I suspect it will show zero for both as I have checked the folder in Explorer and it was always empty, and running the delete method a second time always succeeds (my actual code has a retry loop). It is some sort of short-lived race condition.

    As I mentioned previously, the issue only seems to reproduces when the files are not cached, so it is a bit difficult to reproduce it under the debugger since my test needs to copy or create the tree structure each time, meaning it will be cached at that point.

     

  • User profile image
    figuerres

    also it might be interesting to use VS debugging to step into the Delete() command and see how they wrote it.  the source should be viewable if you turn on the option in VS to debug the framework source code....  

  • User profile image
    BitFlipper

    , davewill wrote

    @BitFlipper: In the original code if the Directory.Delete is wrapped in a try/catch like ...

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    try
    {
        Directory.Delete(folder);
    }
    catch (Exception)
    {
        var dc = Directory.GetDirectories(folder).Length;
        var fc = Directory.GetFiles(folder).Length;
        Console.WriteLine("Folder=" + folder);
        Console.WriteLine("Dir count=" + dc);
        Console.WriteLine("File count=" + fc);
        throw;
    }

     

    what does it show? Zero for dir and file count or something greater? 

    OK so I was able to repro the "The directory is not empty" exception. Your code above showed dc = 1 and fc = 0, but when I used Explorer to look at the folder, it was empty. A subsequent Directory.Delete succeeded in deleting it. That is why a Retry loop is able to work around this issue.

    @figuerres: I might do that. Should be interesting to see how this was implemented.

  • User profile image
    BitFlipper

    , figuerres wrote

    also it might be interesting to use VS debugging to step into the Delete() command and see how they wrote it.  the source should be viewable if you turn on the option in VS to debug the framework source code....  

    Using ILSpy the very simplified pseudocode looks something like this:

    void DeleteHelper(folder)
    {
        Win32::FindFirstFile
        {
            while (true)
            {
                if (findData == file)
                {
                    Win32::DeleteFile(foundFile)
                }
                else if (findData == folder)
                {
                    DeleteHelper(foundFolder)
                }
                else if (findData == none)
                {           
                    break
                }

                Win32::FindNextFile()
            }
        }

        Win32::RemoveDirectory(folder)
    }

    [Any hope of getting the C9 code formatting bug fixed one day? Half the time it insists on putting all code on one line only, forcing me to use a quote block instead. Does anyone know of a work-around for this bug?]

  • User profile image
    BitFlipper

    I did some more testing and also added cbae's implementation. Below is a typical run:

    ---------------
    Run #0
    ---------------
    Creating tree...
    Done creating tree                  00:00:18.916
    DeleteFolderRecursiveStandard()     00:00:29.749
    
    Creating tree...
    Done creating tree                  00:00:19.925
    DeleteFolderRecursiveSyncRetry()    00:00:32.089
    
    Creating tree...
    Done creating tree                  00:00:19.523
    DeleteFolderRecursiveAsyncRetry()   00:00:22.006
    
    Creating tree...
    Done creating tree                  00:00:20.206
    DeleteFolderRecursiveFilesFirst()   00:00:12.247
    
    Creating tree...
    Done creating tree                  00:00:21.131
    DeleteFolderRecursiveCbae()         00:00:06.433

    We can see cbae's implementation is by far the fastest. Below is all of my current test code in case anyone wants to see if they can come up with a faster algorithm.

     

    using System;
    using System.Collections.Generic;
    using System.Diagnostics;
    using System.IO;
    using System.Linq;
    using System.Threading;
    using System.Threading.Tasks;
    
    namespace ParallelDelete
    {
        class Program
        {
            private const int kFileCount = 10000;
            private const int kMaxFileSize = 1000;
            private const int kAvgFoldersPerFolder = 10;
            private const int kAvgFilesPerFolder = 5;
            private const string kTestFolder = @"D:\Temp\TestFolder";
    
            private static Random m_rnd;
            private static byte[] m_rndBuffer;
            
            /// <summary>
            /// 
            /// </summary>
            /// <param name="args"></param>
            static void Main(string[] args)
            {
                var loops = 20;
    
                m_rnd = new Random(123);
                m_rndBuffer = new byte[kMaxFileSize + 10];
                m_rnd.NextBytes(m_rndBuffer);
    
                DeleteFolderRecursiveStandard(kTestFolder);
    
                for (var idx = 0; idx < loops; idx++)
                {
                    Console.WriteLine("");
                    Console.WriteLine("---------------");
                    Console.WriteLine("Run #" + idx.ToString());
                    Console.WriteLine("---------------");
                    
                    DoRun(DeleteFolderRecursiveStandard);
                    DoRun(DeleteFolderRecursiveSyncRetry);
                    DoRun(DeleteFolderRecursiveAsyncRetry);
                    DoRun(DeleteFolderRecursiveFilesFirst);
                    DoRun(DeleteFolderRecursiveCbae);
                }
    
                Console.ReadKey();
            }
    
            /// <summary>
            /// Perform a test run
            /// </summary>
            /// <param name="DeleteFunc"></param>
            private static void DoRun(Action<string> deleteFunc)
            {
                m_rnd = new Random(123);
    
                var remainFiles = kFileCount;
                var dstFolders = new List<string>(new string[] { kTestFolder });
                var sw = Stopwatch.StartNew();
    
                Console.WriteLine("Creating tree...");
                CreateFolderTree(dstFolders, ref remainFiles);
                ShowTime("Done creating tree", sw.Elapsed);
                Thread.Sleep(1000);
    
                sw = Stopwatch.StartNew();
    
                try
                {
                    deleteFunc(kTestFolder);
                }
                catch (System.Exception ex)
                {
                    ShowException(ex);
                }
    
                sw.Stop();
                ShowTime(deleteFunc.Method.Name + "()", sw.Elapsed);
                Console.WriteLine("");
            }
    
            /// <summary>
            /// Delete a folder and all of its subfolders and files
            /// </summary>
            /// <param name="folder"></param>
            private static void DeleteFolderRecursiveStandard(string folder)
            {
                if (!Directory.Exists(folder))
                    return;
    
                for (var retry = 0; retry < 5; retry++)
                {
                    try
                    {
                        Directory.Delete(folder, true);
                        return;
                    }
                    catch (System.Exception ex)
                    {
                        ShowException(ex);            
                    }
                }
            }
    
            /// <summary>
            /// Delete a folder and all of its subfolders and files
            /// </summary>
            /// <param name="folder"></param>
            private static void DeleteFolderRecursiveFilesFirst(string folder)
            {
                if (!Directory.Exists(folder))
                    return;
    
                var files = Directory.GetFiles(folder, "*", SearchOption.AllDirectories);
    
                Parallel.ForEach(files, (file) =>
                {
                    File.Delete(file);
                });
                
                //foreach (var file in files)
                //{
                //    File.Delete(file);
                //}
    
                Directory.Delete(folder, true);
            }
    
            /// <summary>
            /// Delete a folder and all of its subfolders and files
            /// </summary>
            /// <param name="folder"></param>
            private static void DeleteFolderRecursiveAsyncRetry(string folder)
            {
                if (!Directory.Exists(folder))
                    return;
    
                Exception error = null;
    
                for (var retry = 0; retry < 5; retry++)
                {
                    error = null;
                    
                    try
                    {
                        var subFolders = Directory.GetDirectories(folder);
    
                        Parallel.ForEach(subFolders, (subFolder) =>
                        {
                            DeleteFolderRecursiveAsyncRetry(subFolder);
                        });
    
                        var files = Directory.GetFiles(folder);
    
                        foreach (var file in files)
                        {
                            File.Delete(file);
                        }
    
                        break;
                    }
                    catch (System.Exception ex)
                    {
                        error = ex;
                        Thread.Sleep(100);
                    }
                }
    
                if (error != null)
                    throw error;
    
                try
                {
                    Directory.Delete(folder);
                }
                catch (Exception)
                {
                    var dc = Directory.GetDirectories(folder).Length;
                    var fc = Directory.GetFiles(folder).Length;
                    Console.WriteLine("Failed to delete folder '" + folder + "'");
                    Console.WriteLine("    Subfolders = " + dc);
                    Console.WriteLine("    Files      = " + fc);
                    throw;
                }
            }
    
            /// <summary>
            /// Delete a folder and all of its subfolders and files
            /// </summary>
            /// <param name="folder"></param>
            private static void DeleteFolderRecursiveSyncRetry(string folder)
            {
                if (!Directory.Exists(folder))
                    return;
    
                Exception error = null;
    
                for (var retry = 0; retry < 5; retry++)
                {
                    error = null;
    
                    try
                    {
                        var subFolders = Directory.GetDirectories(folder);
    
                        foreach (var subFolder in subFolders)
                        {
                            DeleteFolderRecursiveSyncRetry(subFolder);
                        }
    
                        var files = Directory.GetFiles(folder);
    
                        foreach (var file in files)
                        {
                            File.Delete(file);
                        }
    
                        break;
                    }
                    catch (System.Exception ex)
                    {
                        error = ex;
                        Thread.Sleep(100);
                    }
                }
    
                if (error != null)
                    throw error;
    
                try
                {
                    Directory.Delete(folder);
                }
                catch (Exception)
                {
                    var dc = Directory.GetDirectories(folder).Length;
                    var fc = Directory.GetFiles(folder).Length;
                    Console.WriteLine("Failed to delete folder '" + folder + "'");
                    Console.WriteLine("    Subfolders = " + dc);
                    Console.WriteLine("    Files      = " + fc);
                    throw;
                }
            }
    
            /// <summary>
            /// cbae's implementation
            /// </summary>
            /// <param name="folder"></param>
            private static void DeleteFolderRecursiveCbae(string folder)
            {
                if (!Directory.Exists(folder))
                {
                    return;
                }
    
                DeleteAllFilesCbae(folder);
    
                var fgroups = Directory.GetDirectories(folder, "*", SearchOption.AllDirectories).GroupBy(d => d.Count(s => s == '\\'),
                    d => d, (g, d) => new { Level = g, Folders = d }).Reverse();
                
                foreach (var fgroup in fgroups)
                {
                    Parallel.ForEach(fgroup.Folders, f =>
                    {
                        Directory.Delete(f);
                    });
                }
                
                Directory.Delete(folder);
            }
    
            /// <summary>
            /// cbae's implementation
            /// </summary>
            /// <param name="folder"></param>
            private static void DeleteAllFilesCbae(string folder)
            {
                if (!Directory.Exists(folder))
                    return;
    
                var files = Directory.GetFiles(folder, "*", SearchOption.AllDirectories);
    
                Parallel.ForEach(files, (file) =>
                {
                    File.Delete(file);
                });
            }
    
            /// <summary>
            /// Create a folder tree
            /// </summary>
            /// <param name="folder"></param>
            /// <param name="remainFiles"></param>
            private static void CreateFolderTree(List<string> parentFolders, ref int remainFiles)
            {
                if (remainFiles <= 0)
                    return;
    
                var subFolders = new List<string>();
    
                foreach (var parentFolder in parentFolders)
                {
                    var subFolderCount = m_rnd.Next(1, kAvgFoldersPerFolder * 2 + 1);
    
                    for (var subFolderIdx = 0; subFolderIdx < subFolderCount; subFolderIdx++)
                        subFolders.Add(Path.Combine(parentFolder, "folder" + m_rnd.Next(1000, 10000).ToString() + subFolderIdx.ToString("D2")));
                }
    
                foreach (var subFolder in subFolders)
                {
                    for (var tries = 0; tries < 5; tries++)
                    {
                        try
                        {
                            Directory.CreateDirectory(subFolder);
                            break;
                        }
                        catch (System.Exception ex)
                        {
                            Thread.Sleep(50);    
                        }
                    }
                    
                    var fileCount = m_rnd.Next(kAvgFilesPerFolder * 2 + 1);
    
                    for (var idx = 0; idx < fileCount; idx++)
                    {
                        var fileName = Path.Combine(subFolder, "file" + m_rnd.Next(1000, 10000).ToString() + idx.ToString("D2") + ".bin");
                        var fileStart = m_rnd.Next(0, kMaxFileSize - 1);
                        var fileSize = m_rnd.Next(1, kMaxFileSize - fileStart);
    
                        using (var file = File.Create(fileName))
                        {
                            file.Write(m_rndBuffer, fileStart, fileSize);
                        }
    
                        remainFiles--;
    
                        if (remainFiles <= 0)
                            return;
                    }
                }
    
                CreateFolderTree(subFolders, ref remainFiles);
            }
            /// <summary>
            /// Display the elapsed time
            /// </summary>
            /// <param name="p"></param>
            /// <param name="timeSpan"></param>
            private static void ShowTime(string opName, TimeSpan elapsed)
            {
                Console.WriteLine(opName.PadRight(35) + " " + elapsed.ToString().Substring(0, 12));
            }
    
            /// <summary>
            /// Show the exception
            /// </summary>
            /// <param name="ex"></param>
            private static void ShowException(Exception ex)
            {
                Console.WriteLine("Error: " + ex.Message);
            }
        }
    }
    

  • User profile image
    cbae

    @BitFlipper: Are you going to keep that method name in your production code? Smiley

    BTW, I tried re-running my tests after a reboot, and the times were much higher, although on relative basis, the different methods pretty much fell in line with what I previously indicated. I had no idea Windows caches that much data (~2.6 GB of files in my test). Unfortunately, I still was not able to replicate the I/O error that you received with your original code.

  • User profile image
    BitFlipper

    , cbae wrote

    @BitFlipper: Are you going to keep that method name in your production code? Smiley

    BTW, I tried re-running my tests after a reboot, and the times were much higher, although on relative basis, the different methods pretty much fell in line with what I previously indicated. I had no idea Windows caches that much data (~2.6 GB of files in my test). Unfortunately, I still was not able to replicate the I/O error that you received with your original code.

    Yes we decided we are going to ship it like that Wink

    And from past experience I know that Windows can cache a lot of data.

    Here's a bit of a complication with the DeleteFolderRecursive method... In my actual implementation I need to selectively delete folders. The tree structure I'm trying to delete is a build tree and one part of it contains a cache for components that we retrieve via network from a build farm instead of building it locally. So in addition to passing in just a folder, I do the following...

    bool DeleteFolderRecursive(string folder, Func<string, bool> queryDelete)
    {
        ...
    }

    [Wow, C9's code formatter doesn't allow "<" and ">" characters? Back to block quotes I guess. Really weird...]

    So the idea is you will get a callback for each folder and if you return true then the folder can be deleted, false to leave it. If false, all parent folders of course must not be deleted, but sibling folders that do return true must be deleted.

    So how would you rework your fancy algorithm to do that?

  • User profile image
    cbae

    , BitFlipper wrote

    *snip*

    [Wow, C9's code formatter doesn't allow "<" and ">" characters? Back to block quotes I guess. Really weird...]

    So the idea is you will get a callback for each folder and if you return true then the folder can be deleted, false to leave it. If false, all parent folders of course must not be deleted, but sibling folders that do return true must be deleted.

    So how would you rework your fancy algorithm to do that?

    The main purpose of the "fancy algorithm" Smiley is to delete a list of folders in the proper order to avoid getting the I/O error as well as to use parallel processing for efficiency. You can move the relevant code to a helper method that simply accepts IEnumberable<string> folderlist as an argument (i.e. any arbitrary list of folders).

    Your calling routine would be responsible for any "pre-processing" of a folder list prior to sending it to this helper method.

    I would probably have the following methods for flexibility.

    This editor is so FUBAR.

     

  • User profile image
    cbae

            void DeleteFolder(string folder)
            {
                DeleteFolderList(Directory.GetDirectories(folder, "*", SearchOption.AllDirectories));
            }
    
            void DeleteFolderList(IEnumerable<string> folderlist)
            {
                var fgroups = folderlist.GroupBy(d => d.Count(s => s == '\\'), d => d, (g, d) => new { Level = g, Folders = d }).Reverse();
                foreach (var fgroup in fgroups)
                {
                    Parallel.ForEach(fgroup.Folders, f =>
                    {
                        Directory.Delete(f);
                    });
                }
            }

  • User profile image
    BitFlipper

    OK some interesting results. I did runs with Windows Defender disabled and enabled. Here are the results:

    ---------------
    Run #0
    ---------------
    Creating tree...
    Done creating tree                  00:01:13.496
    DeleteFolderRecursiveStandard()     00:05:17.805
    
    Creating tree...
    Done creating tree                  00:01:11.534
    DeleteFolderRecursiveSyncRetry()    00:05:18.639
    
    Creating tree...
    Done creating tree                  00:01:16.084
    DeleteFolderRecursiveAsyncRetry()   00:03:07.796
    
    Creating tree...
    Done creating tree                  00:01:10.996
    DeleteFolderRecursiveFilesFirst()   00:02:38.414
    
    Creating tree...
    Done creating tree                  00:01:19.383
    DeleteFolderRecursiveCbae()         00:01:41.070
    
    Creating tree...
    Done creating tree                  00:01:25.039
    DeleteFolderRecursiveQuery()        00:01:31.055
    
    
    ---------------
    Run #1
    ---------------
    Creating tree...
    Done creating tree                  00:03:19.079
    DeleteFolderRecursiveStandard()     00:05:16.213
    
    Creating tree...
    Done creating tree                  00:03:15.286
    DeleteFolderRecursiveSyncRetry()    00:05:16.782
    
    Creating tree...
    Done creating tree                  00:03:15.405
    DeleteFolderRecursiveAsyncRetry()   00:02:55.496
    
    Creating tree...
    Done creating tree                  00:03:19.617
    DeleteFolderRecursiveFilesFirst()   00:02:37.973
    
    Creating tree...
    Done creating tree                  00:03:13.352
    DeleteFolderRecursiveCbae()         00:01:38.880
    
    Creating tree...
    Done creating tree                  00:03:27.035
    DeleteFolderRecursiveQuery()        00:01:24.904

    We can see a big drop in folder/file creation performance when Windows Defender is enabled. Deleting folders/files does not seem to be affected by Windows Defender.

    BTW the "query" folder delete method is below. Two interesting things... It is slightly faster than your original method. Second, I have no idea why I needed to remove the last Reverse() call. If I leave it, the folders are sorted in the wrong order. Any ideas?

            /// <summary>
            /// Wrapper call for selectively delete folders recursively
            /// </summary>
            /// <param name="folder"></param>
            private static void DeleteFolderRecursiveQuery(string folder)
            {
                DeleteFolderRecursiveQuery(folder, (queryFolder) =>
                {
                    return true;
                });
            }
    
            /// <summary>
            /// Selectively delete folders recursively
            /// </summary>
            /// <param name="folder"></param>
            /// <param name="queryDeleteFolder"></param>
            /// <returns></returns>
            private static void DeleteFolderRecursiveQuery(string folder, Func<string, bool> queryDeleteFolder)
            {
                if (!Directory.Exists(folder))
                    return;
    
                var deleteFolders = new List<string>();
                var deleteFiles = new List<string>();
    
                GetFoldersRecursiveQuery(folder, deleteFolders, deleteFiles, queryDeleteFolder);
    
                Parallel.ForEach(deleteFiles, (deleteFile) =>
                {
                    File.Delete(deleteFile);
                });
    
                var fgroups = deleteFolders.GroupBy(d => d.Count(s => s == '\\'), d => d, (g, d) => new { Level = g, Folders = d });//.Reverse();
    
                foreach (var fgroup in fgroups)
                {
                    Parallel.ForEach(fgroup.Folders, f =>
                    {
                        try
                        {
                            Directory.Delete(f);
                        }
                        catch (System.Exception ex)
                        {
                            
                        }
                    });
                }
            }
    
            /// <summary>
            /// Get a list of all folders that should be deleted
            /// </summary>
            /// <param name="folder"></param>
            /// <param name="folders"></param>
            /// <param name="queryDeleteFolder"></param>
            private static bool GetFoldersRecursiveQuery(string folder, List<string> deleteFolders,
                                                         List<string> deleteFiles, Func<string, bool> queryDeleteFolder)
            {
                var deleteFolder = true;
                var subFolders = Directory.GetDirectories(folder);
    
                if (queryDeleteFolder != null && !queryDeleteFolder(folder))
                    return false;
    
                foreach (var subFolder in subFolders)
                    deleteFolder &= GetFoldersRecursiveQuery(subFolder, deleteFolders, deleteFiles, queryDeleteFolder);
    
                if (deleteFolder)
                    deleteFolders.Add(folder);
    
                deleteFiles.AddRange(Directory.GetFiles(folder));
                return deleteFolder;
            }
    

  • User profile image
    cbae

    @BitFlipper:

    Replace the .Reverse() with:

    .OrderByDescending(d => d.Level)

    That will ensure that it's processed in the correct order no matter what.

  • User profile image
    JoshRoss

    ---------------
    Run #0
    ---------------
    Creating tree...
    Done creating tree                  00:00:13.959
    DeleteFolderRecursiveStandard()     00:00:09.089
    
    Creating tree...
    Done creating tree                  00:00:13.578
    DeleteFolderRecursiveSyncRetry()    00:00:11.765
    
    Creating tree...
    Done creating tree                  00:00:15.051
    DeleteFolderRecursiveAsyncRetry()   00:00:06.350
    
    Creating tree...
    Done creating tree                  00:00:13.608
    DeleteFolderRecursiveFilesFirst()   00:00:02.513
    
    Creating tree...
    Done creating tree                  00:00:15.878
    DeleteFolderRecursiveCbae()         00:00:01.181

    Impressive results on the SSD, for Cbea's delete method. I'm going to have to put that in a command line utility folder.

    -Josh

  • User profile image
    BitFlipper

    On my SSD drive, with and without Windows Defender enabled:

    ---------------
    Run #0
    ---------------
    Creating tree...
    Done creating tree                  00:00:16.052
    DeleteFolderRecursiveStandard()     00:00:06.216
    
    Creating tree...
    Done creating tree                  00:00:15.317
    DeleteFolderRecursiveSyncRetry()    00:00:06.937
    
    Creating tree...
    Done creating tree                  00:00:15.651
    DeleteFolderRecursiveAsyncRetry()   00:00:02.684
    
    Creating tree...
    Done creating tree                  00:00:14.728
    DeleteFolderRecursiveFilesFirst()   00:00:02.463
    
    Creating tree...
    Done creating tree                  00:00:15.185
    DeleteFolderRecursiveCbae()         00:00:01.993
    
    Creating tree...
    Done creating tree                  00:00:15.735
    DeleteFolderRecursiveQuery()        00:00:01.783
    
    
    ---------------
    Run #1
    ---------------
    Creating tree...
    Done creating tree                  00:00:04.108
    DeleteFolderRecursiveStandard()     00:00:06.114
    
    Creating tree...
    Done creating tree                  00:00:03.737
    DeleteFolderRecursiveSyncRetry()    00:00:06.799
    
    Creating tree...
    Done creating tree                  00:00:03.669
    DeleteFolderRecursiveAsyncRetry()   00:00:02.444
    
    Creating tree...
    Done creating tree                  00:00:03.781
    DeleteFolderRecursiveFilesFirst()   00:00:02.612
    
    Creating tree...
    Done creating tree                  00:00:03.854
    DeleteFolderRecursiveCbae()         00:00:01.660
    
    Creating tree...
    Done creating tree                  00:00:04.368
    DeleteFolderRecursiveQuery()        00:00:01.594

    So Windows Defender slows down writes to my SSD drive by 4 to 5 times? Hard to believe.

  • User profile image
    Blue Ink

    , BitFlipper wrote

    *snip*

    Here's a bit of a complication with the DeleteFolderRecursive method... In my actual implementation I need to selectively delete folders. The tree structure I'm trying to delete is a build tree and one part of it contains a cache for components that we retrieve via network from a build farm instead of building it locally. So in addition to passing in just a folder, I do the following...

    *snip*

    I guess you should get marginally better performance by using the Directory.EnumerateXxx methods instead of the GetXxx ones.

    Not a major improvement, but might be worth a try.

    The only obvious disadvantage is that you cannot report progress.

  • User profile image
    BitFlipper

    @Blue Ink:

    I implemented your suggestion by changing cbae's DeleteAllFiles:

    /// <summary>
    /// cbae's implementation
    /// </summary>
    /// <param name="folder"></param>
    private static void DeleteAllFilesCbae(string folder)
    {
        if (!Directory.Exists(folder))
            return;
    
        var files = Directory.GetFiles(folder, "*", SearchOption.AllDirectories);
    
        Parallel.ForEach(files, (file) =>
        {
            File.Delete(file);
        });
    }

    ...to...

    /// <summary>
    /// cbae's implementation with Blue Ink's Directory.EnumerateXxx suggestion
    /// </summary>
    /// <param name="folder"></param>
    private static void DeleteAllFilesCbaeBlueInk(string folder)
    {
        if (!Directory.Exists(folder))
            return;
    
        Parallel.ForEach(Directory.EnumerateFiles(folder, "*", SearchOption.AllDirectories), (file) =>
        {
            File.Delete(file);
        });
    }
    

     

    Below are the results. Interesting.

    File count    = 50000
    Max file size = 100
    Avg folders   = 4
    Avg files     = 8
    Path          = D:\Temp\TestFolder
    
    ---------------
    Run #0
    ---------------
    Creating tree...
    Done creating tree                  00:01:11.545
    DeleteFolderRecursiveStandard()     00:02:11.975    1.00
    
    Creating tree...
    Done creating tree                  00:01:11.029
    DeleteFolderRecursiveCbae()         00:00:30.207    4.37
    
    Creating tree...
    Done creating tree                  00:01:12.892
    DeleteFolderRecursiveCbaeBlueInk()  00:00:21.372    6.17
    
    Creating tree...
    Done creating tree                  00:01:12.714
    DeleteFolderRecursiveQuery()        00:00:28.205    4.68
    
    
    ---------------
    Run #1
    ---------------
    Creating tree...
    Done creating tree                  00:01:11.903
    DeleteFolderRecursiveStandard()     00:02:08.346    1.00
    
    Creating tree...
    Done creating tree                  00:01:10.548
    DeleteFolderRecursiveCbae()         00:00:29.414    4.36
    
    Creating tree...
    Done creating tree                  00:01:11.879
    DeleteFolderRecursiveCbaeBlueInk()  00:00:20.904    6.14
    
    Creating tree...
    Done creating tree                  00:01:11.452
    DeleteFolderRecursiveQuery()        00:00:24.815    5.17
    
    
    ---------------
    Run #2
    ---------------
    Creating tree...
    Done creating tree                  00:01:13.300
    DeleteFolderRecursiveStandard()     00:02:10.624    1.00
    
    Creating tree...
    Done creating tree                  00:01:14.106
    DeleteFolderRecursiveCbae()         00:00:27.068    4.83
    
    Creating tree...
    Done creating tree                  00:01:12.720
    DeleteFolderRecursiveCbaeBlueInk()  00:00:21.526    6.07
    
    Creating tree...
    Done creating tree                  00:01:13.552
    DeleteFolderRecursiveQuery()        00:00:25.749    5.07

  • User profile image
    cbae

    @BitFlipper: Parallel.ForEach() must be doing some expensive transformation of the IEnumerable<T> to a collection that is more easily partitioned into multiple threads (i.e. something that has GetRange()).

  • User profile image
    Blue Ink

    @BitFlipper: That's really surprising. I did expect some improvement from better cache performance and from being able to spawn tasks right away, but I didn't expect it to be that noticeable. Good to know.

    What is really surprising, though, is that I couldn't replicate your original results:

    ---------------Run #0---------------
    Creating tree...Done creating tree 00:03:08.807
    DeleteFolderRecursiveStandard() 00:00:45.119
    Creating tree...Done creating tree 00:02:45.864
    DeleteFolderRecursiveSyncRetry() 00:00:27.545
    Creating tree...Done creating tree 00:02:11.058
    DeleteFolderRecursiveAsyncRetry() 00:00:24.427
    Creating tree...Done creating tree 00:02:07.927
    DeleteFolderRecursiveFilesFirst() 00:00:34.049
    Creating tree...Done creating tree 00:02:18.533
    DeleteFolderRecursiveCbae() 00:00:36.724
    Creating tree...Done creating tree 00:02:12.798
    DeleteFolderRecursiveCbaeBlueInk() 00:00:25.751
    ---------------Run #1---------------
    Creating tree...Done creating tree 00:02:14.448
    DeleteFolderRecursiveStandard() 00:00:19.669
    Creating tree...Done creating tree 00:02:01.838
    DeleteFolderRecursiveSyncRetry() 00:00:20.027
    Creating tree...Done creating tree 00:02:03.387
    DeleteFolderRecursiveAsyncRetry() 00:00:28.846
    Creating tree...Done creating tree 00:02:09.768
    DeleteFolderRecursiveFilesFirst() 00:00:31.808
    Creating tree...Done creating tree 00:02:13.899
    DeleteFolderRecursiveCbae() 00:00:35.900
    Creating tree...Done creating tree 00:02:02.326
    DeleteFolderRecursiveCbaeBlueInk() 00:00:25.571
    ---------------Run #2---------------
    Creating tree...Done creating tree 00:02:05.843
    DeleteFolderRecursiveStandard() 00:00:18.936
    Creating tree...Done creating tree 00:02:06.574
    DeleteFolderRecursiveSyncRetry() 00:00:20.139
    Creating tree...Done creating tree 00:02:02.603
    DeleteFolderRecursiveAsyncRetry() 00:00:21.381
    Creating tree...Done creating tree 00:02:09.475
    DeleteFolderRecursiveFilesFirst() 00:00:31.912
    Creating tree...Done creating tree 00:02:03.150
    DeleteFolderRecursiveCbae() 00:00:36.180
    Creating tree...Done creating tree 00:02:05.772
    DeleteFolderRecursiveCbaeBlueInk() 00:00:22.872

     

    Much to my surprise, the fastest method in this part of town seems to be DeleteFolderRecursiveStandard(), by a large margin.

    I took the code from your earlier post, and just added the EnumerateFiles method at the end. Test conditions were pretty close to the ones of your last test (50000 files, 100 max file size, 4 folders, 8 files).

    As you can tell from the other timings, it's not a fast machine. I wonder if it couldn't be the OS...

  • User profile image
    cbae

    @Blue Ink: I misread the times from BitFlipper's last post. For some reason, there was a second column of numbers in his list of times, and I thought those were the times. LOL.

    Aside from the relative difference in times between methodologies, it's odd that your results had such wild variations from run to run as well. Are you sure that you didn't have some background process running?

Conversation locked

This conversation has been locked by the site admins. No new comments can be made.