Tech Off Thread

9 posts

Forum Read Only

This forum has been made read only by the site admins. No new threads or comments can be added.

sending file through web services

Back to Forum: Tech Off
  • User profile image
    matrixr

    i'm transfering files that are about 0-100mb in size through a web service, what is a recommended chunk size?

  • User profile image
    figuerres

    matrixr wrote:
    i'm transfering files that are about 0-100mb in size through a web service, what is a recommended chunk size?


    dare I ask WHY?

    in soap? or binary?

    why not just use http to do the data and just use a ws to manage the xfer?

    like FTP kinda....

    control on one connection, data on the other.

  • User profile image
    matrixr

    i cannot change the web service, method signature is as follows:
    [WebMethod]public byte[] Download(string file, int offset, int bufferSize) { }

  • User profile image
    Matthew van Eerde

    What's your network throughput to the server?

  • User profile image
    matrixr

    ~100-200k/s

    another question, using performance monitor i see gen0:gen1 is 2:1, gen1:gen2 is 1:1 and % in gc is > 6% when downloading a file around 50mb using 100kb chunks.

    here is the method that calls the web service
    void Download(string file)
    {
        BinaryWriter br;
        try
        {
           br = new BinaryWriter(File.Open(...));
           int fileSize = ws.FileSize(file);

           while (bytesLeftToDownload)
           {
              br.Write(ws.Download(file, offset, chuk));
           }
           br.Close();

           CompareHash(...);
        }
        catch finally ...etc
    }

    string CompareHas(...)
    {
        using (FileStream fs = new FileStream(...))
        {
              MD5 md5 = new MD5Crypto...;
              byte[] hash = md5.ComputeHash(fs);
              StringBuilder sb = new StringBuilder(32);
              for (int i=0; i<hash.length; i++)
              {
                 sb.Append(hash[i].ToString("X2"));
              }
              return sb.ToString();
        }
        return "";
    }

    this is a windows service by the way, tried using CLR Profiler but it didnt work.

  • User profile image
    Rossj

    I had to switch back to FF to post this because of formatting in Safari Sad


    matrixr wrote:
    ~100-200k/s
    void Download(string file)
    {
        BinaryWriter br;
        try
        {
           br = new BinaryWriter(File.Open(...));
           int fileSize = ws.FileSize(file);

           while (bytesLeftToDownload)
           {
              br.Write(ws.Download(file, offset, chuk));
           }
           br.Close();

           CompareHash(...);
        }
        catch finally ...etc
    }


    The thing that worries me the most is that the best use of WS is to make chatty requests, rather than frequent requests, and the call to Download() is going to be made a lot of times (also the fileSize doesn't seem to be used).

    You are duplicating a lot of functionality here, I would be tempted to use plain HTTP and provide the MD5 in the header if you know it ... something like ..

    GET /DownloadFile.aspx?name=thefilename
    Host: Myhost
    MD5: Current-MD5-IfKnown

    That way DownloadFile.aspx can just tell you that the file has not changed (based on the MD5). If you don't have it then you send an empty MD5 and you will receive the file ... in a lot more efficient way that the current method.

    Making such frequent requests using SOAP (or even the binary formatter) is going to perform badly and waste precious bandwidth.  How much smaller is that GET request above than a single call to Download() ?

    I don't know if you are stuck with the WS, but I would suggest having a quick chat with the system designer and ask if maybe they could fix the design.

  • User profile image
    Sven Groot

    Although I think if possible you should try another solution, if you must use WS to transfer files I suggest you use the WS-Attachments specification, which MS has implemented in the Web Services Enhancements.

    Note that the .Net Web Services engine will parse the entire message into a DOM, so sending a 100MB file that way will cost enormous amounts of server memory and likely also time.

  • User profile image
    Matthew van Eerde

    Rossj wrote:
    You are duplicating a lot of functionality here, I would be tempted to use plain HTTP and provide the MD5 in the header if you know it ... something like ..

    GET /DownloadFile.aspx?name=thefilename
    Host: Myhost
    MD5: Current-MD5-IfKnown

    That way DownloadFile.aspx can just tell you that the file has not changed (based on the MD5). If you don't have it then you send an empty MD5 and you will receive the file


    HTTP already has a similar method relying on dates rather than hashes.

    Have the server added a Last-Modified-On header in responses, containing the modification date of that file.  (The exact format of the date is important... see the HTTP spec for details.)

    The browser will then remember that date for that file.  (Most browsers already implement this functionality.)

    From then on, when that user tries to download the file again, the browser will (invisibly to the user) add an If-Modified-Since header with that last-modified date.

    The server should keep an eye out for If-Modified-Since headers.  If the If-Modified-Since date comes after the file's Last-Modified-On date, the server should issue a
    304 Not Modified
    status code instead of sending the file again.  Most browsers will understand this status code, and simply show the user the file they already have available.

  • User profile image
    littleguru

    Best is using attachements (as Sven mentioned). 

    Without them you need to transfer the double size. Webservices are encoding binary data into a format that allows them to display them in XML (if you are using SOAP - for binary it does not matter). The so called: Base64 encoding.

Conversation locked

This conversation has been locked by the site admins. No new comments can be made.