Tech Off Thread

16 posts

Forum Read Only

This forum has been made read only by the site admins. No new threads or comments can be added.

Downloading a file with more then 1 connection

Back to Forum: Tech Off
  • User profile image
    Chadk

    Im thinking about doing this downloader manager, as i dont like the ones thats on the market atm.

    I want to support to split up the download, and download it in parts, like most modern clients do, aswell as resume support, etc.

    But i have no idea how to do this.

    So i was wondering if anyone had a link, where it is explained how to technicaly do it.
    Or maybe an opensource project, that does it?

    Im gonna do it in C#.

  • User profile image
    jonorossi

    Have you tried Flashget? Why don't you like the current ones?

  • User profile image
    Chadk

    jonorossi wrote:
    Have you tried Flashget? Why don't you like the current ones?

    Yes. And i dont dare install thigs like that. Many tends to show banners in the application, or even come with adware.

    I also like doing programs myself, so i can fix bugs, add features i might want, etc.

  • User profile image
    Sven Groot

    To do this, you need to specify you want only part of a file, and for multiple connections you obviously will need to send multiple requests at once (on different threads or something) for different parts of the file.

    For HTTP, you need to use the Range header. I think at least in theory you should be able to use the HttpWebRequest class, add that header, and then read the response as a stream and save it into the appropriate positions in the result file.

    For FTP, if you are using .Net 2.0 it's easy, since the FtpWebRequest class has a ContentOffset property which specifies where in the file downloading should start.

    EDIT: I use GetRight myself. It's not free, but no ads or anything, and it has served me well for nearly a decade now. Smiley

  • User profile image
    PerfectPhase

    Chadk wrote:
    
    jonorossi wrote: Have you tried Flashget? Why don't you like the current ones?

    Yes. And i dont dare install thigs like that. Many tends to show banners in the application, or even come with adware.

    I also like doing programs myself, so i can fix bugs, add features i might want, etc.


    I'm all for the do it myself mentality, but surly there are more interesting projects to spend your time on!

    I use free download manager and it is clean, no spy ware and doesn’t phone home (apart from updates), I have checked this quite closely!

    EDIT: there is also an open source downlaod manger you might want to have a look at http://francis.dupont.free.fr/truedownloader/index.html

  • User profile image
    Chadk

    PerfectPhase wrote:
    

    I'm all for the do it myself mentality, but surly there are more interesting projects to spend your time on!


    Im bad at finding new projects to do.

    If you give me a project that is more interresting then this one, ill drop it.

    Seems like you are good at finding new projects, when you can say something like that Smiley

  • User profile image
    littleguru

    I was using FlashGet myself. It is full of ads... Keep us informed about your download manager project.

  • User profile image
    Rossj

    A mixture of cat and curl (with -r). no adverts.

  • User profile image
    Chadk

    Rossj wrote:
    A mixture of cat and curl (with -r). no adverts.

    That made no sense to me.


    So basicly what i do, is:

    1. Find out the lenght of the file. I guess this is done by sending a dummy request, and get the lenght
    2. Find out how many connections to use, and how much each connection is gonna download(I guess its lenght/connections).
    3. Save the data from each from each connection into its own file
    4. When its done, i open the files 1 by 1, in a certain order(The first data, first, obivious), and append it into one large file.

    Am i correct?

  • User profile image
    Rossj

    Chadk wrote:
    
    Rossj wrote:A mixture of cat and curl (with -r). no adverts.

    That made no sense to me.


    It was a joke, nevermind.

    Chadk wrote:
    
    1. Find out the lenght of the file. I guess this is done by sending a dummy request, and get the lenght


    A HEAD request should tell you this.


    Chadk wrote:
    
    2. Find out how many connections to use, and how much each connection is gonna download(I guess its lenght/connections).


    Yes, but I wouldn't go crazy with the number of connections, economies of scale and all that. Don't forget to use the range header to just get a block of data, and handle it elegantly if the server does not support it.

    Chadk wrote:
    
    3. Save the data from each from each connection into its own file


    Or write a file of the correct size and just write inplace at the correct locations?

  • User profile image
    figuerres

    Chadk wrote:
    Im thinking about doing this downloader manager, as i dont like the ones thats on the market atm.

    I want to support to split up the download, and download it in parts, like most modern clients do, aswell as resume support, etc.

    But i have no idea how to do this.

    So i was wondering if anyone had a link, where it is explained how to technicaly do it.
    Or maybe an opensource project, that does it?

    Im gonna do it in C#.



    well depending on what you want...

    I know there are at least 2 ports of bit torrent to c# you could look at.


    question:  why the "more than one connection" ?

    if I have a file on server x and I am at pc Y the file will come down to me at a given speed.

    if I have two connections 2 server X then the speed per connection will be slower then 1 connection.
    it's not faster.

    now if I have the same file on 3 servers ( X1,X2,X3)
    and my PC has a large connection then I could pull some of the file from each server and make it faster as long as I could get it faster then any one server could send it to me.
    YMMV and all that....

    so do you have a way to have the same file on N ( more than one)  servers ?

    if so then BitTorrent is the thing to look at.

    if not then just FTP or HTTP download the files.....

    or also..... look at Arg.... the name -- it's the MSFT thing for back ground downloads -- not fast -- it's for using idle time to download in the background....

    BITS -- Background Initelegent Transfer Service

    part of XP and I have seen a .net wrapper for it in the PAG work for click once deployment with .net 1.1

    PAG is a MSFT thing for how to build good software etc....

  • User profile image
    figuerres
  • User profile image
    ElucidWeb

    I wrote one in VB6 a while back, its quit simple actually.  HTTP supports this header attribute called Partial and all you do is pass to partial what chunk of the binary file you want to recieve so I cant recall the exact syntax but its something like Partial: 12345/94873 bytes an d it will sent you the file starting at byte 12345 and ending at byte 94873 so if you have a bunch of download classes and then a download manager class all you need to do is spawn multiple download classes asynchronously and then have them report to your download manager class. When each completes the download manager class can manage the pieces and piece it all together as long as each class maintains its proper identity and ordering sequence the binary file you get in the end is the combination of multiple chunks! Its a great learning experience to write something like this, however with the invent of BitTorrent and High Speed Pipes to the home these days its usefulness has been lost. The FTP part of it is a bit harder to do but its the same similar concept.  I attepted to write a BitTorren application in .NET a while back but the documentation on the protocol is very confusing and not very good.  Also it has some rather strange ways of doing things, the protocol doesnt really use any standards at all its all custom stuff which is sorta (I need to watch my language) backwards.  What I have been thinking of doing is writing a new version of BitTorren that uses XML and all that non sense istead of Bencoding and all the wierdness that BitTorrent uses.  

  • User profile image
    figuerres

    ElucidWeb wrote:
    I wrote one in VB6 a while back, its quit simple actually.  HTTP supports this header attribute called Partial and all you do is pass to partial what chunk of the binary file you want to recieve so I cant recall the exact syntax but its something like Partial: 12345/94873 bytes an d it will sent you the file starting at byte 12345 and ending at byte 94873 so if you have a bunch of download classes and then a download manager class all you need to do is spawn multiple download classes asynchronously and then have them report to your download manager class. When each completes the download manager class can manage the pieces and piece it all together as long as each class maintains its proper identity and ordering sequence the binary file you get in the end is the combination of multiple chunks! Its a great learning experience to write something like this, however with the invent of BitTorrent and High Speed Pipes to the home these days its usefulness has been lost. The FTP part of it is a bit harder to do but its the same similar concept.  I attepted to write a BitTorren application in .NET a while back but the documentation on the protocol is very confusing and not very good.  Also it has some rather strange ways of doing things, the protocol doesnt really use any standards at all its all custom stuff which is sorta (I need to watch my language) backwards.  What I have been thinking of doing is writing a new version of BitTorren that uses XML and all that non sense istead of Bencoding and all the wierdness that BitTorrent uses.  



    I may be wrong but I was under the impression that some of the stuff BT does with the data is what makes it able to give a valid file in the end and know what parts are mssing or corrupt.

    perhaps I'll go back to the DNR audio and review what they said about how it works ....

  • User profile image
    ElucidWeb

    figuerres wrote:
    
    ElucidWeb wrote: I wrote one in VB6 a while back, its quit simple actually.  HTTP supports this header attribute called Partial and all you do is pass to partial what chunk of the binary file you want to recieve so I cant recall the exact syntax but its something like Partial: 12345/94873 bytes an d it will sent you the file starting at byte 12345 and ending at byte 94873 so if you have a bunch of download classes and then a download manager class all you need to do is spawn multiple download classes asynchronously and then have them report to your download manager class. When each completes the download manager class can manage the pieces and piece it all together as long as each class maintains its proper identity and ordering sequence the binary file you get in the end is the combination of multiple chunks! Its a great learning experience to write something like this, however with the invent of BitTorrent and High Speed Pipes to the home these days its usefulness has been lost. The FTP part of it is a bit harder to do but its the same similar concept.  I attepted to write a BitTorren application in .NET a while back but the documentation on the protocol is very confusing and not very good.  Also it has some rather strange ways of doing things, the protocol doesnt really use any standards at all its all custom stuff which is sorta (I need to watch my language) backwards.  What I have been thinking of doing is writing a new version of BitTorren that uses XML and all that non sense istead of Bencoding and all the wierdness that BitTorrent uses.  



    I may be wrong but I was under the impression that some of the stuff BT does with the data is what makes it able to give a valid file in the end and know what parts are mssing or corrupt.

    perhaps I'll go back to the DNR audio and review what they said about how it works ....


    It is, in fact when I was attempting to write the BT Application I read so many documents on how BT works my brain was hurting.  But alot of the stuff it does the way it does it is done a particular way for no reason.  Its just done that way because its done that way, which to me is probably because of Brams initial design and it was never changed.

  • User profile image
    staceyw

    figuerres wrote:
    
    Chadk wrote: Im thinking about doing this downloader manager, as i dont like the ones thats on the market atm.

    I want to support to split up the download, and download it in parts, like most modern clients do, aswell as resume support, etc.

    But i have no idea how to do this.

    So i was wondering if anyone had a link, where it is explained how to technicaly do it.
    Or maybe an opensource project, that does it?

    Im gonna do it in C#.

    well depending on what you want...

    I know there are at least 2 ports of bit torrent to c# you could look at.


    question:  why the "more than one connection" ?

    if I have a file on server x and I am at pc Y the file will come down to me at a given speed.

    if I have two connections 2 server X then the speed per connection will be slower then 1 connection.
    it's not faster.

    now if I have the same file on 3 servers ( X1,X2,X3)
    and my PC has a large connection then I could pull some of the file from each server and make it faster as long as I could get it faster then any one server could send it to me.
    YMMV and all that....

    so do you have a way to have the same file on N ( more than one)  servers ?

    if so then BitTorrent is the thing to look at.

    if not then just FTP or HTTP download the files.....

    or also..... look at Arg.... the name -- it's the MSFT thing for back ground downloads -- not fast -- it's for using idle time to download in the background....

    BITS -- Background Initelegent Transfer Service

    part of XP and I have seen a .net wrapper for it in the PAG work for click once deployment with .net 1.1

    PAG is a MSFT thing for how to build good software etc....


    I have to agree. In most cases, you will just be putting more overhead on the wire for no gain.  Unless the server is the bottleneck and does some kind of throtle per socket, so multiple sockets could potentially speed it up, but with todays servers it is probably a waste of time.  Now for very large production data (i.e. 1+ GB) and multiple high speed pipes between partners, I can see value in it.

Conversation locked

This conversation has been locked by the site admins. No new comments can be made.