Comment Feed for My idea: file distribution reimagined (Coffeehouse on Channel 9)

Re: Re: Re: Re: Re: Re: Re: Re: Re: Re: Re: My idea: file distribution reimagined

blowdart — Mon, 13 Jul 2009 16:50:33 GMT

SHA1 collisions are at possible in 2^52 iterations right now. It's not easy, but it's doable.

Re: Re: Re: Re: Re: Re: Re: Re: Re: Re: My idea: file distribution reimagined

Long Zheng — Mon, 13 Jul 2009 16:47:41 GMT

I don't think I said there's no possibility of collisions. Like someone said, the chances always exist, but its very small.

On the other hand however, SHA-1, SHA-256 and SHA-512 has not been compromised. By that I mean there's no known way to artificially create two files that have the same hash. Something that MD5 has recently failed against.

Re: Re: Re: Re: Re: Re: Re: Re: Re: My idea: file distribution reimagined

blowdart — Mon, 13 Jul 2009 16:20:53 GMT

True, but saying there's no possibility of collisions is 100% wrong.

Re: Re: Re: Re: Re: Re: Re: Re: Re: Re: My idea: file distribution reimagined

Bent Rasmussen — Mon, 13 Jul 2009 16:18:13 GMT

Yes but with these odds, who really cares. I like Sven's idea, I've had the same thought myself but not sure it's worth the extra bits.

Re: My idea: file distribution reimagined

Allan Lindqvist — Mon, 13 Jul 2009 12:22:35 GMT

isnt that how overnet/edonkey/emule works? there you have a hash representing the file and a bunch of chunks also with their own hashes, then you ask around (your peers/a central server) who has the hashes you want :)

Re: Re: Re: Re: Re: Re: Re: Re: Re: My idea: file distribution reimagined

Tommy Carlier — Mon, 13 Jul 2009 10:41:49 GMT

Yesterday I watched a video of a presentation by Scott Hanselman and Phil Haack and Scott said something that is very appropriate here: “A system is very secure until it is not.”

You cannot tell that a system is 100% secure. It's secure until someone breaks it.

Re: Re: Re: Re: Re: Re: Re: Re: My idea: file distribution reimagined

Long Zheng — Mon, 13 Jul 2009 10:08:27 GMT

You could argue nothing is definitive, except very close to 0 or 1 :P

Re: Re: Re: Re: Re: Re: Re: My idea: file distribution reimagined

OnlyJack — Mon, 13 Jul 2009 08:53:05 GMT

Shareaza tried the same integration between http/torrent/emule/gnutella and other p2p network: the program itself is amazing and has one of the best GUIs ever seen, however this sort of integration failed miserably and never took off.

I think it's already a lost battle, if you want to make an useful project please make an adblocker for IE that doesn't suck.

Re: Re: Re: Re: Re: Re: Re: My idea: file distribution reimagined

blowdart — Mon, 13 Jul 2009 08:46:21 GMT

Ah no, lets be strict. It's not impossible, it's just very very improbable. There is a difference and all hash algorithms will collide at some point, it's in their nature.

Re: Re: Re: Re: Re: Re: My idea: file distribution reimagined

Long Zheng — Mon, 13 Jul 2009 06:37:31 GMT

Under the SHA-256 hashing algorithm, it is currently impossible to compromise hashes.

There is no "filing" invalid hashes or "assigning" hashes to files they don't belong to. A hash either matches or it doesn't. You're assumping a hash has some sort of relationship to files in the database. It doesn't.

This is the same way most online services check if you are the user you claim. They store your username and your password's hash. If it matches, then it's assumed you know your original password, without it ever storing your original password. If it doesn't match, then it doesn't really care what you typed, it didn't match.

Re: Re: Re: Re: Re: My idea: file distribution reimagined

Christian Liensberger — Mon, 13 Jul 2009 04:21:10 GMT

How hard will it be to compromise hashes, or file invalid hashes, or assign hashes to files they don't belong to?

Re: Re: Re: Re: My idea: file distribution reimagined

Long Zheng — Mon, 13 Jul 2009 02:23:40 GMT

Actually

If I download a file from microsoft.com/download/ I'm sure that this one comes from Microsoft. There is little reasons to not believe that.

That is because you "trust" Microsoft's website, their URLs and their download servers. You'll have to utilise the same trust in the distribution system I've explained, but you no longer have to trust the servers, since as long as hashes have not been compromised (which SHA-256 has not), it can be guaranteed if you trusted what Microsoft wanted to provide to you, you'll get it.

Re: Re: Re: My idea: file distribution reimagined

Sven Groot — Mon, 13 Jul 2009 01:41:42 GMT

Using a combination of hash and file size would also decrease the odds of a collision, without needing extra hashes. If there is a collision there is no other option but to show all possible matches and let the user decide.

Re: Re: Re: Re: My idea: file distribution reimagined

Bent Rasmussen — Sun, 12 Jul 2009 22:07:27 GMT

Just get the hash and find statements about the hash. If you don't find any conflicting statements on it or any negatives, then you may have some trust in it. If you do, then distrust it. No need to rely on one service's claims about a particular hash. Also, if you trust the provider, you could also have several users supply file metadata. I believe Bitzi has a tool that hashes and supplies metadata to the service. If so, then multiple reruns would provide statistical basis for trust. So you bind metadata to the context of a user. If a user is malicious, then all metadata from that user can be wiped out or hidden.

Re: Re: Re: Re: Re: My idea: file distribution reimagined

Bent Rasmussen — Sun, 12 Jul 2009 21:47:53 GMT

True. That would completely defeat any purpose.

Re: Re: Re: My idea: file distribution reimagined

Christian Liensberger — Sun, 12 Jul 2009 17:52:45 GMT

I still have a problem with identification. What if two files have the same hash and a bad guy wanted that to happen. You probably need to have both files up. Now if you validate you don't know which file it is.

The other thing is if you have two hashes for the same file. In that case a virus/malware could be identified as being something else because the calculated hash matches with one of these two hashes...

If I download a file from microsoft.com/download/ I'm sure that this one comes from Microsoft. There is little reasons to not believe that. If I validate a file with your service I'm never sure because it is the users who generate the data. There's a lot of noise going on... I'm not sure if something like that qualifies as a robust service!

Re: Re: Re: Re: Re: My idea: file distribution reimagined

AndyC — Sun, 12 Jul 2009 17:39:48 GMT

But in order to do that, the hash needs to have as many bits as the file. Sure, from a security point of view it's very difficult to intentionally generate a file with the same hash, but if you're trying to uniquely identify a file from potentially everything in existence it becomes a bit of a problem. Especially when your child downloads XXX material as opposed to the latest Disney cartoon....

It's not an issue for GUIDs so much because (a) the time factor makes it unlikely you'd regenerate the same one, (b) even if you did generate the same one again, it's unlikely you'd also be using it for the same purpose.

Re: Re: Re: Re: My idea: file distribution reimagined

Long Zheng — Sun, 12 Jul 2009 14:12:38 GMT

If you use anything besides a hash, it breaks the system. You need a method to identify unique files in a way that can be reproduced with 100% certainty on at least two different computers that has no communication with each other. You cannot do this with GUIDs.

Well you could, but that'd require you to query a server every time you needed to identify a file, which is not applicable in the nature of the internet today.

Re: Re: Re: My idea: file distribution reimagined

blowdart — Sun, 12 Jul 2009 14:06:38 GMT

Actually a more sensible option IMO, rather than use two algorithms is to salt the hash with a unique value per file, say, a GUID. It depends what you think the hash is for - uniquely identifying a file, or for guaranteeing it hasn't changed since it was uploaded.

Re: Re: My idea: file distribution reimagined

Long Zheng — Sun, 12 Jul 2009 12:54:18 GMT

First, I would like to quote a very insightful comment on "hash collisions" on my blog.

http://www.istartedsomething.com/20090710/reimagining-file-distribution-universal-downloads/#comment-76909

Okay, but even if the odds are less than a single atom in the known universe, doesn't mean it couldn't happen. I guess there's two ways of looking at this. The first way is to simply ignore it, and that's what I believe GUID does. It since turned out pretty good for GUIDs. The other method is to try to reduce the chances even more, and with that you could concatenate another hash string generated by a different hash function. This doesn't decrease the security risk, but it does reduce the risk of a single hash collision.

Once the project gets started (which the odds are currently looking pretty good), I'll obviously have to do more real world testing and see which path to go down.

Re: My idea: file distribution reimagined

AndyC — Sun, 12 Jul 2009 11:37:12 GMT

But how do you deal with the fact that multiple files will have the same hash (basic rules of entropy)? It might start off ok, but eventually you need some way to say File X isn't the same as File Y, regardless of the fact they have the same hash.

Re: Re: My idea: file distribution reimagined

Long Zheng — Sun, 12 Jul 2009 08:12:51 GMT

how is the authority that makes sure that a certain hash really represents a certain file

Easy. We don't care about what the file is, or says it is. That is ultimately up to the provider. It's the same trust you have with every provider right now with filenames. I trust the filenames on Microsoft's servers are indicative of the files they say they are. I trust the files on Sourcefourge are for the downloads I was looking for.

The whole reason I need hashes for this system is so that files are normalized. Their content is what's important. You can name a file any number of ways. But for distributing the file, it doesn't really matter what it's called. It matters though where it reaches its destination, which is why using magnet links you can still append a "suggested name" to the result. That has the same affect as filenames in a HTTP URL. It is suggested, but you don't always have to save it as such.

Re: My idea: file distribution reimagined

Christian Liensberger — Sat, 11 Jul 2009 22:50:19 GMT

The idea is nice. :) But there are a still some concerns that I have and that need to be thought out before this could roll. One of them is: how is the authority that makes sure that a certain hash really represents a certain file? if that is done by the community, you need to allow to attach multiple files with the same hash. otherwise someone could "block" a hash. now it might happen that I get multiple results for one hash.

Re: Re: Re: Re: Re: Re: My idea: file distribution reimagined

Christian Liensberger — Sat, 11 Jul 2009 22:40:34 GMT

that's also what came immediately into my mind when I read the post :D

Re: Re: Re: My idea: file distribution reimagined

May28th2018 — Sat, 11 Jul 2009 17:31:42 GMT

Yum and apt-get take a human readable name as an argument

IE, I want firefox

yum install firefox

Or you can do it via the yum or apt gui. There is no need for any further complexity.

Sergey Brin had a similar idea using hashes for copy protection on the web called COPS.

http://infolab.stanford.edu/~sergey/copy.html

document chunk hash psuedo code here.

Sergey's idea resembles bittorrent more than it does longzheng's file dist system though as bittorrent tracks files by dividing it up into pieces and assigning hash integrity keys to them.

At any rate it never materialized. I don't see this hash based software repository system for Windows materializing either. The only companies capable of implementing it successfuly would be download.com, wise, installshield, ect... Otherwise it would have no traction or interest.