CCR Programming - Jeffrey Richter and George Chrysanthakopoulos

I'd prefer Blaise to answer your questions here I am far from qualified to explain this stuff...
C
Charles wrote:Microscopy is certainly an area where this could be highly useful. Consider also astronomy... Navigating galaxies and other celestial bodies will never be the same!
C
Charles wrote:I sent a C9 join request mail to Blaise... As you might imagine, he's a rather busy fellow...
C
jsampsonPC wrote:Is Blaise online as a Niner?
I am still wondering how exactly it shows "versions?" of the images as you zoom in and out, if it's drawing from a single image...does it only display a portion of the image's binary?
I wonder if someone has already thought about combining this with
Microsoft Max. That would be an interesting new twist on the whole 3D presentation stuff.
Though the problem with Max currently is that there haven't been new versions for a very long time. It once started as a showcase application for WPF, but according to the Max forum people are very upset that the last supported WinFX CTP is the January(!) CTP.
mawcc wrote:
jsampsonPC wrote:Is Blaise online as a Niner?
I am still wondering how exactly it shows "versions?" of the images as you zoom in and out, if it's drawing from a single image...does it only display a portion of the image's binary?
I can only guess, but I think that's exactly what's happening. When zoomed out, you only need to load e.g. every 10th pixel of the image. If you zoom in a little you have to add e.g. every 5th pixel and so on, until you are on zoom level 1:1 which displays all pixels. At the same time the portion of the image in view gets smaller and smaller, so the overall amount of data that has to be in memory basically stays the same.
So you either see all of the image in low resolution or a small portion of the image in high resolution. The tricky parts of course are to decide when to load what and how to do it quickly.
Charles wrote:Imagine using this technology for "touring" 3-D image compositions of planetary bodies, like Titan. The usefulness of this technology for scientific imaging is incredible. Astronomy, biology, ecology, you name it. I am so excited right now.
jsampsonPC wrote:
mawcc wrote:
jsampsonPC wrote: Is Blaise online as a Niner?
I am still wondering how exactly it shows "versions?" of the images as you zoom in and out, if it's drawing from a single image...does it only display a portion of the image's binary?
I can only guess, but I think that's exactly what's happening. When zoomed out, you only need to load e.g. every 10th pixel of the image. If you zoom in a little you have to add e.g. every 5th pixel and so on, until you are on zoom level 1:1 which displays all pixels. At the same time the portion of the image in view gets smaller and smaller, so the overall amount of data that has to be in memory basically stays the same.
So you either see all of the image in low resolution or a small portion of the image in high resolution. The tricky parts of course are to decide when to load what and how to do it quickly.
But the amount of data cannot stay the same over the entire experience. He's got, roughly 100, pictures on his screen. As he moves any one of them to a higher Z-index, that picture has to display more and more of it's full binary-value. As he moves it to a lesser Z-index, it displays less of its binary-data. That would require the user to first traverse through its entire binary-source, select only what the user needs to keep in memory for that moment, and then display the selected information. However, if you're going to be zooming in and out rapidly, then it would be best to have the entire pictures memory in cache, but then, you would need EVERY picture's highest-resolution binary-value in cache always, to be able to rapidly navigate from one picture to the other, and quickly modify the z-index of any given image...that simply cannot work, it would utterly kill your processing speed.
How the heck is he doing this!?Blaise! Come to my rescue!
Charles wrote:Imagine using this technology for "touring" 3-D image compositions of planetary bodies, like Titan. The usefulness of this technology for scientific imaging is incredible. Astronomy, biology, ecology, you name it. I am so excited right now.
mawcc wrote:Look at Google Earth, I think it's similar.
bluvg wrote:How dare they show this to us and not offer a beta?!This is totally amazing!!
OK, I joined!
This response has just been overwhelming. Thank you all!
Very briefly, the multires issues raised by jsampsonPC and others basically revolve around the question of how an image can be represented in a way that allows the whole thing to be accessed at very low resolution, or a small part to be accessed at high resolution,
or anything in between. Also, each such access needs to involve limited server disk IO and limited processing on the client side. The ideas first used to implement this efficiently and elegantly were developed in part by my old thesis advisor, Ingrid Daubechies.
Do searches on "wavelets", "wavelet image compression", and "JPEG2000" to learn more. We don't use JPEG2000 (or, in fact, wavelets) in Photosynth, although when Seadragon was an independent company we did use JPEG2000. The basic ideas are similar, though.
The Seadragon collection model generalizes these multiresolution concepts to collections of images (or other, non-image visual content, like text and vector maps), not just single images. Again, the design requirement was to allow massive collections of massive
images to be opened remotely without too much work either for the server or the client.
Astronomy and microscopy applications: YES. Among many others. We hope to get to a point where there's an open platform to develop on, so that these ideas and applications can be developed the right way by the people who really understand what should be made
and how.
Beta coming soon.
cheers
Blaise
jsampsonPC wrote:
Charles wrote:Microscopy is certainly an area where this could be highly useful. Consider also astronomy... Navigating galaxies and other celestial bodies will never be the same!
C
Astronomy. Hmm. That raises an interesting question which I don't believe was addressed in the interview. Of course this type of image-manipulation will work great with highly-diversified imagery. A building front, or St. Peters basilica.. You have so many unique aspects of those buildings, dare I say it is easy to line up the images.
Compare that to an evening sky, where every star is very much visually-ambiguous. Of course inspecting satellite closely will give us the higher value of diversity, but from a distance, wouldn't this be too much of a feat for the application to properly distinguish one star from another? I suppose this problem may even exist in Microscopy, too. Obviously viewing a colony of bacteria will look very much the same in many different locations.
10 years ago I used an application will build panoramas from images taken along a horizontal plane. Each image had to be highly unique in order for the software to line up the images, and sometimes we got fuzzy couplings - especialy in places where threes were dominant. Perhaps I'm allowing my old knowledge to govern my understanding of this new technology too much
Jonathan
PS
Was anybody able to convince Blaise that he needed a c9 account?
Richard.Hein wrote: Besides, you just need to put up a frame of reference, like a grid, don't you?
Richard.Hein wrote:How could it be more difficult to line up stars, which are clear points of light, especially with high-res telescope photographs? If they can do it with window sill corners (which seems a lot more visually ambigous to me), why not stars? Besides, you just need to put up a frame of reference, like a grid, don't you?
Charles wrote:jsampsonPC,
you are missing an important ingredient in both your understanding of astronomical digital photos and PhotoSynth...
As of today, there are millions of photos in various databses of astronomical objects. So, let's say there are 10,000 images of Galaxy X taken at different times from different angles from different telescopes or the same telescope (like Hubble). This is very much in the realm of PhotoSynth compositional requirements: they are digital images of the same object taken from different points of view at different times. Consider the Basilica example in the video.
C
Charles wrote:Mapping the sky is something that is being done today, with images. There will of course be a default perspective, the Earth...
jsampsonPC wrote:To be overly-simple, a cluster of white dots doesn't really represent any unique characteristics which we, or computers, can immediately point to and say "That's over there". Before you mention constellations, keep in mind that this only works when all of the required points (or most of them) are viewed".
Richard.Hein wrote:If you take an image from the Earth or a satellite, it has a cone of view. If you know the time and position of the camera on the Earth which takes the photo, you can calculate exactly which star is which.
Could you see stars from the side, as if the cone of view turned 90 degrees off one axis? If you know the distance between stars in the direction of the cone's origin to the center of the cone's base. That is measurable. If you do that you can reconstruct an image from the side. For the back you could do the same thing, but you don't really know what is in the back the moon say, without a photo from the dark side, but you could fill in the blanks for stars and such, and rotating bodies obviously can be expolated, so the only limitation is distance and "hidden" bodies that are never in the cone of view of any of our current photos, which is of course a lot of stuff in space.
Cool, Virtual Time Travel!jsampsonPC wrote:I had thought about something slightly different. What about all the old photos I find of my great grandparents standing by some barbershop door, or down at their local foodmart.
What would the results be if I put one of those inIt would be neat to see locations from 80-100 years ago. Imagine how many photographs of Disney there are, spanning decades.
This could be used as a historical learning device, too.
Richard.Hein wrote:Cool, Virtual Time Travel!
jsampsonPC wrote:I had thought about something slightly different. What about all the old photos I find of my great grandparents standing by some barbershop door, or down at their local foodmart.
What would the results be if I put one of those inIt would be neat to see locations from 80-100 years ago. Imagine how many photographs of Disney there are, spanning decades.
This could be used as a historical learning device, too.
The technology is amazing but I would like to see some proof that this works in real-life circumstances too (I have not seen this c9 video yet, sorry if blaise explained it all )
My questions:
- How does photosynth deal with different light conditons? Not only day and night (which is fine, you can have two 'worlds' one is day, other is night) but 11am and 5pm.. Clouds and sunshine, rain and snow, wear and tear. Paintjob and graffitis.
- How much work is put on the user to group the right photos together?
- What cluster do you think would make the most sense for a global database of 'photosynth space'? Country? City? District? etc? You are not going to match all pictures in the world against all.. are you?
- As a research company, when are you publishing the algorithms used to match pictures together?
- Is the demo WPF based? (I see DirectX mentioned with seadragon so maybe not. Whe are you not yet on the WPF bandwagon? )
that's all for now. Blaise, I couldn't find a biography of you, although I saw your work related to the Gutenberg Bible.. So cool what technology enables. Probably the Princeton Physics and Applied Math background helps Were your born in the US or in Europe?
Gabor
RGabo wrote:The technology is amazing but I would like to see some proof that this works in real-life circumstances too (I have not seen this c9 video yet, sorry if blaise explained it all
)
My questions:
- How does photosynth deal with different light conditons? Not only day and night (which is fine, you can have two 'worlds' one is day, other is night) but 11am and 5pm.. Clouds and sunshine, rain and snow, wear and tear. Paintjob and graffitis.
- How much work is put on the user to group the right photos together?
- What cluster do you think would make the most sense for a global database of 'photosynth space'? Country? City? District? etc? You are not going to match all pictures in the world against all.. are you?
- As a research company, when are you publishing the algorithms used to match pictures together?
- Is the demo WPF based? (I see DirectX mentioned with seadragon so maybe not. Whe are you not yet on the WPF bandwagon?)
Gabor
jdughi wrote:Clusters - yes - all of the above. That's the goal. Of course - there are ways to cheat and not just match points - many photos have text that can be extracted, GPS coordinates baked into EXIF can be used, etc.
jsampsonPC wrote: IDEA!
I was in the restroom, looking in the mirror and washing my hands when all of a sudden I got an idea regarding this technology. It's simple:
"Why just photos?"
Imagine navigating around your photogallery, err, walking around St. Peters Basilica via a series of images all found within the "StPeterBasilica" directory. But guess what, there aren't just JPEGS in there sucka, there are WMV, MOV, and plenty of other video formats - and those videos have A start-frame!
Suppose the FIRST frame of a movie was also surveyed when when you're walking around. You look to the front of the Basilica, click a tiny camera icon, and whammo...you're linked up to a 20 second movie of a child feeding birds infront of the Basilica, with the backdrop matching the relevant photography.
What do you guys think?
ElucidWeb wrote:Did someone almost wreck outside at 1:54 or was that just my imagination?
Cool. Incredible.
Anyway, recorded audio quality is too bad.
plz take more clear voice later. Charles.
IDEAS!!
Time travel is going to be the big one. Time Traveler in a geowall based holodeck.. (dual polarized usb projector for xbox360?)
anyhow, I'm curious if there are more apps from this team on the drawing boards? VideoSynth being an obvious one. Pixel tracking from video proccessing yeilds point cloud results similar to PhotoSynth. Ya know.
PhotoSynth would make the idea front end for a new media quickening.
In my gallery simulator project, we are striving to provide the user with a rich newmedia experience inside the 3d model. Videos that can be watched, paintings that have amazing zoom and artist metadata, an authentic sound environment, etc.PhotoSynth is a step in that direction eh?
I would LOVE to use it sooner then later. I'll definitly use it for an exhibition in October. Is there a special artist beta I can use today?!? Please release this as soon as possible.
ZippyV wrote:Is this technology also able to create a 3d version from pictures of a person? I would guess this is more difficult since it's practically impossible for a person to keep the same position between 2 pictures at a different angle.
Great interview. Nice to see the research curriculum at MS. Do we have sight of a Picasa-style media browser? Do I need a farm to see the same results? Where is WPF?
What an exciting place to work.
zokbari wrote:Cool. Incredible.
Anyway, recorded audio quality is too bad.
plz take more clear voice later. Charles.
Hi,
One could believe that PhotoSynth has been conceived to work in association with Popims Animator and Popims Screens technology.
Visit www.popims.com and you will understand how it will work :
blaise wrote:OK, I joined!
This response has just been overwhelming. Thank you all!
...
cheers
Blaise
When I saw this video (and later checked out the website), this whole....thing (idea, program, heck even icon) screamed Apple! It seems like something Apple would have done. It makes me a little sad MSFT (er, Seadragon) did it, and not Apple. But I am happy nonetheless (and very much impressed) that Microsoft did do something like this.
It is truely stunning. But what I would really like to see is for this to open up somehow/get ported to other platforms. I don't care if it goes opensource, but I would just LOVE to use this instead of iPhoto (yet still have it integrate with all the other apps).
Obviously, I am a Mac user, but that doesn't mean I have to hate everything else. It really feels like something I would see on a Mac. Blaise, if you're listening, I hope that if it can't be brought over to the good side that it at least gets the integration it deserves on it's own platform. I think that's one thing MS could really learn from Apple (well, NeXT actually) and that's the fact apps should work together, and it shouldn't be hard for one thing to work with another.
Comment removed at user's request.
Suggestion: If a bunch of photos are stitched together in 3D space, some of which contain GPS info in the EXIF metadata, it should be pretty easy to calculate GPS data for the untagged photos in the mesh and assign them appropriately.
I don't have a GPS and have been searching for an easy way to blast that info into my photos.
Suddenly, taking a lot of essentially free pictures (no film) of a structure with a digital camera makes a lot of sense. After PhotoSynth has been released and is easy to use, it will revolutionize real estate because it will be very easy to show a 3D visit of a house once a lot of pictures are taken. It will also increase sales of digital camera memory modules (one module for a set of pictures of a structure). It will also affect flickr partially. Flickr mostly has two kinds of pictures, people and structures, structures part of the pictures will require something like this because now (after PhotoSynth), a set of pictures together are more important than an individual picture.
Hi,
Do you know when this is available to download in the UK?........
Read full article here.Penn State researchers have "taught" computers how to interpret images using a vocabulary of up to 330 English words, so that a computer can describe a photograph of two polo players, for instance, as "sport," "people," "horse," "polo."