jonudell

Derik Stenerson on the past, present, and future of the iCalendar specification

JonUdell — Mon, 13 Oct 2008 13:30:00 GMT

Derik Stenerson first came to Microsoft on an internship as a Test Engineer on Microsoft Mail. After graduating, he joined Microsoft full time in the email group and worked in various roles on email and scheduling products, including Schedule+ and Exchange. His passion for calendaring and scheduling led to work on the iCalendar standard (IETF RFC 2445) and later on a hosted self-service scheduling solution for small businesses. For the past few years Stenerson has been exercising his other passion for user centered design while building features for Microsoft Dynamics CRM.

Derik Stenerson

Next month marks the tenth anniversary of RFC 2445. To celebrate the occasion, Derik joins Jon Udell on Interviews with Innovators to discuss the past, present, and future of the venerable iCalendar specification.

Scott Prevost explains Powerset's hybrid approach to semantic search

JonUdell — Fri, 26 Sep 2008 09:12:00 GMT

Scott Prevost is General Manager and Director of Product for Powerset, the company whose semantic search engine was recently acquired by Microsoft. In this interview he describes the history of Powerset's natural language engine, and explains how it works as part of a hybrid approach to indexing, retrieval, and ranking.

Scott will expand on these topics in his keynote address at Web 3.0 in October.

Scott Prevost

JU: The notion of search enhanced by natural language understanding has a long history. I was just reading Danny Sullivan's rant about how he's been hearing about this for years, but it's never amounted to anything.

Of course, people are all over the map on this topic, but nonetheless you guys are doing certain demonstrable things, and working on other things. So I'd like to find out more about how the technology -- which was acquired from Xerox, where it had been worked on for a long time -- actually works. What you mean by natural language understanding, how you're applying the technology, and where this is going.

SP: Well, there are a lot of questions tucked in there, but maybe we can start with what we licensed from PARC, what was formerly Xerox PARC. They had been working for 30 years in a linguistic framework called LFG -- lexical functional grammar -- and they built a very robust parser. It's probably parsed more sentences than any other parser in the world.

What it allows us to do is take apart every document that we index, sentence by sentence, uncover its linguistic structure, and then translate that into a semantic representation we can encode in our index.

JU: Can you confirm or deny something that Danny Sullivan reported, which is that it takes on the order of two months to index Wikipedia one time using this method?

SP: [laughs] That's a very, very old number. It all depends on the number of machines, but we do it now on the order of a couple of days.

JU: And it scales linearly?

SP: Yes. And in fact we're working really hard to bring those numbers down. We have a very small data center right now. We're looking at what it takes to stand up a 2 billion document index, and it's absolutely attainable.

I think Danny Sullivan realized, when he wrote another article on the day we launched, that we're doing something different. He called us an understanding engine. It's not the case that we're just applying linguistic technology at runtime, by parsing the query and then trying to use the same old kind of keyword index for retrieval. We're actually doing the heavy lifting at index time.

We're actually reading each sentence in the corpus, pulling out semantic representations, indexing those semantic representations, and then at query time we try to match the meaning of the query to the meaning of what's in the document. That allows us to both increase precision and improve recall.

JU: When you say semantic representation, what it means -- or anyway what's evident in the current version -- is subject/verb/object triples, basically. That seems to be how things are organized.

SP: That's one small part of what the engine does. It's the part we've exposed in the user interface in a very direct way. But actually those are only three of several dozen semantic roles that we uncover at index time, and all of those roles go into selecting documents, and snippets of documents, when we present the organic results.

JU: Really? So even though the patterns aren't exposed in the advanced query interface, they're still used?

SP: That's right, they're still used.

JU: What would be an example of one of those other patterns, and how it's applied?

SP: So, you ask a question like: "When did Hurricane Katrina strike?" The 'when' is a certain kind of semantic role that we've indexed, separately from the subject, verb, and object. There are a number of other roles like that: location, time, other types of relationships.

JU: I saw a private demo, about a year ago, in which one of the most striking examples was something like: "Companies acquired by IBM between 1996 and 2003". At that point, I think the light bulb goes on in people's heads about what this could really be.

That class of query isn't exposed yet, but it's an example of what's possible, right?

SP: Absolutely. That's exactly the direction we're moving in. Initially most of the work we've done has been on the index side. Now we're starting to catch up on the query side, which allows us to complete the loop and do queries like that.

JU: The other piece that's visible on the website, in addition to the Wikipedia stuff, is the Freebase material that you've recently integated. That's an interesting case because there you can pull semantics directly from Freebase. So this becomes more of a query-time interface to something which is already structured and understandable.

SP: Yeah, that's right. Freebase is kind of like Wikipedia, except it's all structured data. Unlike with our core technology, which turns unstructured data into structured data, with Freebase we just go directly to the structured data. But it uses the same linguistic technology on the front end to parse the query, which then gets mapped into a Freebase database call.

But by using linguistic technology to parse the query, we're able to match very flexible ways of saying things. We don't have to imagine every possible way someone might ask for a particular piece of information. The linguistic engine takes care of a lot of that for us.

JU: That's why I can type in something like "Barack Obama's book" and get back the answer Dreams From My Father directly from Freebase.

So, what was the intent of including Freebase along with Wikipedia. What are you trying to show there?

SP: That the linguistic technology can be used with both structured and unstructured data. Freebase just has a lot of really great information.

One of the things about a natural language front end is that it encourages people to ask questions and expect answers. With the Freebase database, it's pretty easy to provide direct answers right at the top of the search results page, which users find to be a nice experience.

Of course you have to be very high-precision, so we've tuned the Freebase stuff for precision rather than recall.

JU: Tell me about the natural language landscape: the variety of approaches that exist, the style that you're using, how that compares to others, how all this fits into the history of the technology.

SP: The technology goes back a long way, three decades or so.

JU: Longer, actually.

SP: Yeah, really since the beginnings of AI people have been trying to use computers to understand and generate language. There have been a number of different approaches: purely symbolic approaches, statistical approaches. We really use a hybrid.

The Xerox technology uses a particular grammatical formalism, and we do use symbolic approaches to our semantic rules. But we also then put these semantic features into our index, and use machine learning and statistical approaches to retrieve and rank results.

It really is a combination. We try not to be religious about these things, but just use best of breed, and choose the right tools for the jobs we're doing.

JU: One of the things that Peter Norvig at Google is always saying is that the real secret to their success is vast quantities of data, and that in the end you don't really need AI, you just need lots and lots of data and the ability to crunch through it.

I assume you would argue that the natural language techniques are also helpful, and that as the quantity of data in your possession grows, the power that it brings to the table will also grow.

SP: Yeah. One thing we try to do with the natural language technology is give a leg up to the statistical and machine learning approaches. If you look at a search engine that just uses keywords, the information you have about the page is pretty slim.

We're trying to capture more information about each page that we index, which enhances our ability to retrieve and rank. For example, it allows us to retrieve documents where there are no keyword matches, but there's a good meaning match.

JU: For example?

SP: So, consider a query like: "What politicians were killed by disease?" Powerset will retrieve documents that don't include the words 'disease' or 'politician' or 'kill', but that are about particular politicians who died from particular diseases.

JU: Is the process of mapping generic terms to specific instances a hybrid of human editorial effort and statistical techniques?

SP: Yeah. We use things like WordNet, which is a giant dictionary or thesaurus of the English language that shows how various word senses relate to each other. We use that with some editing on our own. We also use machine learning techniques to figure out some word relationships, and which are most helpful in retrieval and ranking. So it really is a combination.

JU: When did you start this work?

SP: The company was founded three years ago, and I joined two years ago. But of course the work at PARC goes back 30 years.

JU: You obviously have an academic background in this field.

SP: Yeah, I have a Ph.D. in computation linguistics, as do probably about twenty other people at Powerset.

JU: What's your take on how this engine will start to surface through the various Microsoft online properties?

SP: The two areas where we can make a big impact are, first of all, improving core relevance, which is an absolute must for every search engine. And then also the user experience. Some of the technology -- and you start to see it in the Wikipedia search engine that we put out -- some of it really allows us to do different things in the presentation of these results. Thing that can save the user time, by getting the answer right on the search results page.

Our goal is to continue to work on improving relevance, and we've shown that by using these semantic features we can drive large relevance improvements, but there's still a lot of work to be done there.

JU: In that case, the improvements would be under the covers, the person using Live Search wouldn't know that you were contributing to the relevance of the result.

SP: That's correct. Another way it can happen is by creating a different quality of snippet or caption, things that highlight the parts that match the query instead of just bolding the keywords. Actually highlight the answer right there on the search results page, so you don't have to click through to determine if it's the right page.

JU: There's a related area called entity extraction, and there's been a lot of action there. For example there's a company called ClearForest, recently acquired by Reuters, which has put a lot of work into entity extraction. What's the story on that front?

SP: A lot of companies are working on this, we have our own in-house effort for name recognition and entity recognition, and this is of course really helpful as a kind of light semantic layer. But for us, it becomes deeper because we can start to relate all kinds of entities to one another, based on where we've seen them, and also with the help of things like Freebase.

To follow up on how you'll see the impact in things like Live Search, beyond the improvement in relevance and in the quality of snippet, I think you'll see features like related searches, other ways of presenting information similar to the Factz that are shown in our Wikipedia product, I think you'll see a lot more work on the instant answers, with a database that extends beyond Freebase.

Without committing to particular deliverables, these are the kinds of things I think you can expect to see. And you'll also continue to see growth on powerset.com, where we can be a bit more daring in terms of ways of presenting search results.

JU: Well thanks for your time. This has been interesting, and I'll be fascinated to see how things unfold over the next few years. I've got a feeling you'll have access to a pile of resources to work with...

SP: Yeah, we're really excited about it. As a startup, it's hard to build a full-scale web search engine. Having the resources available, and the really smart people at Live Search, is just a tremendous boost to us.

Kristin Tolle on biomedical initiatives at Microsoft Research

JonUdell — Thu, 18 Sep 2008 15:37:00 GMT

Kristin Tolle is the Senior Research Program Manager for Biomedical Computing for External Research in Microsoft Research. Projects run the gamut, she says, from "bench to bedside". In this interview she discusses two major biomedical initiatives: Cell Phone as a Platform for Health Care, and Computational Challenges of Genome Wide Association Studies.

Kristin Tolle

JU: Give us a sense of the kinds of biomedical projects you're working on internally, as well as those you're working on with external partners. I spoke with George Hripcsak, one of the researchers awarded a grant under the Computational Challenges of Genome Wide Association Studies (GWAS) program, and I know there are others involved there and in other programs as well. I'm interested in what Microsoft brings to the table in terms of helping these folks out with their computational and data management challenges, and also what kinds of things Microsoft learns from these engagements.

KT: The different programs inside of External Research run the gamut from the devices and mobility space, for home health care and elder care, all the way to genome wide association studies. So, we fund projects all the way from bench to bedside.

Because we're a software company, we'll focus on the IT parts, and there's a reason for that. These are often the parts that don't get funded elsewhere, or only get funded sparsely. Our purpose for going into medical funding was to fill those gaps.

JU: And why do you think those gaps exist?

KT: I think it's a misperception, by a lot of the funding agencies, that either something doesn't fall into their area, or that it's not as important as the actual research being done.

The problem is -- and this is why we're funding this area -- you cannot do medical research without computing. You just can't.

JU: Of course not.

Areas that we've funded...well, the biggest RFP we ran this year was Cell Phone as a Platform for Healthcare, and that was 1.4 million dollars toward trying to reach rural and underserved communities with retro technologies like cellphones and televisions, because those are ubiquitous.

JU: Oh, absolutely. I've spoken to Joel Selanikio, who was recently awarded a MacArthur Grant to use handheld devices for field data collection in the third world. It's a huge opportunity, though as you say it's the sort of retro technology that doesn't make people's eyes light up in Silicon Valley, they just don't see the opportunity the same way.

KT: It's true that they don't. But interestingly we've got a lot of researchers in-house, whether we're talking about that situation or about genomics, who have a keen interest in working in these areas. So for example, we gave Fone+ devices to a couple of the people who were winners of that award. The Fone+, which was developed by Microsoft Research Asia, is a phone that sits in a cradle, it's got RGB out to a television set, and USB input ports for mouse, keyboard, etc. So basically it enables your phone to work like a PC.

Now the beauty of this is, if you hook that up to a microscopy device that can do instant visualization of blood cells, determine whether or not somebody has malaria, and display that on a television screen, you've now just set up a lab for doing microscopy anywhere in the world there's a TV and a cellphone.

Another example is something we did with Washington University in St. Louis. They're developing low-cost ultrasound probes. Same thing. They're USB out, and designed to work with laptops, but now with the Fone+ you can plug it into this little cradle and now you've got an ultrasound anywhere in the world where there's power, a TV set, and a cellphone. You can even control the ultrasound device from the phone itself, it's just an amazing technology.

So that's an example where Microsoft Research has developed a technology that facilitates providing health care to rural communities. Although it wasn't initially designed for that, it was initially designed for education. But I took it and sort of twisted it..

[laughs]

...and said, hey, that'd be really good for the cellphone as a platform for healthcare project. I got them to give me a bunch of phones and cradles, and started sending them out to the researchers who had won awards for the RFP I ran this year.

JU: What kinds of things have you heard back?

KT: We've only funded them six months ago, so we won't see results probably until sometime next year. But I've actually seen a demo with the Fone+ and the ultrasound unit already working, so that was impressive. Washington U. is ahead of the game, I'd say.

JU: Are the folks you funded to do these things expected to bring technical chops to the table, in order to extend these devices? Are you working with them to provide support?

KT: Yes, we expect them to bring something to the table. And the ones who win the awards have superior technology. We had 145 people submit to the cellphone as a platform for healthcare. We'd originally planned to fund a million dollars, so that's about 10 projects, but we had to extend it to 1.4 million because we wanted to get to 10% acceptance rate. But even that, for us, is generally fairly low. Usually our acceptance rates are much higher. But we were just bombarded by people trying to come up with solutions for this space.

It was disappointing to only be able to do 14 proposals because when I looked back through them, I'd say 85 were fundable and on the bubble. Isn't that terrible? You wish you could do more.

So, I know you've talked to George about the genome wide association study, but I'd like to head in that direction in terms of some other things we bring to the table.

When I went looking inside MSR for collaborators, what I learned was that there's a plethora of them. It's kind of surprising we hadn't been funding this area before, and it's no surprise to me now that it's become a strong pillar of funding for our organization. In fact we've trimmed a lot of other programs and will be focusing a lot on the healthcare space this time around.

When I went hunting for collaborators I had no trouble finding them, even though I was new to the team, and that was because people consider healthcare the killer application for what they're working on.

But we also had a rich group -- you know, we have a couple of MD/PhDs working here, Eric Horvitz and David Heckerman -- and David does a lot of work in the development of vaccines for HIV and malaria. But he's branching out now into this GWAS area. So he's been looking at Lou Gehrig's disease...

JU: We should define the term GWAS, for people who aren't familiar.

KT: Sure. Genome wide association studies look across the genome to find if there are particular genes implicated in disease. That's one side of it. Another side is looking across the genome to check for reactions to different pharmaceutical agents.

In simplest terms, these studies are what will deliver on being able to provide personalized medicine for all of us in the future.

JU: Exactly, because it's a scan of an individual's complete genome, looking for markers and correlations.

KT: Absolutely right, that's correct. And I believe it will really deliver on personalized medicine for the masses.

And the thing is, it's happening already. We've got 23andMe popping up, Navigenics, people are going to start using their genomic information to make informed decisions about the type of healthcare they receive. They'll be taking that to their doctors and assuming they'll be able to work with it.

So we need to push the IT component of this down so that doctors have access to the information and know how to utilize it. Right now, that's the clinical gap between the research that's taking place in this area and the doctors who are performing the services needed.

JU: So in this case you funded about a half dozen individuals to look into different aspects of this GWAS research...

KT: ... yeah, very different...

JU: Right. So what do you hope will result from it?

KT: This was a new area for us, for Microsoft Research. We'd been dabbling in genomics for a while, but here we wanted to cast a wide net, find out what was going on out there, and find out if there were potential collaborations we could take from there.

When you find people whose work you can help facilitate, you form strategic collaborations with them to take that research to the next level.

Of course we bring a lot of resources to bear on this space. For example, the Microsoft Computational Biology Tools that we've published out on CodePlex, open source.

The other thing we bring to bear is a deep knowledge of machine learning and knowledge representation. And a number of researchers who've been working in general fields, but are now turning their attention to genomics.

A couple of new examples: John Winn, and also Christopher Bishop who literally wrote the book on machine learning and pattern recognition.

JU: This is a pattern I'm seeing often in these external partnerships. In all areas of science, as you say, scientists are necessarily becoming computational in the work they do, it's just the nature of the beast. But they don't necessarily have deep domain expertise in either algorithms or data manipulation and analysis. There are lots of folks at MSR who are deep in those areas, and who can effectively partner with these folks to move things forward.

KT: And it's not just that we have these underlying analysis and infrastructure technologies, we also have the human-computer interaction technologies to make that stuff usable for clinicians, or even the public themselves. So we've got people doing interesting work in how do you make something more understandable? How do you do machine translation across sex, age, status, education?

It's the same type of machine learning problem that you have with regard to going across language. You have to translate between languages, but you even have to translate within a language between different cultures, different demographics.

JU: What does Microsoft learn as a result of these collaborations, and what is Microsoft able to do with that?

KT: Well, the overall goal for External Research is to facilitate time to discovery, and to do so in a way that extends the arm of Microsoft Research. What we learn are which directions to move in. You know, we have publishing and tenure track promotion in Microsoft Research just as in academia. So if we can make our researchers more effective in reaching their goals to publish papers in Science and Nature, that's a fabulous thing. We've facilitated them and extended their reach.

There's also corporate responsibility here as well -- Microsoft, as a company, investing in areas that are important for the future. It's also important for us to keep abreast of the times, and the things taking place now. And finally, we learn things that we may incorporate into our products through tech transfer.

JU: It may be early days to talk about tech transfer from your biomedical projects, but I'd imagine one obvious outcome will be related to the kinds of devices that will be part of the HealthVault program, as sensors start to exist in people's homes, monitoring their vital signs, and transmitting them to the cloud.

KT: Absolutely, there's no doubt about it. The more that we invest in applications, and in sensors that can feed HealthVault, the richer their offering becomes.

The other thing is that we feel we're helping the public become more knowledgeable about their own healthcare. I think that's a common goal we share with the Health Solutions Group, Peter Neupert's organization.

They have other goals as well. So for instance, they have Amalga on the clinical side, and also a project targeted at researchers, trying to take people through the literature search for drug discovery. We're working in conjunction with that. One of the projects we funded under GWAS was a system to predict possible adverse drug reactions based on genome wide association studies.

Then the Columbia project -- George Hripcsak, whom you spoke with -- he's creating tools for researchers to integrate clinical information into the genetic analysis. Well, George's project is being built on top of Amalga. So there's a lot of synergy with the Health Solutions Group. And that's not unplanned. When I was starting out I met with Peter Neupert, back when he had eight people in his organization, and I interviewed him to find out what areas we should be investing in for healthcare. I'd also visited various schools and talked with people in their biomedical programs to find out what they were investing in as well. Then I tried to identify areas that would be relevant both to Microsoft Research and the Health Solutions Group. So, it's not just serendipity.

JU: So your own background is in bioinformatics?

KT: Biomedical computing. I had to form a multidisciplinary PhD committee because my school didn't have a program for this, though they do now, at the University of Arizona. So I had to form a multidisciplinary committee to get a PhD focused on machine learning for healthcare, with computational linguistics thrown in. It was tough, but it was worth it.

JU: Natural language processing was part of your focus as a student?

KT: Absolutely. Although the systems I developed were much more broadly utilized by different organizations. In fact, homeland security has some of the code I developed, which they use to scan for terrorist activities. It was initially developed for the National Library of Medicine to scan through unstructured text and identify keywords for indexers, and also to create small indices so that you could search faster and more accurately for publications in PubMed and CancerLit and other digital libraries. But you could see there were other implications. In fact it's also been used by the Department of Justice to make correlations among police reports.

So it's a generic technology, but my piece of it was targeted toward healthcare, and that's where my background and interests have always been.

JU: Are there other areas you'd like to discuss?

KT: I think we've covered the two major ones. I see us really investing long term in the area of devices, sensors, body sensor networks, ubiquitous and pervasive computing. That'll be a fundamental theme going forward, because it's been one of the more successful areas that we've made investments in.

But I also see us keeping a strong eye on the "omics" -- proteomics, genomics, metabolomics, you name it.

A third important area, and I don't know if it will be short term or long term, which is to address the other thing we talked about, and I don't have an RFP in this, but machine translation for people to be able to understand health care documents.

The average person cannot go out on Medline and read the literature on their disorder.

JU: It is amazing, though, how much context people can assemble for themselves under pressure of intense need.

KT: No doubt about it. But it would be better if we could create facilitating interfaces that would enable people to more readily understand and interpret that information. There's a lot of it out there, it's information overload really, and if we could make it a little easier for them, that would be very valuable.

JU: I wonder how much of that will be done by machine translation, and how much by crowdsourcing various experts at various levels. I think probably both will happen.

KT: Yeah.

You know, another important area -- and I was a bit disappointed when we ran our GWAS RFP that we didn't get anything concrete in this area -- was data visualization for genome wide association studies.

I think that's because it's such a hard problem. These studies are computationally challenging as it is, there's a lot of data that gets generated. Then to visualize it, now you're adding another level of computational complexity such that it's already not realtime just looking at the data, then how do you take it to that next level of visualization? That's going to be an important emerging area going forward.

So for instance, we've been talking with the folks at Oxford about getting a Surface there for collaborative visualization of cancer pathology.

JU: Not just in this area, but in general, we are so underserved by our ability to make sense of large complex data.

KT: Yeah, and we have these cool technologies. I think the WorldWide Telescope could be redeployed in many environments, and I think healthcare is one of those killer applications. We were talking with the National Cancer Institute, and one of the things they'd like to do is take a slice out of the liver while the patient is still on the table and be able to zoom in and zoom out -- it's the same technology.

JU: It's a similar kind of thing. To the extent that we can, in different fields, standardize on data formats and define multidimensional data spaces, we can indeed have browsers and viewers for those spaces. What the Telescope does in its domain is create a browser for a web of astronomy data. So yes, we need to have browsers for webs of genome data, and all kinds of scientific data.

KT: We had a recent paper by Bongshin Lee, she's done a distance encoding tree -- she calls it Detective -- and it's a scalable visualization tool for mapping multiple traits onto evolutionary trees. So we're trying to tackle it inside Microsoft Research, but I was hoping to see more people outside MSR show interest so we could start forming interesting collaborations in that area.

JU: Well, this has been a lot of fun. I hope to follow up on some of those Fone+ applications, that sounds really inspiring.

KT: Yeah, that's the reason I've gone into this area. It is inspiring. There's not only corporate responsibility, there's personal responsibility for me as well, and that's why I like working in this particular space. It's genuinely gratifying to be able to make a difference in an area that, no question about it, is beneficial to society.

JU: Well you've landed in the perfect spot to do that, and it sounds like you're having a blast.

KT: Yes, I am. Well, thanks very much.

JU: Thanks Kristin.

Roger Barga on Trident, a workbench for scientific workflow

JonUdell — Thu, 28 Aug 2008 17:41:00 GMT

Roger Barga, a principal architect with Microsoft's Technical Computing Initiative, is leading the development of Trident, a "workflow workbench" for science. In its first incarnation, the tool will enable oceanographers to automate the management and analysis of vast quantities of data produced by the Neptune sensor array. But as Roger explains in this interview, it's not just about oceanography. Every science is becoming data-intensive. Trident's graphical workflow authoring, reusable data transforms, and support for provenance -- the ability to reliably track and reproduce all the analytic steps leading to a scientific result -- is being used by astronomers too, and is expected to find its way into many other disciplines as well.

Roger Barga

JU: We're here to talk about the Trident, the scientific workflow workbench for oceanography. Give us the 50,000-foot overview, then we'll zoom in.

RB: Scientists are increasingly dealing with large volumes of data coming from disparate sources. The process used to be manageable. You'd get post-docs to convert the raw data from the instruments into readable formats, there was a manual workflow to process the data into useful data products.

JU: Those were the good old days. Or maybe not so good.

RB: Right. Because the time to get from raw data to those useful products was often measured in weeks or months. But now our ability to capture data has outpaced our ability to process and visualize it. And its rising exponentially with the rapid deployment of cheap sensors.

The oceanographic project we're working on, Neptune, is just one example of this. Astronomy, and all other sciences, are experiencing the same trend.

JU: Neptune is a University of Washington oceanographic project ...

RB: ... it's actually an NSF project. The proper name is Ocean Observatories Initiative, and it's being funded for several hundred million dollars. The University of Washington is one of the partners. Monterey Bay Aquarium Research Institute and a number of coastal observatories as well are involved.

JU: So fiberoptic cables are being laid, and lots of oceanographic data will be pouring in.

RB: Exactly. It's transformed oceanography from a data-poor discipline to a data-rich one. They're going to be able to monitor the oceans 24x7 over long periods of time. So the kinds of processes they can study were never within reach before. They could collect data when there was an episodic event, or when they could get funding. Now they'll be collecting permanently.

JU: What's the scope of the sensor network?

RB: They're laying the trench in Monterey to test and deploy the sensors. NSF is reviewing the larger program, and getting ready to fund the Neptune array which will be off the coast of Washington and Oregon. The Canadian version of the Neptune array is up and running and collecting data, but the software infrastructure is still being built as we speak.

JU: What quantities of data is the Canadian array producing?

RB: Gigabytes per day. It can easily handle a couple of high-def video streams coming from the ocean floor.

JU: Really?

RB: Yes. And also in-situ devices that can sequence organisms. It really is like not only taking Internet and power out to the ocean, but also a USB bus that instruments can be plugged into.

JU: What are some of the experiments that become possible with this setup?

RB: For example, being able to understand sediment flows across the ocean floor, how temperature and salinity change, how fresh water flows in from rivers, what kind of life exists at those margins. And understanding that interesting narrow band where life thrives in the ocean. Too high up and the tides affect it, too low and there's not enough light. But really, there are a myriad of things like that.

JU: So an experiment, in this data-intensive new world, involves formulating a hypothesis, looking for patterns in previously-collected data, and then seeing whether data collected in the future supports the hypothesis.

That means you not only need to run an analysis on data, but that you have to be able to repeat that analysis on an evolving body of data. Hence the need for the workflow automation that you're providing in the workbench.

RB: Yes. Another aspect is the need to calibrate and tune the models. If they can do that based on long-term monitoring, it'll remove a lot of the uncertainty in our understanding of the oceans. Versus now, where the data are so sparse that it's hard to validate the model.

JU: I guess also that as your understanding of the data and the models evolves, you might want to rethink what data you're capturing and how you're interpreting it. So, what is it that you've built with Trident, and how does it help you do those things?

RB: Jim Gray was the first person who had the vision of an oceanographer's workbench. His insight was that scientists really want to interact with visualizations of the ocean, but there was a huge gap between the raw data and those visualizations.

Managing information and managing data is one of Microsoft's core strengths. In External Research, we look for partnership opportunities where can bring our technology, learn from applying it to data-intensive stress tests that involve even more data than our commercial products currently handle, and figure out how to use or extend our technology to provide a solution.

Jim pointed out that workflow was one of the key missing ingredients. We looked at the in-house tools, and Windows Workflow was the engine of choice...

JU: ...although it didn't exist at the time Jim floated this idea, right?

RB: Well, yes, it was around in alpha and beta form internally. Jim knew I was doing some of my research using Windows Workflow. Of course he left the solution up to us, but he accurately identified workflow as being a way that the scientist could not only manage the data transformations that were needed, but also create a library of solutions that could be shared and reused.

If you look at how Microsoft works as a company, we build platforms and then we expect ISVs to come in and bridge the gap between the platforms and the user communities. That's the role our group has played. We're looking at the requirements of the scientists, we're looking at the platform Microsoft provides, and we're building on that platform to provide a custom solution to the scientists that will not only accelerate their work, but change how they do science -- enable them to ask and answer questions they couldn't before.

We partnered initially with the University of Washington and Monterey Bay Aquarium Research Institute, or MBARI. They're already gathering data from sensors, so they could describe the spectrum of data we'd have to ingest into our workflows. The University of Washington has a visualization tool called COVE, which scientists are adopting as the preferred way to look at the ocean floor. You can think of it as Virtual Earth for the ocean. If there's bathymetry data, you can pull it in and se the ocean floor.

JU: What kinds of data transformations are needed to get from the sensor outputs to COVE's inputs?

RB: There are probably about two dozen kinds of data sources we need to be able to ingest, based on the instruments and the types of data they put out. Typically it's streaming data in NetCDF format, or some other common format. So the first step is to recognize what kind of data format an instrument or model is kicking out, and transform it into an internal structure that our tool can use.

JU: But the workflow engine is abstracted from the instrumentation data formats and from the visualization tools, right? It's a mechanism for reproducibly running transformations, and managing that pipeline.

RB: Right. But let's start with how we interacted with the scientists. Jim Gray would ask scientists: "What are the top 20 questions you want to ask, and queries you want to run?" From that, he'd get an understanding of how they viewed the data, and what kind of processing was required.

We took the same approach, and asked the scientists which top 20 workflows they perform and which top 20 visualizations they like to see. Then we went through them from top to bottom, talking about the transforms and data integration that were required. We wound up with a set of two dozen transformations that were common across all of these workflows. That became the library of activities -- reusable chunks of code -- that the scientists could call upon to author not only these 20 workflows, but the next 20.

JU: Can you give a couple of examples?

RB: Sure. Regridding. You have two data sets, one's from a model and the other's from a set of deployed sensors out in the ocean. They're on different grid coordinate systems and you need to be able to bring those two together. That may require some interpolation, you might need to drop or add data points, transform coordinates, join data sets.

JU: There might be a temporal variant of the spatial gridding as well, to align different time scales?

RB: Right. Some instruments are getting things every second, some are getting them every 15 minutes. You can ask the user: "Do you want interpolation to take place? Do you want the system to match up the points?" Based on these inputs, the correct workflow gets configured and they see the resulting visualization for the region of ocean they're interested in.

JU: It sounds like some of these primitives will wind up being fairly general, not just specific to oceanography.

RB: Indeed they are. We're producing a version of Trident for oceanography, but many of these activities could be useful for other sciences as well. People in earth sciences, for example, are also using NetCDF and many of the same operations.

We expect that by building a tool which is extensible, and agnostic in terms of the science it supports, you can imagine it being used, for example, to understand the interaction between oceans and warm air currents.

JU: What does the Trident user see and do?

RB: We realized that the authoring experience for scientific workflow is very different from, say, business workflow. In business, you'd have your accountant write your expense report workflow. They'd lock it down, they'd deploy it, everybody would use it from then on, and nobody would touch it until it came back for bug fixes or enhancements.

What we found with scientists is that they want to borrow somebody's workflow that does what they want, or close to it, load that workflow, and then start authoring from that point on.

So we implemented that in Trident. You can search for workflows by purpose, or by the inputs they process. You click on one, and load it into a visual browser because while the oceanographers understand the workflows, they don't want to see C# or Java, they want to see something visual -- boxes that represent the transformations they want to apply.

JU: We've mentioned the Windows Workflow Foundation. For folks who aren't familiar with that system, how would you characterize it? How is it like and unlike a script execution engine?

RB: What's unique about workflow, versus scripting, is that with workflow you tease apart the notion of a schedule, which is the sequence of actions you'd like to have performed. If you were to look inside of each of those steps, you'd see code similar to what you'd find in a script. But on top of the sequence of steps you have an orchestration engine. When you pass this workflow -- this sequence of steps -- over to the orchestration engine, it runs the code inside each of the boxes, but as each one completes, control passes to the orchestration engine.

So we have an abstraction layer, we've opened up the opportunity for reuse, the steps or activities become building blocks. In addition, the orchestration engine can monitor the execution of the workflow, or change the way it executes -- for example, by running blocks in parallel on a multicore machine.

JU: What struck me about the Workflow Foundation was the way in which workflows can be very big or very small. As small as the sequence of interactions with a form on a web page, in which case the orchestration engine can be embedded entirely in the code that's behind that web page.

Or it can be a very big thing. But in any case, since it's part of the .NET Framework, it can exist in a variety of places. It can run locally on a laptop, it can run on a server in the cloud. There's an interesting amount of flexibility in terms of how workflows can be deployed. An application could embed Trident, or Trident could be used as a service.

RB: That's right. That's the magic of it. Yes, it could be hosted in an environment that the scientist is already familiar with. Or for a big institution, you could post it up as a service. Anybody could access it from a browser. And that's part of our mantra here. If we provide this to the scientists, we have to make sure it works with the tools they're comfortable using. You should be able to point your Linux box running Firefox at this tool.

But to your other point, we're experimenting here with workflows that are resource-seeking. You could launch one, perhaps even on your cellphone, and that scheduling engine's going to look for systems that have resources for that workflow, tap into them, and give the user on the cellphone the impression it's running locally.

JU: You've mentioned that the workflow style encourages a level of modularity that you might not otherwise get. It also provides a level of monitoring, control, and auditing. The reason that's important goes back to the idea of reproducibility.

A friend of mine is an HPC expert, and one of his pet peeves is that when people look at HPC they tend to focus on how much raw horsepower can be thrown at a problem. His question is: "Who's worrying about reproducibility and correctness?" It's a really important question.

In your environment, as I understand it, one of the things that you get is the ability to capture and replay and analyze what happened in a workflow, and the ability to faithfully reproduce a sequence of steps. You talked about enabling things that scientists couldn't do before. It's not only that they couldn't analyze large quantities of data, but also that they couldn't automate their own methods, and be able to reflect on them in an automated way.

RB: Right. Even if we couldn't run a workflow faster, and even if we weren't processing a lot more data, one of our key features is support for provenance.

JU: Explain what you mean by provenance.

RB: Think about it in terms of art. For a given piece of art, we're able to establish through authorities that it's original, where it came from, and who's had their hands on it through its lifetime. Provenance for a workflow result is the same thing. Minimally we want to be able to establish trust in a result. If you think about how that happens, it often starts by considering who wrote the workflow. So with Trident you can click on a result and interrogate the history of the workflow: who wrote it, who reviewed it, who revised it, when it first entered the system.

We do versioning as well, so you can look at an old result and know that it was created by an old version of the workflow. And then have the ability to run the new version on the old dataset to see if it makes a difference.

We capture execution provenance so you know exactly how your result was created. We capture provenance on the workflows themselves so you know who created them, and who's touched them.

You might be thinking about creating a community, where you click on a workflow and can say: "OK, I trust that post-doc."

JU: I've been reflecting on what Microsoft brings to the world of science, in yours and in other collaborations that I've been talking to MSR folks about. One is clearly the special competence and expertise in data management and processing. Even for computationally-oriented scientists, that data expertise isn't necessarily a core competence.

Another is the software tradition of version control. Again, that hasn't been a traditional strength of scientists. So this looks like a fruitful partnership on both fronts.

RB: Agreed. It would be nice to get Catharine van Ingen, or perhaps Alex Szalay to chime in how how this is being used for astronomy. Because we're giving drops of this code to our e-science researchers for use in other areas.

JU: I'd love talk with Alex. I had a couple of in-depth conversations about the WorldWide Telescope, one with Curtis Wong and the other with Jonathan Fay, and we touched on the work Alex has done. He's using your stuff as well?

RB: Not him personally, but his project -- Pan-STARRS -- is. Catharine van Ingen and Yogesh Simmhan are co-architects of that system along with Alex. And they're bringing workflow to the table. It's becoming the way scientists upload their data into Pan-STARRS and get it back out, and Trident is the workflow engine for that.

You've probably also heard about other activities here in External Research. Perhaps the scholarly communiations aspect?

JU: Yep. I've talked to Pablo Fernicola about the Word add-in for authoring scientific papers in the National Library of Medicine XML format. And recently I got the overview of External Research from Tony Hey.

RB: When you think about Trident in the context of scholarly communication -- and to your point about the importance of provenance, we see eye to eye on that -- not only can we use these tools for e-science data management, but we're focusing on reproducible research. When Trident has finished running a workflow, we'll create an XML structure that describes how to call back into Trident to recreate the result. We're really keen on the idea that not only is it easier to do the science, and publish the science, but actually reproduce it. And that XML description should be able to be embedded in the published work.

That's really exciting. It's been talked about in the computational sciences, but never addressed end to end with a tool that's instrumented, that produces an XML standard the community can own which describes how the science was done, and that gets carried along with the publication, either physically or by reference, and we store this execution script in a database somewhere.

JU: It's a really big idea.

RB: It is, I think it could be transformational.

JU: I do too.

RB: Right now, reproducibility means that that you happen to know the person who did the experiment, or you happen to capture enough stuff in your lab notebook or on your whiteboard, then you have a chance of being able to do it again. But imagine being able to click any result, and automatically and transparently reproduce that result.

JU: In reality it won't necessarily be the case that you can punch a button and have everything replayed exactly. But having the documentation, at that level of detail, and in that form, would be an incredible asset.

RB: Agreed. The hope is that here in External Research, because we're building these tools not just in the context of one science project, but many, you can have community tools that bridge communities. We're talking to people in the earth sciences doing atmospheric studies, and their workflows and analyses are so similar to what the oceanographers are doing. But right now, since those two communities aren't talking or sharing tools, it's very difficult for one community to interact with the other.

JU: That's a really nice point. Well, thanks Roger!

RB: See you later.

Lewis Shepherd discusses the Institute for Advanced Technology in Governments

JonUdell — Thu, 14 Aug 2008 17:01:00 GMT

Before joining Microsoft's Institute for Advanced Technology in Governments, Lewis Shepherd spent four years at the Defense Intelligence Agency where he helped usher in a new era of collaboration.

In this interview, he discusses how the Institute's small team of seven is exploring the nooks and crannies of Microsoft's research efforts and technology portfolios, looking for ways to help governments meet the diverse set of enterprise challenges they face.

Lewis Shepherd

JU: Microsoft's Institute for Advanced Technology in Government is a mysterious new organization that hasn't been heard from much. Readers of magazines like Government Computer News may have seen some notices about it, and may have noted that former CIA Assistant Director Jim Simon is the founder, and that it's attracted some other folks who formerly worked in government roles -- Aris Pappas from CIA, you from the Defense Intelligence Agency.

But not much else is known. So, what's this all about?

LS: Well, I'd say a better word than mysterious would be quiet. And that's because we're new and small. The Institute was set up by Bill Gates and Craig Mundie in 2004. They decided that Microsoft should play a more strategic role in the eyes of government.

Actually, in our title, there's a final letter, S. It's the Institute for Advanced Technology in Governments, plural. We're not strictly focusing on the U.S. federal government, which the backgrounds of the people involved would imply. It's actually governments at all levels.

We've worked a bit with state and local governments recently. In the past year we've increased our headcount to seven, and the seventh was an interesting addition. Bob Hayes is a British citizen, he lives and works in Cambridge UK, and he has experience at all levels of UK government. He began as a beat cop -- a bobby -- and has worked in and around the national security community in the UK for his entire career.

JU: You folks have close ties to Microsoft Research, but don't consider yourselves to be formally a research unit. Or do you?

LS: Not formally, but we do work closely with MSR, along with product groups. Jim Simon reports directly to Craig Mundie, so we have visibility into the entirety of strategic and future-oriented work that Microsoft is doing. Not just strictly MSR, but also incubation, Live Labs, Office Labs, forward-thinking people in various product groups.

A lot of it is personal. We're just seven people, I joined just seven months ago. It's been a wonderful way to see inside this tiny little 90,000-employee company.

JU: Governments are specialized kinds of large enterprises, so there are all sorts of potential applications for Microsoft's enterprise-oriented technologies.

LS: That's a big part of it. The core mission is to assist our federal government, state governments, and eventually we hope local governments and NGOs, to focus on their enterprise-wide problems. Of which there are many. As bureaucratic organizations they're a lot like commercial organizations, but they have particular unique challenges that you probably don't really understand unless you've had the pleasure and frustration of working inside a large federal government organization. If you have, as we have, you really understand the pain, particularly within our national security community. The intelligence community, the Department of Defense, these are massive bureaucratic constellations of organizations.

Five out of the seven in our group have some background in that national security community. I didn't have a career in it. But coming from a different kind of public sector background, and then a Silicon Valley background, I spent four years at the Defense Intelligence Agency where, post-9/11, I tried to bring some new thinking to the intelligence community. Along with a lot of other people, we were able to do some of that.

Along the way, as I looked out at the different strategic partners that government has in the technology world, we certainly viewed Microsoft as important, just because we -- like most others -- were on a Windows and Office platform, and were using a lot of other Microsoft products, but to be honest, it was limited to that. We thought Microsoft was a product vendor.

One thing that began to change my mind was, as a government executive, I used to visit Microsoft annually in Redmond. The account team that supported our agency began to hear from me that we'd noticed Microsoft spending six and seven billion dollars a year in R&D. I started to wonder: Where's that money going? And how much of it was focused on assisting with government problems? The answer to that was, at least consciously, in the minds of MSR, none of it.

Yet here I was in the intelligence community, working with the DoD fighting this long war on terror, surrounded by some of the keenest early adopters in the world who were looking to push the the limits of technology. So I began to talk to Microsoft and found they had indeed set up this quiet group in 2004 to consult with government, both inside the intelligence community and elsewhere, on these kinds of enterprise problems, and to bring to bear some of the more interesting and promising fruits of Microsoft Research.

JU: There are a bunch of Microsoft technology initiatives that intersect with the interests of governments as large IT-supported enterprises: identity management, data management, systems management, service-oriented architecture, application development. That's all playing out in governments as in other enterprises, but I suspect that's not what you mean by advanced technologies in governments?

LS: Correct. Although many of those are very much of interest, and frankly, federal and state governments aren't always aware of the leading edge in commercial software technology. So one of our roles is to make them aware of the leading edge, and of best practices. We do that in a way that doesn't come across like a sales pitch, because they don't need to hear that. And if it makes sense to advise government leaders to innovate in ways that don't necessarily require Microsoft products, that's a plus for the Institute and for Microsoft's role in assisting governments. So we've done that several times.

Our sales guys understandably focus on what they can sell today, and in the next quarter. But often government needs to know how to make better use of what it already has, or how to use something that isn't a Microsoft product.

But here's a case study that's more along the lines of what we mainly focus on: Microsoft Surface. It's gotten a lot of buzz as you know, and is now being commercially rolled out in the entertainment space.

When Surface was still a research project, Jim Simon -- who loves to poke into the nooks and crannies of MSR and incubation projects -- saw it, and talked with the team, and realized they were mainly focused on it as a gaming platform.

He thought about that for a while, and said there were two additional markets, and he knew people in each.

One is the big-G gaming world of casinos. In venues like Vegas and Atlantic City, the entertainment experience involves a holistic view of customers, from the moment they show up at the hotel, day and night on the casino floor, at the shows, at restaurants. The touch-enabled UI really supports that scenario.

The other, of course, is the national security world, particularly DoD. It so happened than when I came on board and first learned about Surface, I had previously, at the DIA, had experience with touch tables, a different kind of technology -- a touchscreen on a pool-table-sized device -- that we were one of the first customers for. It was sold to us by a large defense contractor, and it did a great job for us in 2004 and 2005. But each device was $250,000.

JU: And what were you able to do with it?

LS: Defense planners typically stand around a big sheet of paper, or a map, trying to collaboratively plan out a day's or month's or year's campaign. Doing it that way, or on a sandbox, is the traditional way, and there hadn't really been any innovation.

With the touch table device, you could show a map on this horizontal surface, and data layers. Think about Virtual Earth or Google Earth, the ability to do that on a 6-foot by 9-foot table becomes very appealing.

JU: Was this commercial software on a custom device, or was it all custom?

LS: It was all custom. $250,000 a pop. We bought two of them, and put them on two different floors, for two different teams to use. After a while we realized it'd be nice if the teams could collaborate, but they weren't networked, so we had to pay our contractor another $100,000 to connect them.

When I first saw Microsoft Surface, and realized the entire thing ran on essentially a state-of-the-art PC, and that the APIs were going to be open enough for developers to put any kind of Windows software onto it, and that it would all be networked...it really opened up the possibilities.

And then when you realized that, because of the scale Microsoft operates on, the price would be $10,000 instead of $250,000, it just blew my mind.

So now that same defense contractor has become one of the first to develop on the Surface platform. They know their customer, they know the scenarios for defense, intelligence, homeland security, state and local police. And I think the Institute played a small but important role in opening the eyes of a lot of people to the kind of difference a Microsoft platform could bring to that environment.

JU: What are some other kinds of connections like that that you're making, or want to make?

LS: Well, I've been extremely interested in robotics, and the emergine large-scale appeal of the Microsoft Robotics Studio. You and I have chatted about this. The appeal isn't so much robotics, per se, but rather the back-end architecture that takes advantage of advances in concurrency and high-performance computing and distributed services.

You mentioned service-oriented architecture, and SOA has been a buzzword in government and other kinds of enterprise circles for a while now. Well there really are multiple services being developed and deployed in lots of different environments. The ability to orchestrate enormous numbers of those services is something you can do natively with the Robotics Studio, whether or not you intend to develop a robot.

JU: It's a fascinating outgrowth of that project, and the implications are only beginning to sink in. The software infrastructure is extraordinarily general-purpose.

LS: Right. Among overnment early adopters, the people we've seen take a good deal of interest have been in DARPA, and also in a new organization in the intelligence community called IARPA. Once you get the right people looking at this, they understand what's really behind it, and the power of it.

JU: You've also been known as a proponent of Web 2.0 methods, and were responsible for bringing Intellipedia to life.

LS: I was one of the people who did. It was a great team.

JU: What are the opportunities in that realm?

LS: I've watched with great interest the rise of Enterprise 2.0. I think most people credit Andrew McAfee with coining that term. He and I have spoken on a number of panels, and we've talked about how government organizations can nurture bottom-up development of advanced capabilities using things like blogs and wikis, and take advantage of the emerging power of social media, without the kinds of constraints you inevitably get in a government bureaucracy.

Not only do you have all the usual bureaucratic problems of large organizations, but there's also a hypersensitivity to security, and also -- within the civil service -- the disincentive to innovation that happens when people are career civil servants.

So how do you nurture grassroots adoption of these technologies? It's very personal, you have to find the right people. I was lucky to have an inside role in the intelligence community.

When you say Intellipedia, people may or may not know about it, but it's been a phenomenal success story in the intelligence community as a community. That word, community, was openly mocked for decades because the sixteen different agencies -- and particularly the big ones everybody knows about, CIA, NSA, DIA, NRO, the alphabet soup of them all -- really didn't collaborate that well, if at all.

The 9/11 and WMD commissions went into great detail about this. What I and others were able to do was to begin working in small ways on identifiable chunks of value that we could create for community-wide use on shared networks.

This work began in 2004. I don't think there was any great flash of inspiration in deciding to basically plagiarize Wikipedia on a secure network, as a large-scale network for socially-authored and socially-maintained intelligence that had been kept in stovepiped databases.

The first pilot was in 2004, and it opened as an enterprise system for the whole intelligence community in 2006. Intellipedia has been a big success. There was an initial period of hockey-stick growth. That's leveled off some now, and the challenge -- as in any enterprise -- will be to continue to evangelize the business practices of social networking, and the value they bring within a large diverse set of organizations.

JU: Of course this wasn't an example of advanced technology. Wikis and blogs are just the kudzu of the Internet. It was more an exercise in social engineering than a deployment of any new or advanced technology, and appropriately so.

From a Microsoft perspective, then, is it about applying advanced technologies from MSR in this environment? Is it about bringing some of that grassroots sensibility into the Microsoft platform?

LS: I think it's both. In the federal space, particularly intelligence where you have knowledge workers on steroids, we have an interesting mindset within the account teams. They're not only focusing on what can be done with SharePoint 2007, with Office 2007's XML capabilities. They're also seeking out bits of code being worked on in Live Labs, in Office Labs, and elsewhere. Popfly, for example, It's being heartily evangelized by Microsoft teams within the federal government, and it's gaining enormous receptivity.

JU: I've talked about this with John Montgomery, the Popfly lead.

LS: In fact there's so much interest, he's almost to the point of being overwhelmed.

What I see is a changing mindset about Microsoft, and the role it can play in government. It's not just about are we on a Windows platform. It's about what can I use, on my computer or mobile device, that'll enable me to do things I couldn't do before. If those are Microsoft things with a Windows label, that's great. If they're not, if they're cool, funky, web-centric things like Popfly, that's great too.

JU: Tell me if this fits into your charter. A big aspect of what I think of as Government 2.0 is the emerging availability of various sources of government data. There's a growing consensus that data will be made available, and that's happening, but in a way that reminds me of how things were, and mostly still are, on the scientific web. Yeah, there's the data, go grab the gzipped tarball and have fun with it.

As opposed to offering a service layer interposed between both applications and human being.

I see an interesting possible role for Microsoft, and I see it as extension of something that's happening in the relationship between MSR and the scientific community. I've recently been talking to a lot of people in Tony Hey's area. These folks are what I'd call informaticians, and they're working closely with scientists in various fields.

In every branch of science, now, the work revolves around the collection and analysis of previously unimaginable quantities of data. One of the things I'm seeing Microsoft consistently doing in its partnerships with scientists is to provide both infrastructure and consulting expertise, to help people wrap their arms around large datasets and make them useful in ways they wouldn't otherwise be.

I'm wondering if there isn't scope for something analogous in the government space, as these datasets begin to be made available, but not necessarily in ways that enable citizens to ask and answer meaningful questions, or relate the raw information to policy.

LS: You've hit on something that's really important, and yes, it's an interest of ours. It's very hard to do, but if you do it, the value is tremendous.

I'll give some examples of things that we're thinking about, and one that we're working on.

One thing we're thinking about, as a model, comes from one member our group I want to mention, George Spix, because he's such a great guy, a lot of people around Microsoft know George. He's the only guy in our group who's been with the company for a long time, before that he worked with Seymour Cray.

JU: Here's something about George you may not know. The Microsoft Conference Center recently hosted the annual space elevator conference, and George was the guy who gave the go-ahead for that.

LS: That's closely related to the example I was going to mention, which is the WorldWide Telescope. George also did some work on that, and if you think about it, it exemplifies what you were talking about.

JU: Absolutely. The WorldWide Telescope is the paradigmatic example of a service layer that's been interposed between a previously available but practically inacessible dataset and a set of interoperable applications, on the one hand, and ordinary people, on the other. You're right. It's the perfect prototype.

LS: It is. And as such, it also serves as the perfect educational device for people who are in a position of authority over other large stovepiped datasets. So we've been using WorldWide Telescope as a teaching element: "Here's the future of what your world could be." It's not only an extremely appealing app -- people fall in love with it, Robert Scoble was famously moved to tears by it -- but there's also, as you said, the paradigmatic simplicity, and obviousness, and utility, of opening up data. So we've been using that in a number of ways to stretch the mental boundaries that government officials have about their data, about the accessibility they currently offer, about what new technologies and web-scale computing could bring to their data, and about what that would do for them, and their intent to serve their customers, their users, their citizens.

It really is a mind-blowing way to get them thinking creatively about what could be done.

Another example: machine translation, and some of the hybrid translation approaches that Microsoft Research is pushing the boundaries on. Here we have real examples, already offered within the Windows Live constellation, that people don't really know about.

There's the translator bot that is a Live Messenger client, you can have simultaneous translation among a dozen languages in your instant messaging.

JU: I hadn't seen that myself.

LS: Well, there you go. And there's the live translator plugin for Internet Explorer, I just blogged about that last week. It enables you to surf foreign language websites, with simultaneous translation. It's really changed things for me. I have a lot of interest in Russia, so being able to surf Russian-language sites, with good-enough machine translation appearing right in the browser, it's phenomenal.

And as we show that as a service to be exploited within service architectures, that's something governments find really intriguing. It helps them think about how they could provide better access for diverse populations.

JU: I'm glad to hear that. I've been pushing for a while on this theme, and have recently concluded that we're kind of stuck on the question of access to the data. But that's only the first step. It's great that we're getting to the point where that first step will be taken, but there's so much more that can be shown and done.

LS: I'll give you another example. I've had that same lingering feeling of frustration about the web, and the billions of pages and documents I can theoretically access. But there's no sense-making.

One of the most exciting things I saw last year, even before joining Microsoft, was Photosynth and Seadragon. We're working on making these and related technologies, like Deep Zoom, available to government organizations that have access to very large archives of images which are just sitting there. Yes, theoretically you have access to them, but something like Photosynth enables you to make sense out of them.

JU: What kind of images are we talking about?

LS: Well, some I can talk about and some I can't. But we have been talking to some state governments about their access to the world of Flickr and online collections like that, from the standpoint of homeland security, and the ability of first responders to make sense of the visual environment of today's world, in an up-to-the-minute way, just based on the open source information that's available.

This is something Microsoft has helped a lot of public sector groups, like the Los Angeles fire and police departments, to be real leaders on.

JU: So a lot of documentation of planet Earth is being done in a grassroots, ad-hoc way, for example in the form of photos on Flickr that are tagged and even geolocated. And there might be a government interest in those collections as the most up-to-date record of what exists.

LS: That's right, and it eventually works in a circular way on the provision of government services back to people. If you think about a government bureaucrat, like a building inspector, who goes to a site, takes a photo or two, and certifies that the site is being worked on in a way that conforms to local or county regulations.

Well, the ability to do all that in realtime, with a camera-equipped cellphone, and do it in a secure way, with timestamping and geocoding...

Or think about that capability deployed in child welfare scenarios where there certainly aren't enough government personnel to visit all the domiciles where trouble is reported.

When you think about large volumes of data being transmitted in both directions -- from citizens to governments, and from governments to citizens -- it really opens up the world. We haven't figured out all the ways, but it's fascinating to think about the diverse set of enterprise challenges that governments face, and about the technologies we have in the nooks and crannies of Microsoft that might be able to help.

JU: It sounds like you're having fun snooping around finding them.

LS: I'm having a blast!

Maurice Franklin reflects on the 2008 Space Elevator Conference

JonUdell — Fri, 08 Aug 2008 14:24:00 GMT

Maurice Franklin is a 12-year Microsoft veteran whose career has focused on performance engineering and server scalability. He's also passionate about the concept of a space elevator, and recently organized and hosted a conference held on that topic at the Microsoft Conference Center.

In this interview he discusses reasons to build a space elevator, and describes how the concept, first proposed by Arthur C. Clarke, is evolving toward a practical implementation.

The transcript for this interview appears below. Audio is available at ITConversations.

In a related interview Ted Semon, author of the Space Elevator Blog, reflects on the conference and on the goals and status of the effort.

Maurice Franklin

JU: Maurice, most folks don't know even that there's a serious plan to build a space elevator, and I'm sure even fewer know that those most closely involved gathered this past week for a conference hosted at Microsoft. How did that happen?

MF: Well, I'd qualify the word "plan" ...

JU: Maybe we should call it an intention.

MF: That's a good word. Or aspiration. So, I got involved in the space elevator world in 2002 when I discovered, quite by accident, that there was a conference in Seattle. It was hosted by an entrepeneur, Michael Lane, and a scientist, Bradley Edwards.

Dr. Edwards is the father of the 21st-century concept of the space elevator. He'd heard it was impossible, and didn't believe that, so he got a NASA grant, and came up with something that made everybody say: "Well, that's not how we were thinking about it at all. That's a decades plan instead of a centuries plan."

JU: What was different?

MF: I'd read a 1997 NASA study, it was big science and big engineering. It relied upon a spacefaring technology to get to a space elevator. Those of us who are fans of the space elevator see it as a bootstrapping mechanism to get to that spacefaring technology.

So, for example, one of the prerequisites assumed in the 1997 study was the ability to move an asteroid into earth orbit. And then to put a manned carbon-nanotube-manufacturing station in orbit. Well, you can't do any of that unless you have a space elevator.

As much as anybody has, Dr. Edwards cracked the chicken-and-egg problem. His proposal requires on the order of four or five heavy launches. After that, it self-bootstraps -- if the materials come along, and a lot of other ifs, but that's a game-changing proposal.

The NASA study also required a massive elevator, because there were going to be maglev trains running at high speeds, and that just adds more and more weight.

His plan is much more modest, it only goes 200 kilometers per hour, it's a bit slow, but it's practical, you could reasonably get stuff up pretty quickly.

And he added in remote power beaming, so you don't have to carry fuel for those days of climbing up towards geosynchronous orbit.

In all respects, it's much more practical. If we could get past the technical hurdles, it's something in our lifetime, or at least those of us who are older hope so.

JU: So your role at Microsoft was then, in 2002, and is now...

MF: I'm a performance engineer. No connection with space-related activites at all, just a personal interest. Although that 2002 conference was invitation-only, I showed up. I'd read the NASA study, I'd read Brad's study, I had intelligent questions, and I was accepted, which was very cool.

JU: So how did Microsoft come to be the sponsor and host of the 2008 conference?

MF: One of the collaborators on this project is Dr. Bryan Laubscher. He's an astrophysicist, most recently with Los Alamos National Laboratory. About a year ago he was invited to the visiting speaker program at Microsoft Research. I went to hear him, gave him my card, and invited him to contact me if there was any way I could help.

About a month later he got in touch and said, "I'd like to make Seattle and Microsoft the center of the space elevator universe, so let's get a conference."

Well, Microsoft employees themselves can't usually sign up the Microsoft conference center, so it became my job to find a sponsor. Through various happenstances I got connected with George Spix, now with the Microsoft Institute for Advanced Technology in Governments, who I actually I knew from some work we'd done together in the past, and he said "Sure."

JU: It's interesting how these things play out.

MF: Yes, it is.

JU: So on the face of it, this is a project that has a lot to do with power engineering, and...

MF: ...materials technology...

JU: ...right, and civil engineering. But there are computational spinoffs and synergies too.

MF: Right. One of the subjects that came up at the conference is that even though this ribbon, as we refer to it, will be under more stress and tension than anything ever built by man, it will also -- by virtue of being so darn long -- be very dynamic. It will flutter in the wind over miles rather than feet.

JU: So there will be a need to simulate the resonances.

MF: Yes. It's going to seem to be a simple object, but because of its length and its multiple environments -- gravitational, atmospheric -- it'll be very complex.

Also, it has to avoid what's already up there. If you put something on the equator and then span zero to 100,000 kilometers, you will intercept the orbit of everything, eventually. Repeat those words: Everything, eventually.

JU: Did you say 100,000 kilometers?

MF: Yes, it's very long.

JU: Wow.

MF: Yeah. It's a significant proportion of the way to the moon.

JU: That's way more than I realized.

MF: To touch the earth and stay in the same spot, it has to more or less orbit at geosynchronous. Once you establish that, you have to put enough mass above geosynchronous, call it a counterweight, that the earth is attempting to eject from orbit, to offset the mass, or in this case weight, below geosynchronous, that the earth is trying to pull down.

So you fix that geosynchronous point at 22,300 miles, I think it is. You then get to decide how big that counterweight is versus how far out it goes.

If you put an asteroid of appropriate side, it can be just the other side of geosynchronous.

JU: Ah. So that's what I'm remembering from the original proposals.

MF: Yes, in the NASA proposal, and also in Fountains of Paradise, the Arthur C. Clarke book, it was a large counterweight very close to geosynchronous.

JU: So that's "big engineering" as you've said, and the modern concept is to go smaller.

MF: Right.

JU: In terms of computational challenges, we have this carbon nanotube ribbon which is 100,000 kilometers long, and it's on a collision course with every piece of space junk.

MF: Right. So Dr. Edwards' proposal is make the base movable, by putting it out to sea on an oil-rig-like platform, and then to move it when the computers say it has to move. You induce movement of the base, and then up and up, to miss the space station.

So I think I just described a very complex thing. You've got to move this thing hours or days ahead of a collision that you have to avoide by, say, 5 kilometers to be safe.

JU: That's the safety buffer?

MF: Right. Part of the Air Force monitors every object up there.

JU: Even screwdrivers?

MF: I think they get down to the centimeter, but don't quote me, I'd have to look it up. Anyway they track all this, and call up the space shuttle and say, you need to fly a little bit that way.

But now you have this object that you can't just move on a whim. And when you move one part of it, you move all of it. The space shuttle has a buffer of a kilometer, or 10 kilometers, I don't know, but say it's a cubic kilometer they have to keep clear.

Well if the space elevator's square danger area is a kilometer, it's not a cube.

JU: It's a column.

MF: A column that's 100,000 cubic kilometers, projecting through every orbit known to man. And in order to move it, you have to move it all.

There was a NASA retiree of considerable note at the conference, Ivan Bekey, and he came in and said, "Have you guys really figured this out yet? I don't think you have."

It was a great keynote: "Potentially Fatal Elevator Flaws That Must Be Addressed".

JU: So the strategy you just discussed has been on the table.

MF: Yeah, but not enough to stand up to the scrutiny of somebody like Ivan Bekey, who's saying, no, no, you have to actually figure this out. And people are like, yeah, yeah, you're right, we have to figure it out.

Then there's the question of what kind of communications network runs on the space elevator. There will be radio waves, but you'll want to bring up your browser too, and the latencies will be very different than for a transatlantic cable, so there are interesting computational challenges there.

JU: Fascinating. So as an observer of this scene for some years, what was notable about this year's conference?

MF: Well, for one thing, to see people working through the ideas that Dr. Edwards came up with, for example the collision management problem. On the other hand, there was a presentation on how, having shot the arrow, you get the rest of the material across. He proposed a deployment. But an engineer showed up, somebody working purely on his own time, and he has software to simulate the dynamics, and he said, no, that's probably not going to work, but this might. His idea was to keep the middle and the ends at geosynchronous, then spool out two in-between parts, then let go of the ends.

JU: So the most basic deployment strategy is still very much being discussed and debated.

MF: Yeah. It's such a complex object. This isn't a satellite. It's a thing that sticks through a lot of gravity gradients, in its first 10 kilometers it's beat up by the atmosphere.

So people are digging in, and coming up with different ideas that thematically fit in and move things forward.

JU: People have a general sense of what it would mean to get a several order of magnitude reduction in the cost of moving stuff into space, and of what applications could flow from that, and of the benefits of those applications. What's your take?

MF: Cost is a big deal. We ship things across the country because it costs 15 dollars, but we don't ship things to orbit because it costs 15 million. The space elevator changes the market dynamics completely. Because it acts more like a railroad with high upfront investment and then low operating cost, versus rockets with continuously high operating costs, that drives potential market opportunity.

But as one gentleman pointed out very forcefully, in addition to cost, there's capacity. You can only build and launch rockets so fast. The space elevator not only gets you low price, it's always there, waiting to launch payloads every day. The baseline elevator has, say, a 10-ton capacity every day, versus 15 tons every 3 or 4 months with a fleet of four space shuttles.

He was pointing out that all these big post-Apollo dreams -- going to the moon and Mars, tugging asteroid into orbit to mine them...

JU: ... large-scale orbital solar power collectors and beamers...

MF: ... right, which is one of my favorite applications. So for these things it's not enough to have low cost, you have to lift a lot of stuff up there.

Solar power satellites sound wonderful. Of course there's a whole set of other engineering and environmental problems. But in order to even make a dent, you have to move a lot into orbit. That guy was right. Capacity, capacity, capacity.

JU: What other uses are people talking about?

MF: Space tourism. People go up the elevator for a few hours, see the rim of the earth, that might be a gangbuster business.

JU: How long does it take to get up?

MF: 200 kilometers/hour is kind of slow, I think geosynchronous works out to 6 days. You don't have to go to geosynchronous orbit, though, unless you want to step off and have nothing happen.

A critique that made it to the New York Times was: "If you go to low earth orbit and step off, you fall." True. Of course the comeback is, go past low earth orbit by a bit, then step off and you fall towards the atmosphere, but you don't actually hit it. You bring along a small rocket, fire it, and you're in low earth orbit around the earth.

So you don't have this big fiery explosive launch that gets you to orbit in 20 minutes, but you don't have the dangers that go along with such an enormous expenditure of energy in a short time.

Instead you go up in something that's slower than an Indy racecar but faster than most people drive. It might be several hours to an interesting spot, several days to a more interesting spot.

There are people going up now to look down on the earth at $20 million a pop. If the cost came down to $200K, $20K...

...and there's a sociological argument that it would be a great thing for people to observe the earth from space, because there are no maps.

JU: Yes. It's historically had a powerful effect on the privileged few who have been able to go up and see that.

So there's the solar satellite concept, there's space tourism, what else?

MF: Everything else in between. Things we could do in zero gravity. Today there are experiments with special drugs, special materials processing, but they're experiments. They have to fit in the bay of the space shuttle, it only goes up every 3 or 4 months, you're not going to build an industry that way. But if you could ship your goods up and down every day, to an orbiting manufacturing plant that you just carried up on one of the larger space elevators, that's not necessarily a dream, that could be a business plan.

Two related things I meant to mention. First, it's highly scalable.

JU: Meaning?

MF: If we can build a 20-ton elevator, we can build a 40-ton one. And by the way, the first thing you build with a space elevator is ... a space elevator. The very first payload will be the second space elevator.

JU: To be build in another location?

MF: Right.

JU: Which addresses the single point of failure vulnerability.

MF: Yes. It is vulnerable, for all sorts of unfortunate reasons. But while it costs a lot to get the first one up, the second and third are relatively a lot cheaper.

So they scale out. I'm a performance engineer, that's one of our favorite terms.

And they scale up. There's no reason you couldn't scale up to a million pound elevator. And the thing is, there's no particular limit on the size and shape of the thing you carry.

Today things have to fit in nosecones, basically. The shuttle is a long bay, but cylindrical and 15 feet wide. Here you're limited mostly by how much turbulence it can take during the first 10 kilometers of the climb.

After that it's: "Oh, you want to build the space station." Just launch the space space station, tomorrow, and it'll be there in six days. That's a totally different way of looking at it.

It's like container ships. They've totally changed global trade.

JU: I was going to mention that. There's a book called The Box about the innovation of the standard shipping container, which is the physical equivalent to packets in a packet-switching network.

As a result, although it seems silly that some of the things we buy have criss-crossed the world...

MF: ... interesting meta-discussion there...

JU: ... yeah, but because of that technology, it really is economical for a lot of stuff to move around.

MF: I've seen bottled water from New Zealand. It astounds me. But it probably cost more to make the plastic container, which I consider to be heavy and low-value, than to ship water across the ocean.

JU: So the question remains, is this a leap of faith, like with the early space program, where we don't know the benefits but we intuit that there will be all kinds of spinoffs.

This feels like that, so the more you can identify operations that benefit from arbitraging the different between earth and orbit...but it's not crystal clear to me there's a long list of those.

MF: No, I agree. A critic of the space tourism idea pointed out that, yes, airlines and before that steamships were all built on the desire of people to go somewhere interesting: Disneyland, or a business deal, or an exotic location. Currently space only qualifies on the last point. So the chicken/egg problem won't be cracked by space tourism.

There have to be business reasons. We mentioned solar power. Another possibility is mining asteroids for iron and nickel. Those two items -- a lot of energy, and all this material -- there's a whole chapter in Dr. Edward's book that starts by figuring out your individual share of that. It's a lot.

The way I look at it is that a quarter of the world lives at or near the US standard, there's another quarter trying to achieve that in the next generation, another quarter just starting, and then one behind the curve.

Energy and materials are resources that space is full of.

JU: Clean energy in particular.

MF: It's getting a lot of attention lately, yes.

JU: Of course a lot of folks will point out that there are lots of earth-based solutions for clean energy, so why go to space for that. But I guess the answer is that it's not necessarily either/or.

MF: Agreed. But consider this. Scientific American's January cover story is a proposal to do big solar by 2050. Part of it is to use 1/5 of the American desert. The environmentalists have to swallow that, there's an albedo change in the earth, and so on.

Plus, it turns out that having a desert close to your civilization is fairly unique the world. Europe doesn't have one.

Now factor in population growth.

Meanwhile, satellite advocates say that if you had a band of solar cells a kilometer wide -- and that's a lot of solar cells, no doubt about it -- they'd produce, annually, energy equal to all known remaining oil reserves.

JU: It brings to mind one of the Long Now talks, I think by Vernor Vinge, in which he plots human civilization with population on one axis, and the amount of energy available to be used by an individual in the population on the other axis...

MF: ... something similar was done at the conference, by the way...

JU: ... OK, so you get a stepwise progression where each major advance in the level of civilization is tied to the amount of energy that could be mobilized by an individual.

It's tricky to talk about that right now, in an era when it's critical that we conserve.

MF: But you only get to conserve once. Replace all the fluorescent bulbs with LEDs, and do everything else you can, and you bought yourself ten, twenty, maybe thirty percent. If you want to increase your standard of living you don't conserve, you use up more energy.

JU: Right. So, how cool is it to be a performance engineer at Microsoft, take an interest in this topic, and wind up bringing the conference to the Microsoft conference center?

MF: Yeah. By the way, about 10 percent of the attendees were employees.

JU: I was wondering about that. The conference was fairly small, right?

MF: Around 50 people.

JU: Most of whom are practitioners, engaged in the R&D.

MF: Yes.

JU: But you got to bring it to other folks at Microsoft who, like you, the first time you went to one of these, will make all kinds of connections, and start thinking about the computational aspects.

Well done! And what a treat for you to get the chance to do it.

MF: Thanks. One nice thing was that it got posted on the internal calendar, and a guy I know found it, and he sent his two sons, a junior in high school and a junior in college. One's interested in mechanical engineering, and the other in civil engineering. I think this counts as a fairly large mechanical and civil engineering project!

So I meant to mention one of the interesting side effects of the length of the elevator, which is: What happens when you let go? Remember, everything above geosynchronous is trying to tear the ribbon apart, it's trying to leave the earth. If you let go at the right time of year, and the right time of day, you'll be on your way to Mars really quickly.

JU: [laughs]

MF: I'm serious.

JU: If you climb way above geosynchronous and let go, you slingshot to Mars?

MF: That is the appropriate word. Slingshot. I'm talking about a couple of months. Faster than the fastest stuff we've sent with big heavy boosters for these little bitty probes. You just take the biggest thing you want to send, and just let go, it'll be there in two or three months, for free.

JU: [laughs]

MF: And as a result, guess what's one of the first things you send to Mars using the earth space elevator.

JU: A Mars space elevator.

MF: Yes. There's a book on that, by the way. Science fiction. It's called Red Mars. It has an unfortunate ending involving terrorism. And of course the first space elevator conference, where everybody had read that book, was early 2002. Dr. Edward's said: "We've all read Red Mars, and yes, we have to take this into consideration".

There's been a study of how to defend the space elevator.

JU: I wasn't going to mention this, but that is maybe the worst vulnerability. You're not going to move it out of the way of a 747.

MF: The author's conclusion was that a consortium of private companies would own the space elevator, and the US government would trade defense for access to it.

You'd spend a billion dollars a year parking American assets around the elevator's airspace, and it'd be like, do not fly here, just don't, you will be shot down with no questions asked.

JU: And unlike the problem of defending the ground, this is a relatively well-defined region of the sky that needs to be defended.

MF: Right, it's not the whole continent. But everybody agrees it'll have to be defended. And we have the infrastructure, for better or worse, to do that.

But anyway, the possibility of moving around the solar system using space elevators is a whole other thing. Is that because there's interesting stuff out there? Because we're going to colonize Mars? Because we need more material sent down the elevator?

You might say that's visionary, or you might say it's just being practical.

JU: Well thanks again, this has been fascinating and a lot of fun.

MF: You're welcome Jon!

Ted Semon reflects on the 2008 Space Elevator Conference

JonUdell — Fri, 08 Aug 2008 14:21:00 GMT

Ted Semon, a retired software engineer, chronicles the efforts to develop a space elevator on the Space Elevator Blog, and volunteers for The Spaceward Foundation which administers competitions to develop several of the core technologies that will be needed to build the elevator.

Ted attended and spoke at the 2008 Space Elevator Conference held at the Microsoft Conference Center in Redmond. In this interview he discusses the concept of the space elevator, and the status of current efforts to bring it to life.

In a related interview, Maurice Franklin, the Microsoft employee who brought the conference to Redmond this year, reflects on the conference and on the goals and status of the project.

Ted Semon

JU: How did you become interested in the space elevator?

TS: I've always been a science fiction fan, and I read Arthur C. Clarke's Fountains of Paradise many years ago. The idea of the space elevator seemed so obviously the right way to get up out of Earth's gravity well.

When I retired from the software world a few years ago, I decided to learn what was happening with the concept. There were blogs and websites, but nothing coherent, so I decided to pull the information together myself on the space elevator blog.

JU: At this point were we into the modern era of the development of the concept?

TS: Yes, this was in 2006. I'd read the Brad Edwards book, but it was hard to find out what was currently going on, so I started the blog.

JU: The concept as described by Clarke is quite different from the modern one that's emerging, right?

TS: In some ways yes, in some ways no.

JU: Can you spell out the differences?

TS: OK, there are several. He had located his port on the island of Sri Lanka. Current thinking is that it won't be a land port, it'll be an ocean-going port, so you can move the space elevator if you need to, and get it out of the way of satellites and other things in orbit.

JU: I gather that's a "when", not an "if".

TS: Exactly. There's stuff up there, it's going to intersect the elevator, you've got to deal with that.

JU: And the object being moved, just to be clear, is a 100,000 kilometer strand of material.

TS: Right. It's a carbon nanotube tether, or rope, or ribbon, whatever you want to call it. One end is anchored to an earth port, something like an ocean-going oil platform, and the counterweight ands at 100,000 kilometers up.

By moving the ocean-going platform you can induce a wave that travels up the ribbon. You know which objects in space to worry about, at least the big ones, because you track them. And you know what's going on with the ribbon because you have sensors embedded in it, and climbers going up and down that signal their location. So you should be able to always move the ribbon out of the way of a collision.

JU: So one difference from Clarke's original vision is that the platform is mobile and sea-based. What are some other differences?

TS: Cost. He had imagined the cost would be something like the Earth's combined gross national products for a year, or some enormous number like that. The number now that looks more realistic is on the order of 10 billion dollars.

JU: And what accounts for that lower estimate?

TS: More knowledge now about how it's going to be built.

JU: Maurice Franklin and I discussed this, and his take was that the Clarke scenario assumed a huge mass parked in geosynchronous orbit, and that mass would be very expensive to lift. That ties into another evolution of the concept, which is that it's not now anchored with a large mass at 22,000 miles, but extends far beyond that.

TS: Right. Something like 100,000 kilometers. The counterweight in the Edwards plan is about 600 metric tons, quite a bit smaller than the Clarke scenario.

JU: The reason that's possible is?

TS: Because it's farther out in orbit. Well, it's hard to say an object anchored to Earth is in orbit, but it's 100,000 kilometers out.

JU: At its endpoint.

TS: Yes.

Let's see. He had also talked about the material being a carbon or diamond monofilament. I guess that's similar to carbon nanotube, and we should probably say he was right on that score.

He hadn't talked about powering the climbers, though. They used batteries. In the Edwards concept, the climbers are laser-powered. Lasers will be aimed at photovoltaic cells on the bottom of the climbers.

JU: So the climber is the robot that's attached to the tether, and ascends and descends?

TS: Right.

JU: I've got some sense of what a carbon nanotube is. A sheet of carbon atoms folded into a cylinder. But I'm not at all clear now that translates into a 100,000 kilometer cable. What's the architecture of that cable?

TS: It's composed of fibers. When you buy a 50-foot rope at a store, there's no fiber in there 50 feet long. They're all woven together, and that's what'll happen with carbon nanotubes too.

Right now the longest ones I know of, and have actually seen, are 5, 10, 15 millimeters long.

JU: The individual fibers?

TS: Right. So one challenge is to grow a longer fiber. MIT, for example, is working with a company called NanoComp.

JU: There's obviously a limit to how far you can go in that dimension, so then it's a question of how to compose a ribbon out of these strands, probably at several levels of hierarchy. Just like the way the Golden Gate Bridge cables are multistranded at several levels of hierarchy.

TS: Yes. The textile mills are very good at this stuff. If you give them fibers, they will weave you cables. The issue is going to be giving them carbon nanotube fibers of sufficient length and strength. That's where the bottleneck is now.

JU: What kind of diameter of cable are we talking about?

TS: If you're looking at the Edwards scenario, it's going to be a ribbon that's roughly 20 inches wide. And it is a ribbon.

JU: Why?

TS: When you're in space, you want as wide a surface as possible. So a micrometeorite strike won't sever the ribbon, it'll only poke a hole in it. And if you have it woven correctly, the strain is taken up by nearby fibers.

However the ribbon can be problematic in the atmosphere, because of wind effects. So it may be a cable in the atmosphere, widening out to a ribbon above.

JU: There needs to be a procedure for maintenance and repair, what's being discussed there?

TS: Some people are talking about making the tether into a big loop that's constantly rotated down to Earth where you do the maintenance. Nice in theory, but you've doubled the length. And what do you do about having a cable in the atmosphere and a ribbon above?

Another scenario is that the tether is made of segments. People worry that if the ribbon were cut, the two ends would fly apart. Not so. They'll sit there for some time, then gradually pull apart, but not like a snapping cable.

JU: Not catastrophic?

TS: No, as long as you get to it in time. So you should be able to disconnect and reconnect segments.

Another possibility: The climbers continuously reweave the ribbon as they go up and down.

JU: The carbon nanotube fiber and laser power beaming technologies seem to be two key ingredients in development. And those are what the space elevator games test, right?

TS: Yes. On the carbon nanotube front, there's a lot of work being done by industry and universities, and not only for the space elevator. Most people don't know or care about that, they just see a market for things much lighter and stronger than steel.

And with carbon nanotubes being measured at 2, 4, 6, maybe even 8 gigapascals -- and these are big jumps over a few years ago -- there's a real sense that we're getting close to being able to make a ribbon strong enough to support an elevator.

JU: How strong is that? What are the forces acting on the ribbon?

TS: The original Edwards scenario called for 130 gigapascals. Since then there's been some rethinking. Some good aerospace engineers think it can be dropped to 60 or even 40 gigapascals. That doesn't mean 130 is outside the realm of possibility, but we'll get from 8 to 40 and 60 a lot sooner.

JU: But the existing results are for radically shorter lengths.

TS: Yes, but you just need to something long enough to be woven.

JU: Ropes stretch, though, and we don't have any examples of 100,000-kilometer ropes or cables.

TS: Well, the total amount of cable in the San Francisco Bay Bridge would exceed the length of the space elevator.

JU: Really?

TS: It's different of course because it winds back and forth and around things, but there is some experience there.

JU: In terms of the laser power beaming, is this also a case where development is occurring for all sorts of other reasons?

TS: Exactly. NASA sponsors the space elevator games, but they mainly care about very strong materials, i.e. carbon nanotube tethers, and they care about power beaming, because they see applications for these things.

Boeing has just come out with a solid state laser in the 25 kilowatt range, and they say they can go to the 100 kilowatt range. If you can get 20 of those, that's enough to power your climbers.

JU: What are the current applications of those?

TS: Beaming power to a moon buggy so it doesn't have to carry batteries. Airships that stay up for weeks at a time.

JU: Are any of these concepts real yet?

TS: No, not yet, but the needs exist and they're trying to develop the technology to satisfy those needs.

JU: I gather one potential showstopper is the threat of natural or manmade attack.

TS: Actually the latter wasn't discussed too much at the conference, mostly the former. Micrometeorites, space junk.

That's being addressed in two ways. For small things, make the tethers wide enough, and engineer a replacement lifecycle. For large things, move the elevator out of the way.

JU: That computational grand challenge dovetails with Microsoft's strengths and interests, so that might be one interesting outgrowth of having had Microsoft sponsor and host the conference.

TS: That'd be great!

JU: Give us a sense of who was at the conference, what was discussed, and what emerged.

TS: There are two answers, I guess. First, there were a lot of old pros, people who've been working on this for years, and have come up with the intial concepts and solutions.

Then there are some new people in the last year. Some were invited, some just showed up.

There's an effort to make this into an international campaign. We've adopted the "four pillar" concept. It's something you need for any huge infastructure project. The pillars are: technical capabilities, a business plan, a legal and insurance framework, and public support.

That hadn't come together in the past, but this year we think we've gotten the enthusiasm, and especially the international support, to sustain that four-pillar approach going forward.

JU: We mentioned the Golden Gate Bridge. I recently learned that it wasn't a federal project, it was a municipal project. Likewise, the space elevator would perhaps ideally not be a big federal project.

TS: I don't think it has to be. I gave a talk this year on who I thought would build the first one. Ten billion dollars is a large sum, but not out of the reach of non-governmental entities. Money's an issue, but it's not going to be the showstopper.

I do think you'll need a government involved for defense, and for insurability, because international treaties will have to be written, and I think a government will be able to do that more easily than a business consortium.

I could see a group of US businesses getting together and saying to the US government, we'll take the financial and technical risk, in return please defend our elevator and help us deal with the insurability. If you do, we'll make you a deal: free launches, or discount launches. I'm sure there's a deal that can be made.

JU: In terms of why to do it, obviously everyone close to the concept takes it on faith that it's a good thing to do for all sorts of compelling reasons. To me, the solar satellite concept is maybe the most compelling, is that the application advocates tend to lead with?

TS: Many do. I don't personally. I'm skeptical. We use so much energy, and to put enough stuff into space to create space-based solar power that would make a significant dent, well, the amount of material is huge.

And we're not going to have a space elevator for 20 or 30 years. Meanwhile our problems will keep getting worse. We may have pilot projets, but nothing that'll power your refrigerator...

JU: ...or have any significant effect on greenhouse gases.

TS: Correct. Having said that, the concept is outstanding. And while I'm skeptical, I'm in the minority. Most advocates see it as a huge reason to build the space elevator.

JU: What are the other reasons that come up?

TS: Look at what you get: Enormous capacity, low cost, safer launches, and low environmental impact. Any industry that needs those benefits will want the space elevator.

JU: Such as?

TS: Well, what's making money right now is communications satellites. That's a big and growing industry. It'll be much easier to build satellites that don't have to reach orbit in rockets, and much cheaper to send them up.

Another will be orbital tourism. Being able to go up 100 miles, spend the afternoon, and come down -- we think that'll be a big moneymaker.

Then there are industries that don't exist today, except in labs, that need a space environment. To get them up today, you're talking about thousands of dollars a pound in rockets, and not a whole lot of pounds. With a space elevator it's hundreds of dollars a pound or less, and a lot of capacity.

I think once it's there it'll make a ton of money for somebody, maybe lots of somebodies, because you want more than one space elevator.

JU: Right. The first one is the bootstrap that gets you to others.

TS: Yes, and once you've got two, now you're in business. One of your failsafe scenarios is that you can leverage one to fix the other.

JU: So at the conference, the discussion was more about how to get it done than why to do it. What were the conclusions?

TS: That we're closer than ever. Arthur C. Clarke said that a space elevator would be built 50 years after people quit laughing about it. Well, people quit laughing some time ago. I think his prediction is a bit pessmistic. I think we're looking at 2020 to 2030 to actually be able to put one up. There's a general feeling that this is a real possibility, that it could happen in our lifetimes.

JU: If I were to attend the space elevator games, what would I see?

TS: You'll see a helicopter lifting a steel cable 1 kilometer up, and you'll see teams attaching climbers to the cable, and beaming power to photovoltaic cells on the climbers. We're hoping to have several competitors this year with a real shot at winning the prize.

JU: And the prize is?

TS: There are two. If you can get up the kilometer cable in two meters per second, and you're the only one who does, there's a million dollar prize. If you can do it at five meters per second, and you're the only one who does, there's a two million dollar prize.

JU: Really? So a million bucks for a laser-powered climber to go two meters per second, and nobody's claimed that yet?

TS: That's right. Last year it was 100 meters, and before that 50.

JU: So you've raised the bar?

TS: And the prize money, yes.

JU: And what's the other prize?

TS: For the strongest tether. Also a one million and two million dollar prize. You have to beat that house tether. Yours can be two grams, the house tether can be three, and it's made from commercially available material. So if you bring something new, like a carbon nanotube tether, and it can beat the house tether which is heavier, you can win the prize.

JU: And the lengths are?

TS: Two meters I think.

JU: Oh, OK, so nothing like the kilometer climb.

TS: No, it's a two-meter loop. Yours and the house tether are placed onto a special machine designed for this event, it stresses them equally, whichever breaks first loses.

Nobody's come close to winning that one yet. But last year we had our first carbon nanotube tether. MIT brought it, working with NanoComp. But it had been done so close to the competition that they weren't able to weave it a loop. So they actually tied a knot.

JU: They tied a knot!? [laughs]

TS: Exactly. But this year they'll have more time to prepare, and we know of at least one other team bringing a carbon nanotube tether.

So our ideal scenario for this year is that we have a carbon nanotube tether that blows away the house tether, and a 5-meter-per-second climber.

JU: And it'll happen where?

TS: Not definite yet, but we're hoping for Meteor Crater in Arizona.

JU: It's inspiring to think about this stuff!

TS: It's very inspiring to be on the inside. I got involved just because I was interested, but now I'm a huge fan and I'll do everything I can to help make it happen.

JU: The project seems to be attracting a variety of folks, from all walks of life, who are showing up, and wanting to participate, and finding ways to participate.

Maurice Franklin, for example, a Microsoft employee, has now made a real contribution by organizing this year's conference. But he also talks about some other folks who showed up, uninvited, with relevant engineering credentials, and made real contributions.

TS: That's right. People like Maurice will be the lifeblood of this project. And when you get involved, and start to see that this isn't some science fiction idea that's never going to happen...

JU: ... and that there are serious people, with serious engineering credentials, working the problem in a pragmatic way. It might not happen, but it could. Thanks Ted!

TS: Thank you, Jon.

How Microsoft's External Research Division works with a new breed of e-scientists

JonUdell — Thu, 31 Jul 2008 16:49:00 GMT

Tony Hey, VP for the External Research Division within Microsoft Research, leads the company's efforts to build external partnerships in key areas of scientific research, education, and computing. He's been a physicist, a computer scientist, and dean of engineering, and for five years ran the UK's e-Science program. These experiences have given him a broad view of the ways in which all the sciences are becoming both computational and data-intensive. Microsoft tools and services, he says, will support and sustain the new breed of scientists riding this new wave.

Audio: WMA, MP3

Tony Hey

JU: For this series of interviews I've spoken to a number of Microsoft folks who are working with external academic partners on projects that fall under your purview. The list includes Pablo Fernicola's Word add-in for scientific publishing, Catharine van Ingen's collaboration with Dennis Baldocchi at Berkeley on the analysis of C02 data, and Kyril Faenov's HPC++ project to bring cluster computing to the classroom. These are all pieces of your puzzle, right?

TH: Absolutely.

JU: By way of background, you've been a physicist, then a computer scientist, and then for a time led the UK's e-science program.

TH: Which would be called cyberinfrastructure in the US, yes. I'm on the NSF's advisory committee for cyberinfrastructure, it's a very similar goal.

JU: And then you surprised a lot of people by joining Microsoft. Take us through your initial role leading the TCI [technical computing initiative] and on to your current expanded role leading MSR's external research efforts.

TH: Right. So having been a physicist, and then a computer scientist working on parallel computing for years, and then chair of my computer science department and then dean of engineering, I think I understand the community we're trying to work with pretty well.

Also, as you mentioned, I worked for 5 years running the UK e-science program. That was about huge amounts of distributed data, and collaborative multi-disciplinary research in a variety of fields. The environment, bioinformatics, almost every field of science now has some element of distributed and networked collaboration.

The science agenda was for the tools and technologies to make that collaboration trivial, just as with Web 2.0 your grandmother can do a mashup.

I don't think the UK e-science program achieved that, but I do believe that Microsoft can help make tools and technologies available that will help scientists and researchers do their work.

JU: In your parallel computing phase, you helped write the MPI [message passing interface] specification, correct?

TH: Yes. I've been in this for 30 years, on and off. I have very good friends in the high-performance and parallel computing communities here in the US, and I was involved in European projects. There was a danger that the Europeans would go one way, and the US another, so it was time to see if we could get the community to put together a community standard.

It isn't an ISO standard, there wasn't a big standards body, it was a group of experts who got together with the academics and with the industry players. Rather a small set, and we used to meet every 6 weeks in Dallas airport, so you really had to be dedicated to go there.

JU: [laughs]

TH: But what came out of it was a standard which has stood the test of time. I co-authored and initiated the first draft. It's been much changed since then, and I don't take credit for the final thing, but I did try, with Jack Dongarra, to initiate the standards process, and I think I remember buying the beer at the first session.

JU: What's interesting to me is that despite that, you've been a vocal skeptic regarding raw grid capability. And you've been very careful to stress that in your view, the real challenges have to do with data -- the ability to combine large quantities of data from multiple sources, and enable people to make sense of it.

TH: Yes. I used to work in high-end supercomputing and parallel computing, but what distinguishes this decade is that we'll collect more scientific data than we have collected in the whole of human history. Instead of struggling with the problem of too little data, scientists will be struggling with the problem of huge amounts that they can't process or analyze. And it may be stored in different places, on different continents, so how do you put it together? How do you federate?

That's the real challenge. Very people want to use petaflop computers. Most of the biologists, chemists, and engineers only need lesser capabilities that can be provided by just a simple cluster. And then you put the cluster where the data is, because that's what's difficult to move around.

JU: Yes, Kryil Faenov made this same point in my interview with him. There are only a handful of intergalactic cloud infrastructures of the sort that a Google or Amazon or Microsoft can support, they're one-of-a-kind beasts, and you can't always bring your data to them. So he's interested in enabling organizations to stand up their own more modest clusters at the sites where the data lives.

So, let's discuss the opportunity that you see. In another interview you said:

Rather than wasting the enthusiasm and talents of science graduate students by assigning them the task of building systems capable of handling, analyzing and mining literally petabytes of data, scientists should look to computer scientists and the IT companies to raise the level of abstraction and to provide them with the components of a reliable and functional cyberinfrastructure.

That's the most concise mission statement I've found for what you're doing.

TH: Exactly right. Part of my reason for joining Microsoft was having had a great friendship, and many discussions and arguments, with Jim Gray, from 2001 onwards.

We argued and disagreed on many things, but we also agreed on things, and what we agreed on in particular is that a different paradigm is emerging. So for example there's experimental physics, there's theoretical physics, and now the third paradigm, it's clear, is computational physics based on simulation.

What we're looking at here is data-centric science, where you'll do collections-based research -- like you do in mashups, but now with scientific datasets. And increasingly, you'll use semantics to get from data to information to real knowledge.

So I came to Microsoft partly because of Jim Gray, but partly because I think companies can help. I struggled mightily with just open source tools. I used to produce open source tools myself, as an academic. MPI has a wonderful open source implementation, and that was one of the key things that we did.

But I also know that open source, particularly when produced by academics like myself, well, it works on my machine, but if you want it to work on your machine, that's your problem.

So one of the things I set up in the UK was, in fact, a software engineering center called the Open Middleware Infrastructure Institute, where I put a lot of money in to get these open source codes tested and documented and made more reliable and sharable.

That's why I think that a judicious mix of open source with commercial -- it could be from IBM, from Oracle, from Microsoft -- is the way to provide a more reliable infrastructure.

That's part of the motivation for the tools we're producing around the technologies that scientists use to do their publication, their data mining, and so on. I think Microsoft can really take a lead here, and that's why I joined.

JU: Elsewhere you've said:

Essentially I match up Microsoft researchers with major scientific problems that computer science technology can help to solve.

What are those major problems?

TH: So, I came with a purely scientific mission with TCI. But now I've moved into Microsoft Research, and we have a bigger agenda. In terms of external research, we focus on four themes.

One is health and wellness. That's bioinformatics, medical solutions, and so on. Really exciting, we've got some good projects in that area.

JU: I've talked to Kris Tolle and have done an interview with George Hripscak who's one of the recipients of funding in the genome-wide association studies program.

TH: Kris is great, she and Simon Mercer are looking at the biomedical area, and they've got a wonderful set of projects ranging from high-tech stuff involving RNA and HIV/AIDS down to the last mile of preventative health care, looking at ways in Latin America to take a smartphone and connect it to a low-cost diagnostic tool, like a blood-pressure monitor, and therefore do health care in these remote places.

The next major area is what we call E3 -- earth, energy, and the environment. That includes the astronomy work that Jim Gray started, which we now have followed up with the WorldWide Telescope, which is a wonderful tool.

JU: It's a brilliant thing. I've actually done two in-depth conversations about it for this series. One with Curtis Wong, and the other with Jonathan Fay.

TH: It does exactly the things we were talking about, it takes lots of distributed data sets, and allow you to search and visualize and do wonderful things.

So that's one example of an E3 project. Catharine van Ingen's project is another, and there are others. There's a project called the Swiss Experiment that's putting sensors all through the Swiss Alps to measure environmental changes.

JU: Before we discuss the other two areas, let me just ask: What is a project? I gather sometimes Microsoft Research puts out an RFP, and somebody like George Hripcsak at Columbia is awarded money to pursue his research. In other cases, though, there isn't necessarily funding, it's more of a collaboration, as with Catharine van Ingen and Dennis Baldocchi.

TH: Yes. In all cases, I want us to focus on genuine partnership with the academics. It has to be win/win on all sides. There are all sorts of ways. RFPs are one. Targeted funding, like we used to do in TCI, maybe sponsoring post-docs. But other things too, like delivering tools, data sets, services.

What can we do for the computer science community? That's another of our themes.

I used to teach in a computer science department, and I assure you my department was not atypical. We taught Linux, Apache, MySQL, PHP, Java, and they used a variety of scripting languages -- Perl, Python, and now Ruby on Rails.

To teach computer science principles it's quite clear you don't necessarily need any Microsoft technology. So the question is, how do we engage with academics in the computer science disciplines?

JU: And what are your thoughts?

TH: We have an opportunity. We need to look at what services, what data, what resources we can give them, so we can partner in a way that they feel is beneficial, so they can do research in the way they want to, and we can find out what services they need, and how we can make our tools more valuable.

Microsoft does now have the beginnings of some exciting service offerings. There's Live Mesh, and we have .NET online services coming along...I liked our internal name, CloudDB, better than SQL Server Data Services, SSDS, but...

JU: ...that's how it always goes.

TH: That's the way of it, yes. So that's in beta at the moment, and I hope by the time of the PDC in October we'll have a lot more concrete things to show. What I need to do is see what we can offer the academic community in terms of resources. Can we help them to explore multi-core? Can we get them data sets at scale that we've anonymized, so they can do research they'd otherwise not be able to do?

JU: And Dryad?

TH: Yes. We now have within Microsoft Research some internal resources -- cores -- and I want to make some of that available externally, and put some services around it, such as Dryad or Dryad/LINQ.

At the Faculty Summit I want to ask the community -- and after all, I came from that community -- how can we partner with you so that we can give you things that you value, and get your feedback?

JU: What is the Faculty Summit, who's been invited, and what do you aim to accomplish there?

TH: It's an annual event in the U.S., three or four hundred academics come, mainly computer scientists from the U.S. but there's a sprinkling from around the world -- India, China, Latin America. Really it's an opportunity for us to connect.

I've talked about health and wellness, earth/energy/environment, and computer science. Another area of focus is education and scholarly communication. We'll be unveiling plugins for our tools that make them more useful for scientists to do what they want to do.

JU: The NLM add-in for Word is an obvious example. Are there others?

TH: Yes, we'll announce a Creative Commons plug-in. Many people use Word, PowerPoint, and Excel, and are happy to share their documents. We'd like to give them a plug-in that will help them attach Creative Commons licenses to those documents.

We'll also have a research repository. At the university, I was supposed to monitor the output of my faculty -- 200 academics and 500 post-docs and grad students. What we did was insist on keeping a digital copy of not only publications, but also presentations at conferences, research reports, videos, data...

JU: ...especially data. That's a huge new area.

TH: It is in my view, yes. My undergraduates and engineering faculty never went into the library for traditional library purposes. They went there for a cup of coffee, a chat with their friends, a warm place to work, but not as a library.

So what is the role of the library? My view is very much the MIT DSPACE view that's been promoted. The role of a research library in a university is to be the guardian of the intellectual output of the university. And that needn't just be research, it can be teaching materials.

So we've used SQL Server, and the Entity Framework -- a bit like the RDF model of Tim Berners-Lee and friends -- to capture some semantic knowledge. So it tells you this is a presentation, Tony Hey gave it, the local organizers were so and so, it was done on this date, and so on.

JU: There's also the general notion of wrapping services around raw data sets. I've talked with Timo Hannay at Nature about how often, nowadays, somebody winds up publishing a paper as a "fig leaf of analysis" to cover what's really the publication of some data set.

TH: Timo and I absolutely agree on this. Research repositories which contain text and also data are going to be increasingly important.

JU: Although you're not wild about the term "data services", it's actually useful. I was talking with Jonathan Fay about his discovery of all the astronomical data that's online. On the one hand, it was astonishing to find that it was available at all. On the other hand, in the grand tradition of academia, these were gzipped tarballs that you could only use if you had an extreme amount of specialized knowledge and capability.

What you get, with WorldWide Telescope, is a service layer wrapped around all that raw data that makes it available to a vastly wider audience.

TH: Absolutely. Same with Catharine van Ingen's project. This stuff was locked away in files, and nobody knew what was there. By making it available and exposing it in new ways...you're right, these data services are very important.

And they're the basis of some of our other projects. So for example, Valerie Daggett at the University of Washington does protein folding, but she also does protein unfolding. She regards protein folding ab initio, right from the beginning with just the structure, as too difficult. So she takes the folded structure and unfolds it, and then looks at the possible foldings you can get. She calls this dynameomics. It involves storing detailed simulations, and we've made a database to help her do that.

JU: How would you characterize the nature of the collaboration between Microsoft Research and Valerie Daggett?

So, for example, with Catharine van Ingen and Dennis Baldocchi, it was a really interesting mesh of interests and capabilities. Dennis is a climate scientist who's plugged into a worldwide network of sensors, but he's not an informatician, he's not someone with deep training in how to probe and reshape a body of data. But that's what Catharine brings to the table.

So in this protein-folding collaboration, what's the partnership really about?

TH: It's on two levels. Valerie really is a computational scientist. She does these computationally-intensive calculations, and she uses national supercomputers.

One of the things we've done is give them experimental Windows HPC clusters, so instead of doing it remotely they can actually get a lot of calculations done on local machines.

The other part is that they don't have particular expertise in databases. So Stuart Ozer, who used to be in Jim Gray's group and now is back with SQL Server, collaborated with them to set up a data cube.

JU: It seems like the transfer of database expertise is a common thread in a lot of these collaborations. Although many of these folks may be computationally-oriented scientists, and may know how to work with algorithms and with code, the data management is another kind of discipline, and not one that necessarily comes naturally.

TH: That's right. By the way, we're also active in computational education for scientists. When I did that in the 80s and 90s it was about algorithms and parallelism and things like that. But you're quite right, it's now, in addition to those things, about knowing how to deal with data.

We have projects with two Nobel Prize winners, Phil Sharp at MIT, and Carl Wieman at Vancouver, looking at what you teach biologists and physicists about new skills, in order to produce a new generation of computational scientists who understand the data as well as the computation.

And I'd be remiss if I didn't mention Stephen Emmott. I emphasize the data, but he'd say that the complexity of the modeling that you have to do with this data is as important. And therefore, some of the abstractions from computer science can really help the modeling side of science.

One of our engagements is a joint bioformatics modeling institute in Trento, and that's an initiative of Stephen Emmott and his team.

JU: I guess that most people know Microsoft has a massive research arm, and there's been a lot said and written about internal technology transfer -- something gets invented in MSR, then it's thrown over the wall into a product group. People have heard that story, but this other story about external collaboration isn't so well known.

TH: That's true, though it does link to our research within MSR. In terms of the computer science and education communities, we have wonderful tools here that actually don't end up in products. One of the things I hope to do is make more of these available.

We now, at Microsoft, have two OSI-approved open source license, Ms-PL and Ms-RL. I'd like to make some of our tools, which aren't going into products, available so that we can build communities and show what great tools there are. Tools that really do things the computer science community and science community want.

So, I talked about our four themes -- health and wellness, earth/energy/environment, computer science, education and scholarly communication. In addition we have what we call ARTS: Advanced Research Tools and Services. There we're trying to develop tools and services that academics and computer scientists will find valuable.

And there are many others. We just did a count, and in total, with RFPs and small projects and big projects, we had, over the whole of Microsoft Research something, like 400 projects with external partners in universities.

My challenge is to focus that a bit more, and make sure we capture and build on the ones that are successful.

JU: Well, very good, Tony. Thanks a lot!

TH: Thanks very much, Jon.

How the WorldWide Telescope works

JonUdell — Mon, 14 Jul 2008 09:25:00 GMT

Jonathan Fay is principal developer of the WorldWide Telescope. In this interview he explains how the project has yielded not only a breakthrough software product, but also a reference model for the acquisition, transformation, and visualization of astronomical data. You'll learn not only how the WorldWide Telescope works, but also why it exists: To fulfill the education mission discussed in a related interview with Curtis Wong and Roy Gould.

Jonathan Fay

JF: As long as I've been doing computers, going back to the early 1980s on TRS-80, graphics, and visualization of data and the earth and space, were interests of mine. I'd gotten a department-store telescope one year for Christmas, and loved looking at stuff through the light-polluted LA skies.

JU: So you were in the same boat as Curtis Wong?

JF: Yeah, you could really only see planets and the moon in any detail. But I was passionate about computers and astronomy. Every time computers got more powerful, I'd look into visualizing the Mandelbrot set and the stars as a litmust test.

In 2001 I was development manager for HomeAdvisor, and we were assimilating a research project called TerraServer. Tom Barclay, a researcher who was working with Jim Gray, said, "Hey, USGS has this DEM -- digital elevation model -- data that they'd like me to load into TerraServer. I wonder if you have ideas about what we could do with it."

I'd been very much into 3D visualization. I have this program called LightWave, which goes back a long time but is now used for things like Serenity and BattleStar Galactica, so I started taking TerraServer images and USGS data and creating hills with texture-mapped images.

Then Tom Barclay told me how NASA was using satellite weather data, watching over many days, and getting rid of the clouds so you could see the surface of the earth. They called it the Blue Marble project. I found and downloaded that data, and also some global digital elevation data, and starting creating a hierarchical 3D view of the earth so you could zoom in and browse. Then I worked to bring that into TerraServer, because we had resolution down to a couple of meters.

But this was just a side project, and there wasn't interest in developing it, so I decided to look into visualizing other astronomy data.

JU: This was around the time in 2002 when Jim Gray and Alex Szalay published their paper entitled the World-Wide Telescope?

JF: Right. Jim talked about TerraServer "pointing up" as the next thing. He was already getting himself embedded with astronomers. I didn't see much of that. Tom was babysitting TerraServer while Jim went off into the astronomy end of things, and I was still doing geo, so we weren't collaborating.

After having made some demos, a lot of people thought it was cool, but that was all. So I kept that on the back burner, and moved into some other groups. At the same time I was building my observatory. In Seattle, you take pictures when you can. If you can't push a button and have your observatory open up and take images when you get clear skies, by the time you set up you'll be clouded in. I wanted to automate the whole process, including image processing. That introduced me to the whole pipeline of data collection, processing, and subsequent research.

Although I'm an amateur, I had to drill into the world of data and image processing that professional astronomers had to deal with. I was using the same resources.

JU: I'd like to hear more about that. A lot of us are aware that those data and image resources exist, but it's really unclear how to make use of them.

JF: You know, there is a lot available, but most amateur astronomers had no idea it existed, it was very hard to get to, and even the scientists had a hard time getting access to it. Essentially it was locked up in silos.

JU: If you know where to find the gzipped tarball, and then if you can unzip it and figure out how to use it, without any documentation about metadata and formats...

JF: Right. So, I'd heard about this very large database of stellar objects, the US Naval Observatory's USNOB. It was 100 gigabytes. At that time, there were barely consumer hard drives that could hold that. Forget transferring it over the network, it's 120 CDs, the only way to transfer the data was to ship hard drives around the country.

JU: Yeah, I remember Jim talking about doing that.

JF: I'm just an amateur, but I feel like I need the data, so I found out that this guy named Dave Monet, in Flagstaff, would let me ship him a hard drive and he'd put the data into a Linux-formatted partition and send it back.

On the one hand, I was shocked to see how easy it was for me to get access to the same data that the professional astronomers were using. And by easy, I mean it was possible.

But on the other hand, I realized you had to be really committed, and know exactly what you're doing.

JU: Right. There were no services wrapped around the data to make it useable by anybody other than a 100% focused and dedicated researcher.

JF: As I started doing more with imaging, I had the concept that I should flip my earth inside out and render the sky. One of my friends, Doug George, created a full-sky survey, in gorgeous color, but the software that went around with it would take ten or 15 seconds every time you moved your view. Nothing resembling interactive or realtime.

And here I had this application that dealt with the same quality and quantity of data instantaneously. So I say hey, I can build an engine to go with your data.

And I told him about a company, called Starry Night Pro, that was using some 3D effects but not actual image data from the sky. He wound up licensing his data to them, but the result they got was closed and self-contained.

JU: What kind of imagery was in it?

JF: What we'd now consider a low-to-medium resolution full-sky survey of the northern and southern hemisphere.

JU: When you say low-to-medium resolution, what could you see if you zoomed in on a galaxy?

JF: If you zoom into M51 in WorldWide Telescope, using the Hubble imagery, it'll be about 4000 pixels tall. And in their survey, it's about 4 pixels tall. You can barely make out that it's a spiral galaxy.

We have the entire sky at one arc-second per pixel, and for objects like M51, thousands of pixels tall. And of course every time you go twice the resolution, it's four times the data.

They wanted to fit everything on a CD-ROM. For us, we're talking about terabytes, it's not something you distribute. I thought you should install a small application, and the data comes over the network.

JU: And that's how WorldWide Telescope does it?

JF: Right. Everything except the thumbnails comes over the Net. We use the thumbnails to get the wordwheel functionality with search.

JU: The data file's about 3 megabytes?

JF: There's about 12 megabytes of thumbnails, but yes, the catalog is about 3 megabytes.

So, I had this vision for a product, but the economics were wrong to do it as commercial software in the astronomy market. Plus, they'd want to do something aimed entirely at high-end amateurs, not at professional astronomers, or at the general public who are the outreach targets for professional astronomers.

And then Curtis and I got together. I envied his position in research, being able to explore new things that hadn't been done before.

It turned out that Curtis had been exploring how to create an educational environment with rich tools for exploring space, and he'd been collaborating with Jim Gray on TerraServer, and now he was looking for the technology to make it possible.

Here I had this technology, and was looking for somebody who was enthusiastic about having a purpose for it. So it was the peanut butter and chocolate moment. Curtis passionate from the education side, me from the technology side, happening to be in the right company at the right time.

So I made a demo using with the Sloan Digital Sky data, and Jim went crazy over it. This was the visualization aspect he'd been looking for. It was the front end that makes the data consumable.

JU: Tell us about the WWT's back end, and how it relates to what Jim's team built.

JF: To get the data out of the silos, Jim was involved in the National Virtual Observatory and the International Virtual Observatory Alliance. If you know how to talk these VO standards, you can exchange data, and you can do queries against other people's data.

JU: So on the one hand, these standards enable you to combine data sets that you fully assimilate. But on the other hand, they enable federated query.

JF: Right. A lot of the astronomers were dealing with data extracted from catalogs. You took image data, and then you got the numerical analysis out of it, and stuck that in the database. The transfer of images wasn't really their domain for this round, they wanted to do the stuff you could put into SQL Server.

So while TerraServer put earth image data into SQL Server, the sky image data was lagging behind. But you could query from a source on the Internet, and then join it to some other data coming from another source. Sometimes it required the data to marshall from one machine to another for efficiency, but essentially it meant you didn't have to translate everything into your database.

JU: But I assume that federated query isn't happening in WorldWide Telescope. We're not waiting for requests to go across the network, you've combined the datasets for your purposes.

JF: There are common sets of data that you'll need all the time. It's a relatively small amount, and we download that to your client. The thumbnails, the catalog.

JU: And what's in the catalog?

JF: The Messier objects, the NGC objects, the list of solar system objects,

JU: And coordinates for them...

JF: Yes, and magnitudes, and classifications. For the 10,000 brightest stars. Probably 30,000 objects in all. We'll make that live on your machine so you can zip around in the sky, look at stuff, and say, hey, what's that?

JU: Which is what every planetarium program does, right?

JF: Yes, but that's generally where they stop. They go a bit beyond, by having a bigger download. We do it in 20 megabytes, they may have 250, or a gigabyte, but that's all you'll ever get.

In our case, when you start up and your client contacts the WorldWide Telescope, we give you metadata saying what sources are available: the Hubble collection, the Spitzer collection. The metadata tells you where to go get the imagery. Some of it we'll host in Microsoft's data center, for scale reasons, and to ensure that it's available. But this data can be anywhere: Space Telescope, JPL...

JU: So I'm looking at the list. Which of these many sources are you hosting?

JF: We're hosting a lot of the data we launched with. Partly because we don't yet have a space act agreement with NASA. Even though we've collaborated with a lot of people who are NASA-funded, they're not allowed to acknowledge that collaboration or put anything into a legal document until we have that agreement done. While there are some people we could have just pointed to as data sources, it'd be in violation of internal NASA policies. So we're hosting more than was strictly necessary for the initial release.

But the concept is that you can plug in other sources that we're not even aware of. You just load metadata references into your client, by going to a website for that community or organization, and then you have access to terabytes of their data.

JU: The standards talk about how to represent objects and their metadata. Do they also talk about how you query a source, since they're all going to be huge? What's the query protocol?

JF: At WorldWide Telescope we understand what's called VOTables. There are standard ways to create queries, and standard ways to get results.

There are two ways that can happen. One is that our servers can do the queries, consolidate and cache the results, and we regurgitate the data as needed to our clients. So we do a VO SIA (simple image access) query to Hubble occasionally. When they have new images, we download these 500 megabyte or gigabyte images, which would be a very big download for a client, and we chop them up and create a tiled multi-resolution pyramid that we store on our server. The raw consumer wouldn't have have been able to use that data, but by putting our value-add into the pipeline -- Hubble took the image, Space Telescope processed it and put it up on a web service, we do another step of processing to make it visualization-friendly -- now lots of people can see a thumbnail, click on it, it zoom in, and the instant that they click and zoom they're already seeing the image. And as they zoom in further, they see all the gorgeous detail, but they don't have to download all the data.

JU: Is this engine related to the Deep Zoom technology?

JF: We predate Deep Zoom. It has some similarities, but the difference is that Deep Zoom and Seadragon are 2D technologies that use the graphics engine for doing tiled multi-resolution images. We actually have to align all our images in 3D space because from the earth, space looks like a big sphere at almost infinite distance, but there is a curvature to it.

Imagine taking a round room, and trying to put a bunch of bathroom tiles on it, and grout it. The tiles seem to come together and have parallel lines for a while, but eventually it stops working well. Maybe you can take one line around the equator, but as you go up you have fewer tiles, and weird-shaped tiles, and nothing lines up.

That's the problem we have. We're looking at spherical data, so we had to come up with a new spherical transform that preserves the poles. In previous projects, like Virtual Earth or TerraServer or Google Earth, the poles weren't important, because nobody lives there and nobody needs map directions for driving around there.

As far as the earth is concerned, you can cut off everything above and below a certain latitude and nobody would care. But you can't treat the sky like that. And you can't treat the moon or other planets that way either.

So we had to come up with something called TOAST: tesselated octahedral adaptive subdivision transform. It creates a 360-degree wraparound view that's either a planet surface or the infinite sphere of the sky, and lets you represent it using a 3D graphics accelerator, very rapidly and efficiently. So we can have an image pyramid the way Deep Zoom does, and TerraServer before it, but we don't have to give up the poles.

That was something that didn't exist. There was Mercator projection, which is how you're used to seeing the earth mapped onto a flat piece of paper. It's hard, you have to do weird math to make it work at all. Then there's equirectangular projection. But there was nothing that could deal with storing an image in a spherical projection.

JU: So there are multiple full-sky surveys that you can switch between. So for example you can be looking at the Milky Way in the standard view, then switch over to infrared view and see it as an incandescent band.

Is it the VO standards that enable you to weave those views together in a coherent way?

JF: No, that's where TOAST comes in. What astronomers did before is that, because there was no way to visualize the full sky data, they would store all their images as a bunch of individual...

...OK, you have a sphere in the sky. You put a camera on it and take a picture. What shows up on the film is what's called a tangential projection.

Imagine taking a beach ball with all the stars plotted on it, and putting a light in the middle, and putting the beach ball up against a wall touching at one point. The stars will shine out and hit that wall. All of these beams are projecting from the middle, to where they lie on the sphere's surface, to where they hit on the wall. It's a way of taking something round and making it flat.

As long as you're looking at a very small part of the sky, there isn't very much distortion. But when you start looking at a large part of the sky the distortion becomes huge.

What astronomers did was put these tangential projections into databases, and they even knew how to mosaic them to make bigger chunks. But when it came to anything larger, it broke down. If they made really big mosaics, they had to use projections that couldn't represent the poles, and everything would get more distorted the farther it got from the equator.

So now we have services like NASA SkyView. NASA has over 50 full-sky surveys sitting on servers, and while they participate in the Virtual Observatory, the images themselves are using a private well-dcumented standard. So we gave them code for TOAST.

It used to be that when people made a request for a wide area of the sky, they would return multiple images joined into a mosaic. But now we said, we could ask for just a single tile, at a given level of resolution -- one tile that was the whole sky, or one tile that was a tiny piece of the sky -- but everything was laid out in a very specific grid for our projection.

While their software couldn't do it very quickly, it allowed us to go through and get all the tiles from their servers, for all these different studies, and put them up on our high-capacity servers.

So there's an automated path to get from a bunch of individual pictures of the sky to this full-sky mosaic that can be seen seamlessly.

JU: So where's the TOAST transform being applied?

JF: Right now it's being applied, for that data, on their servers.

JU: So you gave them the algorithm, and they're running it for you?

JF: That's correct. And eventually they'll be able to host the data when they have the capacity, so you could point a WorldWide Telescope client there. And even today theoretically you could do it.

JU: They keep the sources as they acquired them, but make the output of this transform available to queries?

JF: They generate the transform on the fly for each query. If they added a cache and then kept it warm, it would be acceptable for interactive use.

JU: When you look at the source list in WorldWide Telescope, those are the surveys you're talking about?

JF: Yeah, ROSAT and WMAP and things like that. Those are the full-sky surveys. So for the first time ever, we've assembled a view of the sky where you can look at everything from radio wave all the way to gamma. All the way from the longest-wavelength lowest-energy part of electromagnetic spectrum to super-high-energetic particles.

JU: It's completely amazing, and it's wild to be able to cross-fade between them and compare the differences.

JF: We put together a standard for how you can visualize a spherical data set, we've given people the ability to create this data, and we've provided a client that knows how to accept this astronomy data -- both the spherical data and the original tangential images.

So when you have a study from Hubble, they can use the original tangential images the way they came off the camera, and in WorldWide Telescope we figure out the math and do the 3D transforms so that when we align that to the TOAST background from another full-sky survey, all the stars are exactly where they should be and everything lines up.

And because we have the universal coordinate system -- right ascension and declination -- we can put things in the right place in the sky. When you cross-fade you may be looking at apples and oranges, but you're looking at them on the same tree.

JU: Is this going to be a public standard? Can other clients use your services, or other services that support it?

JF: We've offered the algorithms and code to other organizations, like JPL, and we've even told Google that if they're interested in reworking their all-sky surveys to work with this format, we'd help. But they've got such critical mass around their current projections that they don't think they can take that on anytime soon.

JU: There's been some pushback, as you know, about WorldWide Telescope being a Windows-only product. But the project is much broader.

JF: Yes. And part of it is that all the data we support in WorldWide Telescope, and the WTML language we use...

...when people ask me how WorldWide Telescope differs from an astronomy program like Starry Night, I say that it's like a browser, like Internet Explorer or Safari or Firefox, but it's a browser of data in formats that are astronomy-friendly, like VOTables and WTML.

JU: Now WTML isn't the XML syntax you see when you save a tour and look into the file?

JF: Right. That, we're not even documenting. That's the tour XML format. But if you look in your user folder, or add objects to your collections and look in your documents folder, you'll see WTML there. It describes objects, hierarchies, network links, images.

A tour in WTML is metadata that says, this is the tour, what categories it's related to, what objects it visits.

We can also have things that say, there's an article in Sky and Telescope about M51, and it has that object's location in the sky. When you join the Sky and Telescope community in WWT, and you're browsing around and you find M51, you can look down in the context search and see the article, and open it up.

JU: That'll depend on which communities I belong to?

JF: Yes. We always show you the WorldWide Telescope stuff. Then when you log in we show you the union of that and stuff for the community you're currently looking at.

JU: OK, very helpful. Now let's go back to your discussion of projection, and see how it relates to my experience last night. I found the Milky Way, and I wanted to pan west, but it seemed like things wanted to spin around.

JF: There's two ways to look at the sky. First, looking at the full spherical view as if the earth didn't exist. You're earth-centered, but the horizon isn't blocking your view. North is up, south is down, and unless you specifically spin your view, when you move, north will always stay north.

JU: That's the view without the horizon.

JF: Right. With the horizon, the zenith always stays looking up, and as you move around, if you're looking at the zenith, it will always stay at the top. It can never go below the midpoint of the screen.

On a space station where there is no up or down, you'd think you could design anything and people could just float around in 3D space, there'd be no preferred direction. But the reality is that humans get extremely confused. Your brain has a natural desire to have an up and down and left and right, and when you invert those, you don't process things.

So if you were in the View From Here mode, the zenith always stays up. If you're in the other mode, looking at full universe, and you went to the north pole and tried to move beyond, you'd only be able to spin. You would not be able to pull the north pole beyond the middle of your screen, because that's your viewpoint. So then south would start becoming up, and left would be right, and you'd be spinning in the hamster ball.

JU: So if I want to look at the Milky Way, and then swing left to locate the Pleiades..

JF: To simulate looking at the sky, go to View and select the location where you are, and say View From Here. Then it will show you a horizon, and north/south/east/west, and north is straight up. Then it will simulate your eyes. If you're standing up and you look at the horizon, then you look up and up, what happens? When you're looking up, your head is tilted all the way back, touching your back, and you can't tilt any more. To see any further back you'd have to fall over.

So then what do you do? You rotate yourself and look south. That's how your head works, and that's how a telescope with an alt-az [altitude/azimuth] mount works.

We're trying to put on constraints so people don't get lost and upside down and backwards. But unfortunately it's hard to explain what happens when you get to the poles.

JU: Do you provide an unconstrained view?

JF: We do not. We cannot simulate an unconstrained view. The only thing we do allow is that, once you're viewing something, you can rotate the camera's view by hitting Control and then dragging left and right.

JU: Ah.

JF: It's possible that's what was happening to you. We have a Reset Camera if you want to go back to neutral.

The reason for this feature is that when you're making a tour, you might need to orient your view. M51 goes up and down but your screen goes left and right. If you want to zoom in and frame it, you need to rotate your camera like you would a real camera.

JU: OK, that may have been the confusion.

JF: When you get in that mode, we try to make north-south-east-west make sense based on that, but it will do strange things at the poles. We still try to keep north, minus your rotation, up. But that mode is a little strange. We give that feature so people making tours can frame things better, but it's not something we try to document or recommend that people use for normal browsing.

So, if you care about your position on earth, use View From Here. If you want to ignore your position on earth, use the default mode. Then we don't care where you are, we're going to show you the whole sky, and date and location are ignored, it's just the sky, immutable and unmoving. Well, the planets move around on it.

JU: We'll never get to the bottom of all this, but I think you've given us a good sense of what I was really looking for, which was: What's actually been accomplished here? In terms of taking this raw astronomy data and correlating it in a way that's not just consumable in terms of quantities of data transmitted over the network, but in terms of making sense of objects and relationships.

JF: The vision of getting everybody access to all this astronomy data required systematic changes at every single level. We built on some things that Jim pioneered with NVO, and worked from there, but it was very systematic. How people process the data. The client to access the data. The protocols over the wire. Educating people, providing the context for it.

We put a lot of things together, but we also created a systematic model for how to do everything end to end, top to bottom, left to right. Now there may be other people who use the pieces that we've created, and then change them to use different data sources, different visualizations. Say someone creates a Mac client, or an iPhone client, that's possible. Or a mobile phone version of it, or a web-based version. Over time we or others can replace various components, but as a reference model for solving all the problems in order to get the data into people's homes and into their eyeballs -- you had to solve for all of those problems, otherwise people are still blocked from being able to really explore.

JU: Will this end-to-end pipeline be documented?

JF: Things like TOAST, and WTML, and our communities interface will be documented. There will be documentation, tools, and code coming out over the summer to help people understand more. As for some of the protocols, we'll need to do some work to make sure they're ready for us to recommend as standards.

JU: Excellent. Well, thanks Jonathan!

JF: OK, thank you.

The story of the WorldWide Telescope

JonUdell — Fri, 20 Jun 2008 12:18:00 GMT

The WorldWide Telescope was first shown to the public at TED 2008, in a joint presentation by project leader Curtis Wong, manager of Next Media Research for Microsoft, and Roy Gould, a science educator with the Harvard-Smithsonian Center for Astrophysics. In this interview they discuss how -- and why -- the WorldWide Telescope combines many sources of astronomical data and imagery to create a seamless view of the night sky.

Curtis Wong

Roy Gould

CW: I was interested in astronomy as a kid, but when you grow up in Los Angeles, the odds of seeing the Milky Way are pretty slim. I think the only time it happened recently was during the quake when the whole city lost power.

It wasn't until much later that I actually got to see the Milky Way, and other objects I'd seen pictures of, and it was really quite a transformative experience. I always wanted to figure out how to recreate and share that experience.

Early on, in the 80s, I made a little HyperCard stack called MacTelescope, which was my attempt to create the experience of looking at the sky, and then -- if you could manage to see the Milky Way -- to zoom into a section where there are all these interesting globular clusters and nebulae, if you know where to look.

JU: So there was already the idea of taking people on a guided tour.

CW: Exactly. Later I moved from the Voyager company to a company called Continuum, a little think-tank organization started by Bill Gates. The company was thinking about authoring tools and media. The project I wanted to do was called John Dobson's Universe.

John Dobson was a physical chemist at UC Berkeley who got drafted to work on the Manhattan Project. He was there to do the chemistry for tuballoy, which was the code word for uranium. He became a conscientious objector, left the project -- which was pretty hard to do -- and became a Vedantan monk. Then he became interested in looking at the sky but, being a monk, he didn't have any money for a telescope.

He knew that San Francisco shipyards were sources of glass discs, but they were too thin. Conventional wisdom said that you need thick glass to be able to grind a mirror. But he defied wisdom and found a way to use round porthole glass. He also came up with an ingenious way of mounting the telescope, using cardboard concrete form tubes. His design is now one of the most common designs for telescope mounts in the world.

He also created an organization called San Francisco Sidewalk Astronomers, where people who have telescopes are encouraged to take them out into the public and show people what's up in the sky, and explain what's going on.

John spent a lifetime in national parks, and in San Francisco, talking about astronomy to the general public. He was really good at taking complex ideas and conveying them to the public.

I remember once when he was showing the Andromeda galaxy, there was a picture with the galaxy in the background which looks like a kind of fog, with a lot of stars in front, and he said: "The stars you're looking at in front are like raindrops on your window, looking at a distant cloud."

So anyway, we started that project, got about halfway, then other things happened and it got cancelled.

JU: It's a great story!

CW: So I've thought about astronomy for a long time. At Microsoft, I heard a talk by Jim Gray who was applying his database expertise to astronomy.

JU: I looked up a paper that Jim published with a guy from Johns Hopkins, Alex Szalay, and it's called The World-Wide Telescope, but interestingly, the title also includes the phrase: An Archetype for Online Science.

The idea is that, not just in astronomy but in all of science, we're getting to the point where there's less direct observation, and more collection and analysis of data on a really large scale, happening in ways that are computationally assisted, and also assisted by the collaborative properties of the Internet.

CW: Exactly. We're getting this deluge of data. The challenge then becomes how you process, how you utilize. Bringing computer technologies -- SQL, visualization -- can really help.

So Jim had written that paper with Alex in 2002, and he'd given a talk on some of the work he'd been doing with the Sloan Digital Sky Survey at that time.

JU: Yeah, I've listened to that talk, and I was hoping you could help me connect the dots between the work that was done there and the federated virtual observatory which, for him, became a case study in the use of XML web services to create an Internet telescope that was a federation of radio astronomy services.

CW: Right. Alex and several other research scientists came together to create the National Virtual Observatory, which establishes common protocols for accessing astronomical data and imagery.

When Jim told me they were starting that project, I told him I wanted to help. Part of the pitch was a PowerPoint I made about SkyServer that showed how you could embed that data and imagery in a virtual environment.

What they were thinking was that astronomers would be querying the database, and pulling out objects. But I thought that to make this an interesting educational resource, we would need to build an environment in which people could create and share stories about the objects, and could connect those stories to original source information.

Later he came back to me and said that the SkyServer data was released, and he wanted to redesign the website to make the data more accessible to the public. So I helped with that. SkyServer DR2 (data release 2) was the redesigned website. And I used that to make the case for building the learning environment that became WorldWide Telescope.

JU: Under the covers at SkyServer, there was a lot of work done to correlate observations from different sources of data.

CW: Exactly. Alex Szalay did a lot of that work, in collaboration with people from other universities.

JU: And that's the foundation for what we see now in WWT?

CW: That was sort of the first case. It was the Sloan Data, which is just the northern sky. Then at TechFest in 2006, we were working with Alex to take the SkyServer images -- he has an image server that'll give you an image from coordinates, and from that you can get data about other objects in the field of view. For our TechFest demo, our developer Jonathan Fay -- who had been fiddling around with building his own hierarchical multi-resolution tile-browsing engines -- put together an engine that would pull the images from SkyServer and assemble them into a mosaic that you could browse and zoom in and out of. That was the first manifestation.

Meanwhile, we'd gotten some interest from Harvard. Alyssa Goodman found an intern for us who was passionate about education, and she came to work with us in the fall of 2006.

And then in January, as I was emailing back and forth with Jim about our plans, he disappeared after sailing out in San Francisco bay to spread his mother's ashes.

So when TechFest came around again in 2007, we decided to rename it WorldWide Telescope in honor of Jim. We started building it in March. That summer was a big effort to secure image sets and data from lots of different sources, as well as building the engine and defining the authoring environment. We showed a rough prototype at the Astronomical Society of the Pacific meeting ... Roy, was that in Chicago?

RG: Yes.

CW: Roy, do you want to pick up the story from there?

RG: Sure. Let me rewind the tape on my end all the way back. I was smiling when Curtis told his childhood story about being interested in stars he couldn't see. I have the east coast version of that, looking up in New York City, hounding my parents for a telescope, and then when I finally got one, we went up on the roof and there was nothing to see.

JU: Just a little too much ambient light!

RG: Exactly. We thought we saw a star, but it was a plane.

My career has been in science education, and I was familiar with Curtis' work long before the WorldWide Telescope. When I heard his talk in Chicago, and at some subsequent conferences, I saw that it was addressing two of my passions. One is astronomy, what's out there in the night sky, and having a unified view of it. But also, as somebody who communicates science, I've always been interested in the learning interface. It's not just about using the resource, it's about learning from it.

It was clear there were lots of things that could be done, but it took a long time to see what all of them might be. It was only after I used it for a while that I realized what a great breakthrough it is.

First, there's the seamless experience of the night sky. It's true that all these images are accessible in principle, if you know where to find them, and of course many us in the field do that when we prepare curricula or museum exhibits or planetarium shows.

But when you do that, you see the universe in a disconnected way. Once you go on WorldWide Telescope, it's a different experience. It's like you could look up with perfect vision.

JU: And with complete contextual awareness.

RG: Exactly. And I think we take the night sky for granted. We don't realize how important it used to be. When we have floods in the midwest, we call them disasters. Well, disaster literally means against the stars. Catastrophe is the Greek for against the stars. Romeo and Juliet were the star-crossed lovers. It's all through literature, it's part of common speech.

Then you fast-forward to the modern day, and few of us have even seen the stars, let alone have that relationship to them. For me that's number one about WorldWide Telescope. It's really inviting us to take a long look at the night sky again.

Of course you can do that through the WorldWide Telescope, but then you can also look at the night sky from your back yard, or from a dark location, and have taht dual relationship. So you can both see the night sky and, in WWT, you can explore it.

JU: That's one of my favorite things to do. At night, of course, laptops create their own illumination. My first astronomy program ran under MS-DOS, amazingly enough, and I'd take my laptop from that era out in the backyard and use it as a guide. This is the latest and greatest incarnation of that tradition.

RG: And from the educational point of view, this pays enormous dividends. We've done research here about what students know when they graduate from high school about the night sky, and about the universe in general. From that, we know that the majority of high school students graduate placing the stars inside the orbit of Pluto.

JU: C'mon. Really?

RG: It's true. About 52 percent, and that's based on surveys of thousands of students in 37 states.

There are many reasons for that, but certainly one of them is this lack of good images of the sky. It's hard enough to see a picture of the solar system, let alone its context within the galaxy. That's one beautiful feature of WorldWide Telescope, you get a sense of where things are.

We also know that students think galaxies are closer than stars, because they tell us that stars are just point sources, and no matter what your magnification or telescope they remain points, so they must be very far away. Whereas galaxies, whatever they are, are big, and so they must be closer.

But if you go on WorldWide Telescope, and look at the Andromeda galaxy, the nearest big galaxy to us, and the furthest thing you can see with your naked eye, you get a physical feeling. You see all the stars in our Milky Way that are the veil of stars we look through, and you really get a sense that the Andromeda galaxy is vast and distant.

Another thing that's useful is the ability to look at the universe in different wavelengths of light. We see only the visible light our eyes can see, but that's like listening to one instrument in the orchestra.

You can download a Chandra image, an X-ray image of the sky, but what's that about? You can't really figure it out. Or one of the infrared images. In WorldWide Telescope, you can seamlessly crossfade back and forth between images taken at various wavelengths. So you see what's going on in the visible wavelengths, but you also see what's going on that's emitting these other forms of light. And that's important, because most of the action in the universe happens at wavelengths of light we can't see. The WorldWide Telescope automatically aligns these different images.

JU: Can you say a bit about how that's done?

CW: Yes, it was a challenge to register all these different surveys so you can do that kind of cross-comparison. There are some emerging standards from the National Virtual Observatory and others. The AVM (Astronomy Visualization Metadata) standard provides metatags at high precision within objects, to give exact position and scaling and orientation for images in the sky.

JU: Does that work by reprocessing existing survey data?

CW: Right. Generally the data exists. But for example, when Hubble makes these beautiful press-release images that they send out in color, the metadata is usually lost, because the images have been post-processed in Photoshop and other programs. So a lot of these images that we want to put out there for the public needed to have that metadata reintegrated. And sometimes they're composites of many Hubble images to create a large mosaic. So that composite image needs to have metadata put into it.

I think a source image is about 400 megabytes. So while it's technically in the public domain, they don't release it because it's way too much data for most people to download. They've released low-resolution versions, but we have the full-resolution image of the Crab Nebula, and other things, in WorldWide Telescope, because we can enable people to use the high-resolution images without having to download all of them.

JU: So part of it was going back to sources that were notionally available, but not practically available.

CW: Right.

JU: But part of it is about emerging standardization of how these images are described.

CW: Yes. The whole NVO group is working toward common standards and common protocols for image metadata, so they can all be used by everybody. There's AVM, part of the VAMP project -- Virtual Astronomy Multimedia Project -- leading the charge for annotation of imagery and other media in the context of the sky.

JU: And there are presumably many uses for those annotations.

CW: Yes. So we were one of the first guinea pigs for VAMP, and I think Google joined a bit later.

JU: So let's talk about the authoring aspect. I'm a huge fan of multimedia and audiovisual tools for educational and training purpose, and this is a wonderful example of that.

I was looking at what files get created when you author a slideshow in WorldWide Telescope, and it looks like the output is an XML file with coordinates and descriptions. To me that says two things.

First, it says that WorldWide Telescope presentations can be created using other tools, which is interesting.

It also says that presentations created inside WWT could potentially be played elsewhere, in other environments.

CW: Exactly. The whole idea with the authoring, and these guided experinces, is...let's go back to the educational intention. I spent a lot of years building instructional learning, where you bring in experts to tell you about subjects, in context, but also the self-directed discovery aspect of learning. Then there's a third part I wanted to bring in, constructive learning. There's always a duality between instructive and constructive learning.

JU: What do you mean by constructive?

CW: Basically, learning by doing. Where kids who don't know much about astronomy can take tours from experts, and be taken to unfamiliar places, but then pause those tours and explore on their own. In WorldWide Telescope you can always pause the tour, like stopping a tour bus and getting off to look around. At that point you can right-click to get more information, you can zoom into places that the tour didn't deeply explore because it had to keep moving, you can see other objects that are in the neighborhood.

If you notice down below, in the context menu, as you get to different objects you can not only see that object in different wavelengths, but you can also see other tours that intersect with that object.

The goal is that as you start to see more and more guided tours within this space, if you think of objects as nodes or intersections, there are more and more opportunities to cross over from one tour to another. Eventually we might see a kind of hypermedia web of learning where instead of hyperlinking among words, we're linking among stories and paths and ideas.

JU: And that'll include things I make for myself. If I make a narrative about a part of the sky, then the context surrounding the area I'm interested in will be available when someone else plays back that tour.

CW: Exactly. If you take a tour about stellar evolution, and learn about how stars get formed from nebulae, and you get to the planetary nebula stage, you might say, wow, those are really pretty, what's going on there? Then you might intersect with a tour about different planetary nebulae, where you dive deep into that category. Then you might find a tour just about the Ring Nebula, or the Helix Nebula.

Or conversely you may come across it in a different way. You're browsing the sky and you come across the Ring Nebula or the Helix Nebula, and you may say, what are these things? And you can see other things that relate categorically to them, which would then intersect with explanations of how planetary nebulae fit into the grand scheme of the origin of all the elements.

JU: Roy, what tours are you working on?

RG: We're working on two. One is a tour of black holes, in conjunction with a traveling exhibit the Smithsonian is producing.

Another is a tour of alien solar systems and their exoplanets, as they're known. There are more than 300 stars known to have planets orbiting them. Using a small educational telescope -- we have a network of five of these, they're called micro-observatories -- students in middle school and high school can take their own images and study these exoplanets. They can actually characterize them in a surprising amount of detail. They can figure out how large they are, how far away they are from their stars,

What's more, we're using the tour of exoplanets to get teachers who have never used the micro-observatory telescopes in this particular curriculum. You know, you can make a brochure, and send them some images of the night sky, but there's nothing more exciting than having a tour that gives you a sense of the context, the depth of the sky, where things are.

JU: Do you think there will be a citizen science aspect to this?

RG: Absolutely. I think that's going to be a major use. Astronomy is probably unique among all the sciences, in there are more amateurs than professionals. But this is a third category. You've got the amateurs, you've got the professionals, but now WorldWide Telescope makes possible the blossoming of citizen science. Especially given the flood of images we're going to have in the next few years, more than researchers can ever look at. You'll have images that have never been seen by humans, and that opens up a huge possibility.

JU: So part of it's about getting more eyeballs on this flood of imagery. Potentially another part is citizen-driven analysis of data. I wonder if SkyQuery will become part of the suite, so people can start to ask questions, like how many fast-moving objects are in this part of the sky, which is one of the queries Jim Gray mentioned in his 2002 paper.

RG: Yes, I'd love to see that. There are several ways the public can contribute. One is to look for things, and make serendipitous discoveries. With some of the NASA solar missions, for example, where we have so many images of the sun coming back, you can see comets that were never seen before as they get close to the sun. And ordinary citizens have discovered comets that nobody knew existed.

But to me the most important thing is what you just alluded to: Asking questions that nobody had thought to ask, even the professionals.

So for example, what's the volume of a black hole? How big is it inside? If you go on the web and look at the standard references, you'll find answers all over the place, all at odds with one another.

There are questions that researchers just haven't gotten around to asking, that many of the public will ask, and we don't know what those are yet.

In a way, although there's all this wonderful technology in the WorldWide Telescope, but in a sense it's the modern incarnation of a campfire that you sit around and trade stories. Our organization has telescopes in Australia and Chile and elsewhere, and when I go to those countries, I find that the native cultures have all sat around campfires and developed incredible stories about the night sky.

JU: So how does the collaboration work? I've made a little slideshow, it's stored as a file on my computer, how do I share that?

CW: We're trying to encourage the development of communities. Sky and Telescope is forming one, Astronomy magazine is forming one, Meade -- a telescope maker -- is forming one.

JU: So the unit of sharing is the WWT file which gets created when you make a slideshow. It's a bundle of XML and images of thumbnails and maybe audio if there's voiceover. In a lot of cases, those will want to live out on the web where people can link to them. If I post that file, and somebody clicks on it, and WorldWide Telescope is installed, then it'll launch and play the slideshow when you click on the file?

CW: Yes. I was talking to a storyteller from a local Snoqualmish tribe, a lot of their stories happen to be about the sky. I wanted to try to capture some of those as examples of what you can do.

By the way, if you're playing the tour, you can pause and go into edit mode, and it's all open. You can change destinations, you can drop in your own audio narration, music, text, and images.

JU: So you've forked the thing you've downloaded, and at that point you can...

CW: ... put your own interpretation on it, exactly. It's just like View Source. Except easier, because you don't know have to program in HTML.

JU: Yeah, it's quite straightforward.

Do you think that there's a need -- I suspect that there is -- for some sort of universal player that wouldn't require the full application, and wouldn't even require Windows, but would just be a way for anybody to play these things?

CW: Absolutely. That's a really good idea.

So by the way, I want to highlight one citizen science story for you. It relates to Galaxy Zoo, a website that allows the public to help the Sloan digital sky survey catalog and tag the hundreds of millions of galaxies that were covered in the various data releases. In one case, a teacher from Amsterdam was looking at a galaxy and it looked really blue. She reported that to Galaxy Zoo, and they looked at it, and it was something they'd never seen before. So they retargeted the Very Large Array Radio Telescope to take a look at it. And based on those results, they've now secured Hubble time to study that galaxy in great detail.

That's a case where putting the data out there, letting the public look at it, dividing up the sky, and having that feedback mechanism can really advance science. Because when you think about it, as we start to get telescopes like LSST and PanSTARS and these other large telescopes that will generate many terabytes of imagery and data every night, it's going to be impossible for any one person or group to see what's up there. Image recognition's good, but nothing's as good as the human eye and human brain.

JU: Some guidance on what to look for is really useful. If someone's found something interesting, and that justifies spending the resources to take a closer look, that's beautiful. That's exactly how things ought to work.

CW: Right. And I think a lot of these telescope projects are thinking, how do we make this much data available to people? And how do we make that accessible in a simple way? So they've had conversations with us about how WorldWide Telescope might be able to help.

Also, NASA is very interested in how realtime data feeds from them would enable the public -- at the same time -- to have access to mission data.

JU: You start to think about what's possible, and you quickly realize there's an infinite number of interesting possibilities. It'll be great to see how this plays out over the next few years. Thanks!

Making sense of electronic health records

JonUdell — Thu, 12 Jun 2008 09:22:00 GMT

My guest for this week's Perspectives show is George Hripcsak, professor of biomedical informatics at Columbia and one of six researchers recently funded by Microsoft Research through its Computational Challenges of Genome Wide Association Studies (GWAS) program.

George Hripcsak

JU: For starters, what is a genome wide association study?

GH: A genome wide association study involves scanning markers across the human genome to find genetic variations associated with certain diseases.

JU: Specifically what's being looked for is single-nucleotide markers, right?

GH: Yes. Now our role in this project is the phenotype. We're trying to address the phenotypic computational challenge. Often it's simple. Someone has diabetes or doesn't. Or two people have it, but one gets complications and the other doesn't.

JU: So by phenotype you mean the expression of diabetes, in this case?

GH: Yes. Often you start with a disease, and some number of patients, often very small, but up to several thousand, plus a control group of patients without the disease.

You study the entire genotype, and you look for which sites on the genoome are associated with that disease. Then you look into that site. Now the fact that you may find a certain genetic mutation at that site -- that's not necessarily the cause of the difference between the two sets of patients. The cause may be something near that marker on the genome. So you might sequence that area, looking for other information about what genes are in the area, and so on.

JU: So the computational challenge is one of correlation.

GH: The first step is correlation. But...I'm at the zeroth step. There are other people working on the part I'm describing here. First they'll come up with associations, which is a computational challenge in its own way, because there is a vast number -- a hundred thousand, someday a million -- markers that you're looking at, to see if they're associated with this trait, diabetes or not diabetes.

Then they need to figure out what proteins are coded at the marker, or near the marker. In order to get to that point you need the phenotype too. As long as it's something simple, like patients with or without diabetes, it may seem like that's the easy part of the experiment.

But as time goes on, and genotyping gets easier and cheaper -- and as we learn to handle patient privacy, that's the other thing that limits the study, we have to be careful about how we collect and store these data -- the hard part is going to be collecting the phenotype.

JU: When you say "collecting the phenotype" -- that's clinical observation and description?

GH: Exactly. Imagine that every patient who comes into the hospital is given the option to participate in a trial where, in a secure fashion, their genotype is done, and then their information can be used to discover new things about disease. Some number of patients would agree to that, and then all you have to do is take their blood samples, check the DNA, and genotype it. It's relatively straightforward if you have the money to do it.

Then you have to find out what the phenotype of the patient is. But what questions should you ask? We don't know what diseases we might be studying, or what we might discover. We want to know the whole medical course of this patient: When they've been well, and when they've been sick.

What we have for these patients are their electronic health records. And in the future, with Microsoft HealthVault for example, we have the personal health records. And so the question is, with the patient's permission, can we use those data to come up with a reliable phenotype?

JU: OK. Now I see how this ties into your career history. You've done a lot of work in the area of mining clinical records, using a variety of techniques.

GH: Exactly. So in addition to working on the statistical models to do the genome association, we thought it'd be worth investing in the phenotype part of the problem. We've been working on it for 20 years, and it's harder than most people think.

JU: I wouldn't think it'd be easy, but tell us: What are the challenges unique to mining health records and clinical data?

GH: Two of our collaborators are Rich Smiley and Pamela Flood, both faculty members at Columbia, in anesthesiology. They're studying a specific protein, the beta-2 adrenergic receptor, and they're sampling about 2000 patients. They're just studying two snips, it's not a genome wide association, but they need to collect the phenotype on those 2000 patients. It's prohibitive to have a research nurse accurately gather all the information they need to do their study -- and what they're studying is labor, the length of time you spend in pre-term labor and how much pain you experience, and how that's associated with variations on these two sites. It's an enormous amount of work, and it might not be reliable.

But we do have an electronic health record. For each of these patients, the nurse has painstakingly documented what he or she recorded on the patient. Plus we have monitors, and lab tests, all fed into the electronic health record.

JU: But much of what's there is anecdotal, textual, and narrative, right?

GH: Well, it's a mixture of structured and narrative. So ideally we should be able to generate the phenotype rather than have a person do it. We're trying to do that by computational analysis of the health record.

Of course the health care record is intended for patient care, not for research. Anytime you take an information source intended for one purpose and try to use it for another you have to be careful. It takes a lot of processing and interpretation.

Whether it's structured or narrative data, people use different words to encode things. In a cardiovascular study, is it "chest pain"? Is it "angina"? Is it "coronary artery disease"?

The terminology varies, and often the terms are ambiguous. Someone says a person has diabetes, they probably mean diabetes mellitus, a problem with glucose. But there's a diabetes insipidus which is a completely different disease. All they have in common is that you urinate a lot.

JU: So on the one hand you can try to provide a more structured data collection environment that's aware of these distinctions. And on the other hand you can do a lot of text mining, correlation, and natural language processing.

GH: Yes. But remember, the people who are collecting the data are not interested in your research study. As a nation, we're working on improving our terminology, whether you get there by natural language processing or by having the doctor fill out a template. Either way we want to end up with computable knowledge.

But when the purpose is clinical care, not research, we're always going to wind up with these problems.

Furthermore, there's a reason why we speak in narrative style, and not in templates. It's an efficient means of communication. It may be true that it's best for health care providers -- and for all other human beings -- to speak in narrative language, and to have our systems, as they improve, turn that narrative into something structured.

JU: Using natural language processing to extract structure from narrative is something you've been doing for a long time. What can you say about the progress of the state of that art?

GH: We did a study in 1995 where we had 200 chest x-ray reports, and we had 12 physicians review them. Six were radiologists, the ones who generate the reports, and six were internists, who generally use them. They weren't looking at the images, they were just looking at the reports dictated by the radiologist who did the initial reading.

We wanted to see if we could use natural language processing to say yes or no to a set of questions, like: Is this a report indicative of bacterial pneumonia, or of cancer, or of chronic obstructive pulmonary disease? We had six conditions we were looking for, and we compared the reliability of each doctor's reading to the other eleven, and we compared the computer system's interpretation to all twelve. We found that the computer system was about as accurate as the 12 experts.

JU: And what is that system? There are general NLP frameworks, and also domain-specific ones...

GH: This one is a medical system called MedLEE, which Carol Friedman started building back in 1990 or 91. It went into production use in our hospital in 1995, and we've been using it ever since.

JU: And you've been training it as you use it?

GH: Well, improving it. It's not a data-driven system. So as it makes mistakes, we fix it, but it's not a machine learning system.

JU: It's a language understanding system.

GH: Yes. It uses a semantic grammar, it divides all words into classes, so rather than getting into the details of syntax, like noun phrase and verb, it says, this thing is a body part, this is a disease, this is a procedure, this is a medication, and then it has a grammar that has sequences of these classes. It also has some syntactic parsing to figure out negation and things like that, so it's a blended approach. It was used initially for radiology reports, but now it's used for all of medicine.

It was as accurate as humans at answering simple questions. When you get to complex interpretations, it doesn't do as well, but you're still in a situation where a human can't be expected to read a million chest x-ray reports, or discharge summaries. If we can do things that there just isn't the money for people to do, even if the accuracy is a bit lower, that's still useful.

JU: Given that context, how will this apply to the funded project?

GH: So, I've outlined some challenges. Things are narrative, terminology varies. Another is that data are sometimes wrong. Mistakes can be made in recording information on the chart, and often those are mistakes that the doctor would notice and immediately discount. Or it may be a subtle mistake that isn't important to a human interpreting the case, but could matter for a research trial where you're trying to automatically understand what's in the chart.

Often, there's also missing data. The patient may go for care elsewhere. Or a data value may not have been recorded. Or a test may not have been done. So you don't really have a complete record. If you're doing a clinical trial, you have a lot of money to pay a lot of people to spend a lot of time tracking patients, following up with them, measuring everything that needs to be measured. But if you're just using the combination of electronic health record and personal health record, you have to rely on whatever was collected for that purpose.

JU: It's going to be sparse data, for the foreseeable future.

GH: Exactly. So our challenge is to generate a reliable phenotype from that electronic health record and personal health record. Or, if it's not reliable, to know that there's not enough information in those records to make a determination.

JU: So part of the challenge is to infer what's missing. How can you do that?

GE: Let's say you're trying to study complications of diabetes, and you want to do a genome wide association study on people who've had severe diabetes from the point of view of treating it with insulin, but have had no complications, versus people who have had complications, to see if there's a genetic difference. If you can discover why some people don't have complications, can you develop a drug that mimics that in the other people?

To do that, we want to come up with a phenotype of people who have diabetes severe enough to be treated with insulin, but who don't have complications. And we want to use the electronic health record to identify them. What are the challenges?

Well, what if someone comes here for their diabetes care, because there's an expert in this medical center, but when they have complications, they go to the nearest hospital? My electronic health record doesn't have the data about their complications.

JU: Of course this is the promise, and the holy grail, of federated health records.

GE: Right. But this is just one example of many problems that can come up. When you're trying identify someone who hasn't had complications, you don't know if you're missing the data, or if they're truly without complications.

How can you figure it out? Well, you can use information theoretic methods to figure out, look, I have enough information such that if this person had complications, I'd know it. If this person has a history and a physical by an internist, or three different internists over the course of 10 years, and none of them ever mentioned a complication of diabetes, then odds are this patient doesn't have a complication.

JU: So you're interpreting the negative space?

GE: Exactly. Whereas another patient, who has diabetes, and disappears for 5 years, and then comes in and has a complete blood count but not a glucose, and then has some minor dermatological procedure, and then disappears for 5 years, and is here now -- I have no reason to think that person doesn't have diabetes complications. All I know is that he or she came in to have a mole removed. I have no information about diabetic complications, for example an opthamologic complication.

JU: Electronic health records are moving into the mainstream. You mentioned Microsoft HealthVault, Google Health was just announced. Most people have yet to encounter these things in their routine interaction with the health care system. I presume that in five years, many will have.

I think a lot of people have the notion that the information that's being collected will be of value, not only clinically but also to research. Your point is: No, not necessarily. So my question is, if you were the czar of electronic health information, what would you like to see happen in order to merge those two goals?

GE: I'd start with a caution. There's a knee-jerk reaction to say that we need to have doctors document more accurately, and more completely. But the problem is that you end up with a big structured template.

What I envision is an intelligent record that produces a summary for clinicians that they can read, correct, and then write their note which is a succinct summary of their thinking.

Now that doesn't answer your question, which was: How does that then get used for research? But I think that to the degree we make documentation efficient in serving health care, I think it'll also be more accurate for the sake of research.

One thing that can go wrong, for example, is that if you're filling out a record for the sake of billing, you'll have an incentive to use diagnosis codes that optimize billing. Does that then reflect clinical accuracy? And would that then be useful for research?

The important thing is to be grounded in the clinical truth. Put health care first, and then use new computational methods to extract accurate information.

JU: So clinical truth is what the doctor said, in the doctor's own language. Of course there's a lot of shared convention around the terminology.

GE: They learn in medical school, and throughout their professional lives, what to document. Things aren't always called the same, but the nation is working on health care standards in various ways, both for transferring information between systems and for coming up with common vocabularies.

JU: So although many of us would assume that those vocabulary terms need to be fields in a template, you're saying that's not the first and best strategy. You'd like to see that language just used naturally, as doctors speak their narratives, and then we'll harvest what we need out of that.

Do you think natural language processing will get us there?

GE: It's not perfect. We achieved expert-level performance on a simple task. We have less than expert performance -- but not bad performance -- on the more complex task.

JU: How has the system improved since its introduction in the 1990s?

GE: What we've done is expand our breadth. Back then we were doing mainly radiology reports, and now we cover most of medicine. I don't know that the accuracy got better, though.

Modern natural language processing systems often depend on machine learning, and don't have deep linguistic knowledge.

JU: Well, there are both breeds.

GE: In medicine we're seeing more emphasis on statistics than on linguistics, but we believe the right answer is a combination of the two. In our case we've tried some statistical systems too, but our semantic system seems to outperform them.

If you have a specific question, and that's the only one you need to answer, a statistical system is probably the more efficient way to go. What we do is parse the entire report, and spit out everything we can figure out from it.

In the 1995 study our goal was to answer six questions, but the system actually parsed the whole report, said everything it found, and then in those things it said we found which were indicators of pneumonia.

There are various techniques that you can use that do pretty well on a single question, but that don't do well if you give them an entire history and physical, and say, tell me everything there is to know about the patient. That's what MedLEE is good at.

Systems should make it easy for people to express what they need to express -- in this case, the clinical truth. If it turns out that a super-efficient template model works best, then that's great. It's an empirical study. People will experiment over time, and see what works.

JU: You've also mentioned the compromise approach: summarize, then present for approval or correction.

GH: Yes, but clinicians don't want to stop and correct. So we need to work on presenting the structured format that's useful enough to them to justify that effort.

JU: It's a perennial and vexing problem. In some ways, maybe, one of the grand computational challenges. At the interface between the data collector and the human being, the person is always going to regard the collector as an impediment.

So, when does your project start?

GH: We've already started. For that Rich Smiley and Pamela Flood study in pre-term labor, we're already taking data out of the electronic health record for them to do their associations.

It's nice to have a concrete problem to work on. Over the summer, what we're working on is a generic framework. So, how does the next person and the next person do this? And then we'll be working on putting together a pipeline of tools. You'll still need a person there to process the data, but it won't involve reading every chart.

JU: Well this sounds hopeful. Thanks!

GH: Thank you, Jon.

How Mercy Corps syncs databases in Afghanistan

JonUdell — Thu, 29 May 2008 13:34:00 GMT

My guests for this week's Perspectives show are Barbara Willett and Nigel Snoad. Barbara works for Mercy Corps in Afghanistan, as the design, monitoring, and evaluation manager for a number of agricultural development programs. Nigel Snoad is a lead capabilities researcher for Microsoft Humanitarian Systems. Together they've pioneered the use of FeedSync as a way to synchronize data collection and reporting in an environment where Internet connectivity is spotty, and where lightweight, two-way synchronization is essential.

Nigel Snoad and Barbara Willett

JU: Barbara, we want to discuss the database synchronization system that you've partnered with Nigel to develop, as part of the Mercy Corps work in Afghanistan.

BW: I'm the design, monitoring and evaluation manager here in Afghanistan. I arrived last year in March, about the same time we had a consultant in doing a general review. He also brought another consultant who'd worked with Mercy Corp on technical issues, including the development of databases.

JU: And can you explain what, in this context, is being designed, monitored, and evaluated? What are the programs you're supported, and what do those programs do?

BW: In Afghanistan, almost all our programs revolve around agricultural development. We have a number of programs funded by USDA, the British Government, the European Commission, all with the same goal of improving the livelihoods of the Afghan people.

JU: Is your microfinance program among those?

BW: Yes, we have a microfinance program, but it's one of our older ones, and it's self-sustaining now, no longer administered directly by our monitoring and evaluation system.

JU: What are some examples of programs that are?

BW: The ABAD [Agro-Business and Agriculture Development] program is where all this started. It supports business development, and technical capacity training for farmers.

There's also a lot of work in animal health, redeveloping and reestablishing veterinary field units, and training veterinaries, para-veterinaries, and female livestock workers.

JU: So your management challenge, relative to these programs, is what?

BW: It's developing tools that are applicable to multiple programs doing the same kinds of things. Everybody's involved in agricultural development, and interested in measuring improvement in sales and production. It's a challenge to collect that data and share it -- both operational data and impact data.

JU: So the field workers are in various locations around the country, with intermittent Internet access?

BW: Right. And in these circumstances, we want to standardize how we collect, synchronize, share, and report on this information.

JU: You had a pre-existing system based on Microsoft Access, as I understand it, and there were problems synchronizing those databases to your central office.

BW: Initially we didn't have Access, actually. When I arrived there wasn't any centralized system at all. Everything was Excel-based, sharing spreadsheets month-to-month from this region to that region. So we started the Access system, then later we realized it wasn't really working out because of the Internet problems, and because the process was bulky and cumbersome.

JU: Nigel, that's where you come in, right?

NS: Yes. Our humanitarian team was in Afghanistan looking to talk with people, do a bit of show and tell, and mainly get feedback about what people would like, and what they really need. And then use that to iterate what we were doing, and look for partnerships to do pilots.

With Mercy Corps we said, here's what we've got, here's what we're thinking, does that make sense to you?

Mercy Corps was great for that, because they were fairly well advanced in their thinking about how they were using their Access solution, and the architecture of what they wanted to do was quite clear.

JU: So from your perspective, Barbara, what was the outcome? Did it look just like what you had before, except that the synchronization problems were magically solved?

BW: Yes. I just wanted things to share, I wanted to know that we all had the same database, and somehow, whether it's every week or every month or every minute, the information just has to connect.

On paper it looked like we could do that with Access replication, but when we realized the problems that was causing, we realized that this technology Microsoft had been talking about -- which seemed maybe a little beyond our needs -- might actually solve the problems that we had. They wanted to try it, we wanted to try it, so it seemed to dovetail well.

And yes, it made happen in reality what I'd wanted to happen on paper. I was willing to accept weekly or even monthly if that's what it took, but now it happens every 10 minutes.

JU: Is this a situation where the updates that flow in from various locations tend not to conflict with one another?

BW: Yes, conflict resolution hasn't yet been much of an issue. Our biggest issues have been in our own database development, because the database itself is still evolving. So each time that changes, it affects the job mapping and FeedSync.

NS: The conflict resolution stuff is in there, and I think it'll become increasingly important as the size of the data grows, and as the activity from all the endpoints grows.

But we were quite deliberate about trying this in one place, and seeing if from Mercy Corps' perspective it worked out the way the Microsoft team had envisioned. It really was a tight spiral between developing new ideas and technologies, and also proving them and using them.

It was great to be able to do that in a real environment, but also a fairly controlled one, which was our first pilot in Kunduz. But then, you took it all over the country, damn you. [Laughs]

BW: [Laughs]

JU: There was a preexisting synchronization system that was found wanting. What was that, why didn't it work, and why is the FeedSync solution working?

NS: The solution Mercy Corps was originally piloting was based on Access and Access replicas. Which is a great technology, but in Afghanistan they were struggling with an unreliable Internet connection, and those replicas weren't working well. There was a lot of data to send, and there was a peer-to-peer VPN that would work OK sometimes, but flake out sometimes, mainly due to the Internet connection.

And in some cases, there was no Internet connection at all. So you have to send something by courier, be it a file on disk or on a memory stick.

JU: With FeedSync it's the same in the no-Internet case, you still have to sneaker-net the file.

NS: Absolutely. But with Access, when you export to a file, that's a one-way transfer. And there's all kinds of data you want to get back. Corrections to data, if there's a problem. (That's where conflicts can arise.) Then there are the reference lists and the lookups -- names of provinces, names of villages, names of staff people who are attending training sessions. All these things have to flow back to the edge, and be kept consistent.

JU: The point being that FeedSync isn't just lightweight, and more resilient to poor connectivity, but also that it's two-way.

NS: It's a two-way technology, and you've got different versions of the same thing, not one version that you're trying to somehow import and export and merge.

JU: Barbara, is this two-way aspect evident to you as a user of the system?

BW: Absolutely. At first I didn't understand a lot of the terminology, and the discussions and explanations. I kept hearing the word lightweight, and I didn't really understand what that meant.

But when I compared the Access replication, which basically takes the entire database and replicates it to another place -- which takes a long time, and then the Internet cuts out and you've corrupted the whole structure -- now instead of that you're sending just pieces of information. If it doesn't work right now, it'll work in a half hour, it just keeps trying, and it's completely lightweight and easy in that sense

And definitely the two-way street. We were still very much developing things, and even if it were perfectly developed there are still changes that have to happen from our side. As Nigel said: staff lists. People enter training records and they have to apply them to names of staff, but we get new staff people all the time. We have to continually update the names from our side so they have an appropriate list to choose from.

NS: One of the things we thought about when we were building the job manager, which is the piece that runs the FeedSync on people's desktops, is that it's an application that just sits there. You build a bunch of jobs, and a job takes a data source and syncs it with another data source.

In the case of Mercy Corps, that means take a table from an Access database and sync it with a feed on a website that acts as a relay. That's a job, and you have one of those for each table in the database. Of course referential integrity is something you can try and manage, and there's some support for that.

The other piece is that we can run a sync to a file. It takes the table in the database and syncs to a feed, in this case an RSS feed, on a file source. If the memory stick is plugged in, and you've got things set up right, it just works. The user doesn't have to worry whether it's being exported to the right place, or about what the file is called. And similarly for the Internet case.

JU: At this point, is the master database what's up on the server at Live Labs?

NS: Well the master source is really the database in Kabul, but yeah, the replicas are being also managed on feedsync.mslivelabs.com, where the plug is that anyone can go and set up a feed and a synchronization endpoint. All the databases sync to that, and then Kabul syncs to that and gets the data back down. And vice versa.

I should point out that FeedSync used to be called SSE [Simple Sharing Extensions], and this started back then. The first users of SSE in anger, if you like, were Mercy Corps in Afghanistan, which was exciting. But we had a lot to learn. Now they're moving to FeedSync. What that means is a slightly different version of the specification, a different service on the website, and a new version of tools I just gave to Faheem a day and a half ago -- he's the technical manager for Barbara's group. This latest version of the tools is the one that I hope we're releasing publicly in a couple of weeks.

JU: Barbara, I'm sure that going forward you'd like to see a better way to do schema evolution, so that the changes to the database structure can be part of this seamless synchronization.

BW: Yes, that would be ideal.

NS: That's a long conversation...

JU: Yes. Here's something else. I know that Mercy Corp works in parts of the world where connectivity is basically SMS more than Internet, or maybe exclusively SMS. That seems like something that FeedSync could be adapted for. Have you thought about that?

BW: Sure, some of our other offices have been using SMS as a way to share bits of information. We haven't found the need to do that yet, because we don't have that many people far out in the field who would need to enter data. In other programs where there visiting sites and schools all over the country, then yes.

NS: We've already built an SMS adapter for this, it's in testing at the moment. And it does exactly what you suggest. Rather than sending the data over the Internet, it breaks it up into SMS packets. There can be a lot of packets, of course, and it can be expensive. But we were talking to a different NGO in Afghanistan, which operates in very insecure areas where they don't have Internet at all. There, they are very interested in doing something similar to what Barbara is doing, but using SMS. First, because in some of these areas the security is so bad they can't even be seen to be carrying data. Second, if it takes six hours to drive somewhere, the cost of a bunch of 1-cent SMS messages is a lot cheaper than the cost of the petrol involved.

We've tried this, and it works quite nicely. I'm excited about that for the future. But to be honest, there are plenty of issues just keeping the Mercy Corps solution running. I wouldn't want to make you believe that this has been a dream installation where everything worked perfectly off the ground. Just the other day we ran into a problem where the feeds wouldn't sync.

BW: Sorry about that!

NS: Hey, no problem. It's now documented and it's part of the FAQ that'll go out when we release the tools.

JU: Barbara, how does Mercy Corps envision making use of the open toolkit which will be one of the outcomes of this project?

BW: There's been a cautious, wait-and-see approach from the beginning. Our IT has been a little like, oh, I don't know, this maybe could be interesting, let's see how it works. But now that people are understanding more, and seeing that this is not really a pilot any more, there's starting to be interest in ways we can be sharing information regionally, across offices, and how can other countries make use of the same technology. It's working its way into our lexicon.

JU: If there weren't the Internet problems you've had in Afghanistan, would there still be reasons to do things this way?

BW: Yes, I think there are other benefits. It's much closer to realtime, for one thing.

JU: The lightweight, near-realtime aspect is appealing even when there's enough bandwidth to do more heavyweight replication?

BW: Yes. And also, for us, going from one Access database to another identical database is one thing. But some of our initial discussions were about sharing across platforms that may not be identical, but use common variables. Across regions or countries, we all need to report certain pieces of information, but we collect it in different ways, and store it in different ways. If there's a system where we can upload it similarly, that would be a huge benefit.

JU: Great point. You could define a neutral common ground for data exchange.

BW: Exactly.

NS: For Mercy Corp, there's a whole pile of options they should consider when doing sync. FeedSync isn't the be-all and end-all, it's got some particular things it seems to be good for, but that last point about interoperability is really important.

From the start, we've been interested in how to use this to link up disparate systems. Sometimes that might be an Excel spreadsheet to a database. But also different organizations or, in Mercy Corps' case, different countries where they work run slightly different database designs.

That's where I think the real strength of the system will lie. We've got a lot of work to do thinking about how to make that better.

One of the things we're really concerned about is that, if Mercy Corps or another group wanted to roll this out, there would be the support to do that. I don't think we've got that perfect yet by any means, but we showed what we're doing to a bunch of other NGOs, and afterward a number of them were interested in taking this kind of tool -- be it FeedSync or some other -- and using it for their programs.

The best example might be where you've got a whole pile of agencies implementing a program. In Afghanistan there's a thing called the National Stability Program, and it's run all over the country. All the reporting happens in a standard format, but every organization has its own way of managing the process. The challenges are to integrate the data, and pass back success and lessons learned. The big NGOs have their own systems, almost all in Access, all with some of the same schemas because the ministry says this is how you will report, but no way to aggregate all that nicely.

JU: This makes good sense. Thanks!

NS: Yeah, thanks Barbara. And let me know how Faheem's getting along. There may be some issues, but I hope everything's OK.

BW: Yeah, it's good, thanks so much.