Singularity Revisited
- Posted: Dec 01, 2005 at 4:10 PM
- 135,255 Views
- 48 Comments
Download
How do I download the videos?
- To download, right click the file type you would like and pick “Save target as…” or “Save link as…”
Why should I download videos from Channel9?
- It's an easy way to save the videos you like locally.
- You can save the videos in order to watch them offline.
- If all you want is to hear the audio, you can download the MP3!
Which version should I choose?
- If you want to view the video on your PC, Xbox or Media Center, download the High Quality WMV file (this is the highest quality version we have available).
- If you'd like a lower bitrate version, to reduce the download time or cost, then choose the Medium Quality WMV file.
- If you have a Zune, WP7, iPhone, iPad, or iPod device, choose the low or medium MP4 file.
- If you just want to hear the audio of the video, choose the MP3 file.
Right click “Save as…”
- WMV (WMV Video)
Charles Torre again sits down with some of the people behind MSR's Singularity research OS. This time, we drill down into the architecture of Singularity and discuss design decisions, usage of safe code, Channels, SIPs, etc. We even manage to get Galen
Hunt, the OS Guy, up to the white board to map out some of Singularity's architecture.
In addition to the usual Singularity suspects, Jim Larus and Galen Hunt, Manuel Fahndrich and David Tarditi, both Senior Researchers involved in the Singularity project, join in the conversation and provide excellent insights into the languages and tools used to make Singularity.
Enjoy.
In addition to the usual Singularity suspects, Jim Larus and Galen Hunt, Manuel Fahndrich and David Tarditi, both Senior Researchers involved in the Singularity project, join in the conversation and provide excellent insights into the languages and tools used to make Singularity.
Enjoy.
Comments Closed
Comments have been closed since this content was published more than 30 days ago, but if you'd like to continue the conversation,
please create a new thread in our Forums,
or
Contact Us and let us know.
Follow the Discussion
Any chance you guys are going to open up a little bit of the source to the rest of the world? Or are you planning on making this into something?
C
You guys rock! No.1
I get WHY they don't want dynamic loading; but it is a shame, dynamic loading is amazingly powerful.. For one example plugins, how would you implement an extension mechanism for your programs without the ability to dynamically load?
Another thing; there is lots of discussion about compiler checks and with no JIT running what is from stopping someone from writing raw x86 (or whatever it is compiled to) and using pointers to gain admin rights on the system? In fact if you can't trust the running code what is stopping anyone from gaining greater privileges on the system or over-writing the kernel with their own one?
Or do you assume local code is safe to run on the system?
Edit: All of this was answered later in the video... Good questions Charles, thanks.
Thank you again.
Tonatiúh
Keep in mind that at the end of the video we did talk about a part 3
I hear you, the Singularity folks are doing some of the most innovative OS work in the world (though they'd never agree with that). The possible uses of Singularity, and what's learned from it, will take security and reliability to a new level some day...
Much thanks to the Singularity folks who took an hour out of their schedule to talk to Channel 9.
Glad your questions were answered in the video, Manip!
C
Wow, an hour of awesome content and there's still so much more to talk about!
Any ideas when you're going to film the next installment?
A few comments:
The file system run for benchmarks--was it FAT/FAT32/NTFS?
Closed processes: no dynamic code loading?! This seems like a weakness from the developer's perspective. E.g., does this prohibit the ability to load plug-ins? DLLs? Configuration-defined type loading? (Seems like a weak-ness...)
On choosing threads over event-driven architecture:
"You lose the structure of the code if you write it as a bunch of event handlers. You know, each handler makes sense, but you don’t see the big picture. You don't see the forest for the trees. And moreover there's a lot of experience in, you know, windowing code, for instance, that says that this is extremely difficult code for people to maintain, because the first person who wrote it may have had a clear picture in their mind but it goes away… and the next person coming along has no idea what it is."
Amen. But shouldn't good design and design artifacts (comments, documents, drawings, etc.) take care of some of that? Is that too much to ask of developers?ExHeap: very interesting.
Any chance Bartok and stand-alone compiled-to-native-code apps will happen on XP/Vista in the next five years. (Gratuitous, feel-good question.
os research is just plain fascinating...my favorite CS class
NGEN ?
Don't look at the camera!
I don't know why, but I just can't stand it.
edit: ah good, they've stopped
edit2: this is an excellent video
Wow!!!... Niners like sharing my rants and disappointments with them... From some days a go up to now, I was feeling quite guilty because the way I have engaged in some discussions... Indeed I was pity about my self, looking how my awareness have spoted in front of my eyes the faulty manner I have behaved in my last posts... Furtermore, I have thought that the last "Suspension idea from Minh have been implemented and applied to me"... So I comforted my heart telling to my self: "Well, at least you have not been banned; so you may get a lot of profit reading every one's posts and following those whorthwile links many Niners are so kind to share here"... I know for sure, I shouldn't endure to lost those.
Thank you Charles
Thank you Singularity Team
Thank you Eagle
Thank you Niners... I beg pardon if any body have been offended by me, specially tou you: Leighsword, mot256 and Cornelius Ellsonpeter
Tonatiúh
Great video !!!
Thanks to Singularity team, there is still some hope for managed code. Even though we have to wait a few years for a true managed -> native compiler.
Thank you all for the info.
I always can't help it but think of who might be interested in using singularity (or some deviation) in their own domain.. does a system structured like singularity lend itself to applications that are mission critical? i'm especially thinking of very specialized devices in avionics, healthcare etc; systems with a very special purpose that have a very limited set of hardware devices they' are running on.
i suppose the mix of fully verifiable code on the one hand and the rest of the code being unsafe but trusted might have a great appeal to many people in the need of specialized mission critical systems. has there been any interest so far? maybe the ADA standardizing organizations considering c# in the future?
anyways, great video and i'm really looking forward to seeing part 3!
With the new detail on Channels and the ExHeap, I can imagine that a owner-draw GUI might have some preformance issues running on this. With ExHeap buffers (as mentioned durring the whiteboard when talking about how disk chunks would be passed) changing ownership when passed as a message, you would have to have a double-buffer thing going on, to make sure the video system had a screen buffer it owned whenever it had a vsync.
I'd love to play around with that kind of thing -- any ideas from the Singularity Team [when/if] they might release [binaries/code] to the outside world and let us do some [research/play] with it?
It's not the same, you're still dependent on the presence of the framework.
I can't think of any reason why you would be banned. Has anybody been truly banned from this site anyway?
I've never been offended by your posts. Although I have thought about taking out a window or two with a piece of flying furniture.
Wow. Great stuff.
Bring on #3!
i have a question, though: what was the reason for choosing to have a separate GC for each SIP? it would seem to me that in a fully trusted environment like this you could have a single GC for the whole system, integrated with the page manager, would eliminate the need for the exchange heap. memory in one SIP would automatically be 'available' to another process, the consistency of the channel being checked by the verifiable 'handing-off' mechanism you described.
(dupe)
I haven't looked at this thread from yesterday, so I thank you quite later because the kindiness of you response... Really, I have gone scared when you have said: "peace out"... Nice to meet you time to time at least.
As for those windows... Please, throw to me only one... That capable of enabling me to stand stared behind... Looking the road ahead, far away over the horizon... Anyhow, if it is an stone caming from your hand, be sure I will happily recive it in the middle of my chest over my heart.
Tonatiúh
In addition, it allows each process to be separately garbage collected , increasing the scalability and robustness of the system. For example, you could run multiple garbage collections at the same time. You could also avoid a denial of service attack where someone is allocating lots of data, causing other processes to slow down because one garbage collector for the whole system can't keep up.
Finally, it simplifies tracking resource usage and reclaiming resources when a process ends.
Please, don't take me as any kind of literated guy about this matter... But as far as I've understood from the documents available at the Singularity Project site, Each SIP has its own GC because the goal of making Singularity an highly reliable OS... That is, and again, as far as I understood: When a process is runing in a SIP (aka Software Isolated Process), it is totally isolated in its own space and time frames; by design definition, it can't share its space or even its timming amongst the kernel or other SIPs... So if it becomes unstable or corrupted in any manner, or there is the need to shoot down it because any kind of malfunctioning, there will not be left more unrecovered memory than the still not released by the affected process, and that, only for the time the kernel gets aware of and empties the SIP... OS corruption and unstability arises mainly beacuse the resonance of a chain effect of non-released memory.
I hope this may help.
Tonatiúh
EDIT: Resonance of a chain effect of non-released memory is only posible in an OS running with a single GC... I wonder after observing the way Windows XP SP2 (under 256 KB RAM) behaves in serendipitous scenarios prone for unstability, as many of those I am quite used to set up in my daily work on design, programming and research (keeping several heavy applications opened up to the end of the day and not few times thru a three or four days span, without any problems); if such version wasn't a worthwile feed for the design requirements of the Singularity Project, as for to the point I've reached on the Spec# specification papers, there are many underlaying paradigms whose are, conceptually, very similar.
Care tell me who of the four researchers in the video are you?... Perhaps you want spot me your shirt's color... That will suffice.
Tonatiúh
I'm the person on the far right, wearing the dark long-sleeve shirt. Manuel is sitting to the left, then Galen, then Jim.
Thank you a lot for such a rapid reply to a very superficial and personal question, derived only from my inability to hear English... Every time I've started the video, I've lost your names at the very presentation of your selves.
From the above quoted fragment, I understand that there is a mechanism at the implementation side as for tunning the timer of a GC... If it is true, it is there one of the discrete kind (i.e.: a fixed set of types amongst whose the implementor of the process targeted to run inside a SIP may choose), and/or it is there one of the opposite (i.e.: a value between a given range, which could be passed, or settled before calling, to the appropriate GC's method for timing)... Furthermore, I wonder if it is there, in addiition to one or both of the aforementioned types of mechanisms, one of an automatic type; as I think the GC timer could be tied to a relevant SIP runtime feature, which shall adjust cybernetically the former (i.e.: dynamically, as in a file's downloaded amount counter, by using as the GC's cybernetic feedback feature, the frecuency of the memory chunks left unused by the process at the SIP, been collected by the GC itself).
Makes this any sense?
Tonatiúh
Thanks again Charles. Was waiting for this one. Can't wait for #3. Maybe a quaterly installment could be put on the agenda. Maybe you could do a whole show on Transactional Memory locks and invariants, and compare/contrast with traditional lock symantics.
I do have a question. You keep asking "why threads" as if you know of some alternative method. TMK, a process always has at least one thread to get anything done. So if you just use process like old unix systems did (before PThreads), your back to forking and issues with that. Threads are great from a logic flow perspective, but the locks for shared resources become the issue. This transactional invariant memory deal would seem to really help here. As well as any kind of compile time and/or run time tools that can help verify correctness of syncronization in our programs.
It would be cool to have a simple SQL server SIP (naturally managed) with all managed types as proof of concept. Layered ontop of the sql api is the TSQL block and another CLR block that would allow any new SQL type language to directly call down to SQL spi without overhead of the ADO layer - so it would be a peer to the TSQL block. Then you could have a MSH Provider block as another peer. So you could "CD" into any instance of SQL and CD into databases and tables to list and manage table data.
Love it! Cheers.
--
William Stacey [MVP]
Since its still a reserch project, you probably have a few years before its production ready
Nice!
I'm curious... would Monad run on this, or would cmdlets cause a problem?
Man, I'd so love to see my web server running on Singularity
Heck, it'd even be worth Spec#ing the whole thing if I could see that happen.
no, you would have to rewrite at least the win32 console host.. and the runspace probably too, etc...
Thank you.
Actually MSH would be perfect as msh is written in managed code. Naturally they would need to write a SIP managed console and port the msh console calls to the new console abstraction. Actually, I would be very suprized if they don't have this working already or at least working in some way for their research purposes.
--wjs
They did mention that dynamically loading code is a no-no in Singularity... MSH scripted cmdlets are interpreted (I think) so those would work fine, but I bet the compiled ones wouldn't work, unless I'm misunderstanding what they said.
But MSH is not dynamic from the OS SIP perspective. MSH.exe and its dlls are compiled and verified like any SIP. It happens to be able to take a script and run it. But, I think, this is not the same as the process loading dynamic code at runtime. This is the script driving verified logic in the MSH exe. So the script could not drive anything that was already not verified. This is really not much different then your exe loading a config file and running different logic based on what is in the xml file - similar idea I think. Running sql scripts in a SQL SIP is the same kind of idea.
Compiling cmdlets may be a different story as msh needs to dynamically load those. Not sure what the story there would be, but they would somehow need to support that kind of thing. Maybe things like MSH SIP would get special handling or something or there would be some way to verify the cmdlet dll so it could be trusted and loaded. Not sure, but a good question and I would be interested in this as well.
--wjs
Thanks for such an interesting video, and kudos to the singularity team for their outstanding effort so far (and their awesome ability to explain their work). Looking forward to part III.
One thing I've been thinking a lot about lately is how useful it might be to have a CPU that understands IL. In other words, a specific CLR version is integrated directly into hardware instead of being a software abstraction.
I'm not sure whether this is even feasible but it seems to me that it would have a number of benefits:
Essentially, unsafe code would be non-existant. This is kind of what you're trying to enforce with singularity but at an even lower level. I can imagine such a CPU being really useful in, for example, mobile phones.
Anyway, just ranting . . . thanks again,
Kent
In the second video, I saw that you are doing message passing over channels and run everything in its own SIP.
I did a lot of programming for µnOS which is an OO OS written in C++ including a GC.
It also uses channels for message passing (without message contracts) just like QNX does.
So if every driver runs in it's own SIP, can I write a SIP by myself sending messages to the NIC SIP
to tell it to send raw ethernet packets to the net?
Or has the TCP/IP SIP exclusive rights to communicate to the NIC SIP?
In µnOS we dicided to compile device drivers to DLLs.
So the network service, which is a seperate process in user mode creates an instance of the device driver object(s)
in its own process space and uses a well-defined interface to access these driver object.
Only the network service has a channel over which the other applications can create and use network connections.
How about modularity and the extension of a certain system service?
Let's say your TCP/IP SIP currently supports IP, ICMP, ARP, TCP and UDP.
How would I extend it to support SCTP as well?
Can I write an extension module (e. g. in form of a C# class) that is loaded by the TCP/IP SIP at startup to support SCTP?
Or do I have to write the extension in form of another SIP?
I also saw that you are doing process creation, channel management and security inside the microkernel.
Did you thaught of implementing a SIP in user mode for doing stuff like that?
E. g. the QNX process manager does some of these things in user mode.
At all, you did a great job with Singularity!
I hope Microsoft stays tuned with it!
Since data in the exchange heap is completely separate from GC data and has different types, it is also allocated with a variation:
byte[] in ExHeap buf = new[ExHeap] byte[512];
would allocate a 512 byte buffer in the ExHeap. Given a message
message Data(byte[] in ExHeap buf);
and a channel endpoint e supporting this message, you would use
e.SendData(buf);
to transfer the buffer. Note that buffers are not associated with a particular channel. e.g., the receiver of the Data message above can turn around and send the buffer to whoever it wants to; or alternatively, it could free the buffer explicitly via:
delete buf;
The type system/static analysis makes sure that programs don't touch data in the ExHeap they don't own (e.g., after send or delete).
I believe that IBM used to have something similar. Remember the old BASIC that was on a chip in the computer. You could buy the computer, but without that chip, it would do nothing. If using singularity as an OS, you were to redesign the hardware, you could do something of the same thing.
Now, dont get me wrong, I like developing and I think it would be wrong to charge anyone that wants to develop on a machine a lot of money. But its an idea.
Redesign the BIOS of a machine (probally have to have more than 512 bytes for the BIOS), to do the normal POST, but have it have a CLR in it. This would enable you then to write all of your code in a managed (SAFE) language. This would eliminate the need for any unsafe code in either the kernel or the bootloader. Yes you would need unsafe code to exist in the heart of the machine for the same reasons you do now, direct hardware access. However, if you have the BIOS able to allocate pages in this safe runtime enviroment, you would not need to have the kernel have unsafe code; same goes for the HAL.
Then by doing this you would be able to sell a complete packaged solution, increasing profit as well as end user stability.
Also, I like the idea of a transactional filesystem and page manager. I was actually thinking of creating that type of system. I was working with that idea when i came across the videos here. I thought it was interesting that I had thought up something and you guys were already trying it.
Is there any chance of the singularity source being released anytime soon ? Or even Bartok (not so much the source, but the binaries for it). I do beleive there are many open ideas for it, just remember: Dont be greedy if you do a package solution, IBM failed and so will you if you charge too high of a price for development licenses.
The development licenses that I am speaking of, are not for the compiler, or the tools. You said all code is either "Trusted" or "Verified". Taking that to the next level of "Is this code signed" if so allow it to run, if its not, display a message and dont allow execution. Thats the type of license I'm refering to.
Oh btw, you guys wouldnt happen to have a job opening for a 19 yr old CS student, whos been programming since 5 would ya ? lol. Figured I'd atleast ask.
- Steve
I believe that the Singularity guys are thinking of something along these lines, at least on some level. They mention the concept of "typed assembly language", which implies that there is essentially type information associated with the native code that the MSIL for programs is compiled to. This type information essentially allows one to have "safe" assembly language. I believe there is currently some work into incorporating this into hardware, but even if that does not happen, superimposing type information (and I'm not sure if this is possible in an effective way, but bear with me) onto the native (x86 or whatever) code for the system would allow the OS to check the code it is provided for safety, even though the code is precompiled. Since the code in a SIP cannot be changed, the type information could be stripped and the strong guarantee of safety provided by the check would stay intact.
Hope I'm not reiterating anything here...
I do love this project, when I stumbled upon it at first it was like something I had been thinking about myself, except with far, far more thought put into it. Good stuff.
It is often said that c# is more type-safe than c++.
Does the type-safety is provided by the c# to msil conversion or the msil to x86 conversion.
Because if it is the msil to x86 conversion you can ensure the typesafety of c++ code that have been converted to msil.
To write high performance code in java you should write a librairi in C++ and call it from java and when c++ is not enough you write assembler in your c++ code.
This is because the java Jit-compiler dont optimize as much as c++ compiler, also when doinng ahead of time compiling.
you said you where using assembler in c# code.
So my question is, does the bartok ahead of time msil compiler do as much optimization as c++ compiler.
So i understand that you choose c# over c++ because it is a more high-level language giving more simpler code to read.
But you said that writing a device driver in visual basic can cause some problem because of is dynamic feature.
What do you mean .. (Bartok compiler can't convert msil code produced by VisualBasic to x86) or (the type-safety is hard to ensure in dynamic feature)
Is it possible to create a Ruby to msil compiler that keep the dynamic feature of ruby and can be converted to native x86 code like you do for c#.
This way you would have a highly object-oriented dynamic type-safe language
It does not seem to play or is available for download
Thanks
Remove this comment
Remove this thread
close