Comment Feed for Singularity Revisited (Going Deep on Channel 9)

Re: Singularity Revisited

Mike Beaton — Sat, 06 Sep 2008 15:48:44 GMT

Is this video still available?

It does not seem to play or is available for download

Thanks

Re: Singularity Revisited

rysus — Wed, 12 Mar 2008 02:36:25 GMT

BTW,where can i download it?:s

Re: Singularity Revisited

skyde — Fri, 30 Jun 2006 12:37:02 GMT

It is often said that c# is more type-safe than c++.
Does the type-safety is provided by the c# to msil conversion or the msil to x86 conversion.
Because if it is the msil to x86 conversion you can ensure the typesafety of c++ code that have been converted to msil.

To write high performance code in java you should write a librairi in C++ and call it from java and when c++ is not enough you write assembler in your c++ code.
This is because the java Jit-compiler dont optimize as much as c++ compiler, also when doinng ahead of time compiling.
you said you where using assembler in c# code.
So my question is, does the bartok ahead of time msil compiler do as much optimization as c++ compiler.

So i understand that you choose c# over c++ because it is a more high-level language giving more simpler code to read.
But you said that writing a device driver in visual basic can cause some problem because of is dynamic feature.
What do you mean .. (Bartok compiler can't convert msil code produced by VisualBasic to x86) or (the type-safety is hard to ensure in dynamic feature)
Is it possible to create a Ruby to msil compiler that keep the dynamic feature of ruby and can be converted to native x86 code like you do for c#.
This way you would have a highly object-oriented dynamic type-safe language

Re: Singularity Revisited

Calum — Tue, 18 Apr 2006 11:49:18 GMT

kentcb wrote:
Thanks for such an interesting video, and kudos to the singularity team for their outstanding effort so far (and their awesome ability to explain their work). Looking forward to part III.

One thing I've been thinking a lot about lately is how useful it might be to have a CPU that understands IL. In other words, a specific CLR version is integrated directly into hardware instead of being a software abstraction.

I'm not sure whether this is even feasible but it seems to me that it would have a number of benefits:

Security. The security measures currently enforced by a software layer would instead be enforced at the lowest level - in the CPU itself.

Performance. I'm no expert in hardware but I imagine that performance could be greatly improved (not that it's bad now).

Essentially, unsafe code would be non-existant. This is kind of what you're trying to enforce with singularity but at an even lower level. I can imagine such a CPU being really useful in, for example, mobile phones.

Anyway, just ranting . . . thanks again,
Kent

I believe that the Singularity guys are thinking of something along these lines, at least on some level. They mention the concept of "typed assembly language", which implies that there is essentially type information associated with the native code that the MSIL for programs is compiled to. This type information essentially allows one to have "safe" assembly language. I believe there is currently some work into incorporating this into hardware, but even if that does not happen, superimposing type information (and I'm not sure if this is possible in an effective way, but bear with me) onto the native (x86 or whatever) code for the system would allow the OS to check the code it is provided for safety, even though the code is precompiled. Since the code in a SIP cannot be changed, the type information could be stripped and the strong guarantee of safety provided by the check would stay intact.

Hope I'm not reiterating anything here...

I do love this project, when I stumbled upon it at first it was like something I had been thinking about myself, except with far, far more thought put into it. Good stuff.

Re: Singularity Revisited

Rockweiler — Sun, 05 Feb 2006 20:37:18 GMT

In the beginning of the video it is said that you use the Bartok Compiler to compile the Runtime System into a specialize form of an executable that also contains a runtime to run the Runtime for the rest of the system. How does that all work?

Re: Singularity Revisited

Steve411 — Mon, 09 Jan 2006 11:24:51 GMT

I bet its UI sucks more than linux's.

- Steve

Re: Singularity Revisited

mluschas — Tue, 27 Dec 2005 19:02:55 GMT

Looking at this OS, it sortta seems like its attempting to go back to the old days with the mainframes. I wasnt alive then (im 19 and a CS student), but i've heard alot.
   I believe that IBM used to have something similar. Remember the old BASIC that was on a chip in the computer. You could buy the computer, but without that chip, it would do nothing. If using singularity as an OS, you were to redesign the hardware, you could do something of the same thing.
   Now, dont get me wrong, I like developing and I think it would be wrong to charge anyone that wants to develop on a machine a lot of money. But its an idea.
   Redesign the BIOS of a machine (probally have to have more than 512 bytes for the BIOS), to do the normal POST, but have it have a CLR in it. This would enable you then to write all of your code in a managed (SAFE) language. This would eliminate the need for any unsafe code in either the kernel or the bootloader. Yes you would need unsafe code to exist in the heart of the machine for the same reasons you do now, direct hardware access. However, if you have the BIOS able to allocate pages in this safe runtime enviroment, you would not need to have the kernel have unsafe code; same goes for the HAL.
   Then by doing this you would be able to sell a complete packaged solution, increasing profit as well as end user stability.
   Also, I like the idea of a transactional filesystem and page manager. I was actually thinking of creating that type of system. I was working with that idea when i came across the videos here. I thought it was interesting that I had thought up something and you guys were already trying it.
   Is there any chance of the singularity source being released anytime soon ? Or even Bartok (not so much the source, but the binaries for it). I do beleive there are many open ideas for it, just remember: Dont be greedy if you do a package solution, IBM failed and so will you if you charge too high of a price for development licenses.
The development licenses that I am speaking of, are not for the compiler, or the tools. You said all code is either "Trusted" or "Verified". Taking that to the next level of "Is this code signed" if so allow it to run, if its not, display a message and dont allow execution. Thats the type of license I'm refering to.
Oh btw, you guys wouldnt happen to have a job opening for a 19 yr old CS student, whos been programming since 5 would ya ? lol. Figured I'd atleast ask.

Re: Singularity Revisited

Manuel Fahndrich — Fri, 09 Dec 2005 22:37:19 GMT

pacowski wrote:
I thought the exchange heap was very neat, but I was wondering how a particular buffer can be marked to be within the exchange heap. Is there a particular method for marking buffers to be used within a channel? There was a description of the contract that is used to communicate across a channel with specific message formats. I was wondering if in the next video someone could address how the exchange heap and channels interract with two communicating processes. More specifically, is there anything special that needs to be done from a programming stand point or is it all taken care of in the message sending code?

Thank you.

Since data in the exchange heap is completely separate from GC data and has different types, it is also allocated with a variation:

byte[] in ExHeap buf = new[ExHeap] byte[512];

would allocate a 512 byte buffer in the ExHeap. Given a message

message Data(byte[] in ExHeap buf);

and a channel endpoint e supporting this message, you would use

e.SendData(buf);

to transfer the buffer. Note that buffers are not associated with a particular channel. e.g., the receiver of the Data message above can turn around and send the buffer to whoever it wants to; or alternatively, it could free the buffer explicitly via:

delete buf;

The type system/static analysis makes sure that programs don't touch data in the ExHeap they don't own (e.g., after send or delete).

Re: Singularity Revisited

dhi — Wed, 07 Dec 2005 18:13:46 GMT

In the second video, I saw that you are doing message passing over channels and run everything in its own SIP.
I did a lot of programming for µnOS which is an OO OS written in C++ including a GC.
It also uses channels for message passing (without message contracts) just like QNX does.
So if every driver runs in it's own SIP, can I write a SIP by myself sending messages to the NIC SIP
to tell it to send raw ethernet packets to the net?
Or has the TCP/IP SIP exclusive rights to communicate to the NIC SIP?
In µnOS we dicided to compile device drivers to DLLs.
So the network service, which is a seperate process in user mode creates an instance of the device driver object(s)
in its own process space and uses a well-defined interface to access these driver object.
Only the network service has a channel over which the other applications can create and use network connections.

How about modularity and the extension of a certain system service?
Let's say your TCP/IP SIP currently supports IP, ICMP, ARP, TCP and UDP.
How would I extend it to support SCTP as well?
Can I write an extension module (e. g. in form of a C# class) that is loaded by the TCP/IP SIP at startup to support SCTP?
Or do I have to write the extension in form of another SIP?

I also saw that you are doing process creation, channel management and security inside the microkernel.
Did you thaught of implementing a SIP in user mode for doing stuff like that?
E. g. the QNX process manager does some of these things in user mode.

At all, you did a great job with Singularity!
I hope Microsoft stays tuned with it!

Re: Singularity Revisited

kentcb — Wed, 07 Dec 2005 11:01:17 GMT

Thanks for such an interesting video, and kudos to the singularity team for their outstanding effort so far (and their awesome ability to explain their work). Looking forward to part III.

One thing I've been thinking a lot about lately is how useful it might be to have a CPU that understands IL. In other words, a specific CLR version is integrated directly into hardware instead of being a software abstraction.

I'm not sure whether this is even feasible but it seems to me that it would have a number of benefits:

Security. The security measures currently enforced by a software layer would instead be enforced at the lowest level - in the CPU itself.
Performance. I'm no expert in hardware but I imagine that performance could be greatly improved (not that it's bad now).

Essentially, unsafe code would be non-existant. This is kind of what you're trying to enforce with singularity but at an even lower level. I can imagine such a CPU being really useful in, for example, mobile phones.

Anyway, just ranting . . . thanks again,
Kent

Re: Singularity Revisited

William Stacey — Tue, 06 Dec 2005 04:32:21 GMT

"They did mention that dynamically loading code is a no-no in Singularity... MSH scripted cmdlets are interpreted (I think) so those would work fine, but I bet the compiled ones wouldn't work, unless I'm misunderstanding what they said."

But MSH is not dynamic from the OS SIP perspective. MSH.exe and its dlls are compiled and verified like any SIP. It happens to be able to take a script and run it. But, I think, this is not the same as the process loading dynamic code at runtime. This is the script driving verified logic in the MSH exe. So the script could not drive anything that was already not verified. This is really not much different then your exe loading a config file and running different logic based on what is in the xml file - similar idea I think. Running sql scripts in a SQL SIP is the same kind of idea.

Compiling cmdlets may be a different story as msh needs to dynamically load those. Not sure what the story there would be, but they would somehow need to support that kind of thing. Maybe things like MSH SIP would get special handling or something or there would be some way to verify the cmdlet dll so it could be trusted and loaded. Not sure, but a good question and I would be interested in this as well.
--wjs

Re: Singularity Revisited

reinux — Tue, 06 Dec 2005 04:14:58 GMT

staceyw wrote:
"no, you would have to rewrite at least the win32 console host.. and the runspace probably too, etc..."
Actually, I would be very suprized if they don't have this working already or at least working in some way for their research purposes.

They did mention that dynamically loading code is a no-no in Singularity... MSH scripted cmdlets are interpreted (I think) so those would work fine, but I bet the compiled ones wouldn't work, unless I'm misunderstanding what they said.

Re: Singularity Revisited

William Stacey — Tue, 06 Dec 2005 03:48:11 GMT

"no, you would have to rewrite at least the win32 console host.. and the runspace probably too, etc..."

Actually MSH would be perfect as msh is written in managed code. Naturally they would need to write a SIP managed console and port the msh console calls to the new console abstraction. Actually, I would be very suprized if they don't have this working already or at least working in some way for their research purposes.

--wjs

Re: Singularity Revisited

pacowski — Tue, 06 Dec 2005 03:35:04 GMT

I thought the exchange heap was very neat, but I was wondering how a particular buffer can be marked to be within the exchange heap. Is there a particular method for marking buffers to be used within a channel? There was a description of the contract that is used to communicate across a channel with specific message formats. I was wondering if in the next video someone could address how the exchange heap and channels interract with two communicating processes. More specifically, is there anything special that needs to be done from a programming stand point or is it all taken care of in the message sending code?

Thank you.

Re: Singularity Revisited

jachymko — Tue, 06 Dec 2005 03:19:20 GMT

reinux wrote:
Nice!

I'm curious... would Monad run on this, or would cmdlets cause a problem?

no, you would have to rewrite at least the win32 console host.. and the runspace probably too, etc...

Re: Singularity Revisited

reinux — Tue, 06 Dec 2005 02:34:33 GMT

Man, I'd so love to see my web server running on Singularity :D

Heck, it'd even be worth Spec#ing the whole thing if I could see that happen.

Re: Singularity Revisited

reinux — Tue, 06 Dec 2005 01:30:30 GMT

Nice!

I'm curious... would Monad run on this, or would cmdlets cause a problem?

Re: Singularity Revisited

kidzi — Mon, 05 Dec 2005 18:01:29 GMT

scarz wrote:
Thanks for doing these interviews guys, I am really enjoying them. As a CS student I haven’t gotten to learn much about where we are going with future OS’s, only where we've been. This work is extremely intriguing and makes me confident I'm in the right field (I mean come on - an OS research project has me excited). I can't wait to see part 3 of the video. My hopes for this project would be to push it as the successor to Vista. Cut the cord with legacy systems once and for all. With the new Intel and AMD virtualization technology coming out we can do some awesome stuff with virtualization, so why not run all our legacy apps (dos, 95, 98, XP, etc.) in a Virtual PC under Singularity. It seems to me that the transitioning from Vista to something as drastic as Singularity would not be as hard as in the past with this kind of technology. Most of the important applications would be rewritten in safe code within a few years, and anything not worth rewriting can just be run under the virtual machine. Anyways I know that’s not the point of the project I just hope people see the value in the system your building. Keep it up guys; I’ll be following your work.

Since its still a reserch project, you probably have a few years before its production ready :) Even then, I'd think if it does eventually make it somewhere, it would first need to start in an area where the software can be controlled: Embedded, xbox, and maybe later pocket software. After it gains traction and vs2010 provides building things natively, maybe we'll see something. But I find it unlikely it would be on the desktop within the immediate future with MS's committment to backwards support without having a whole native API programming model on top of singularity.

Re: Singularity Revisited

William Stacey — Mon, 05 Dec 2005 16:44:38 GMT

Thanks again Charles. Was waiting for this one. Can't wait for #3. Maybe a quaterly installment could be put on the agenda. Maybe you could do a whole show on Transactional Memory locks and invariants, and compare/contrast with traditional lock symantics.

I do have a question. You keep asking "why threads" as if you know of some alternative method. TMK, a process always has at least one thread to get anything done. So if you just use process like old unix systems did (before PThreads), your back to forking and issues with that. Threads are great from a logic flow perspective, but the locks for shared resources become the issue. This transactional invariant memory deal would seem to really help here. As well as any kind of compile time and/or run time tools that can help verify correctness of syncronization in our programs.

It would be cool to have a simple SQL server SIP (naturally managed) with all managed types as proof of concept. Layered ontop of the sql api is the TSQL block and another CLR block that would allow any new SQL type language to directly call down to SQL spi without overhead of the ADO layer - so it would be a peer to the TSQL block. Then you could have a MSH Provider block as another peer. So you could "CD" into any instance of SQL and CD into databases and tables to list and manage table data.

Love it! Cheers.
--
William Stacey [MVP]

Re: Singularity Revisited

scarz — Sun, 04 Dec 2005 06:37:34 GMT

Thanks for doing these interviews guys, I am really enjoying them. As a CS student I haven’t gotten to learn much about where we are going with future OS’s, only where we've been. This work is extremely intriguing and makes me confident I'm in the right field (I mean come on - an OS research project has me excited). I can't wait to see part 3 of the video. My hopes for this project would be to push it as the successor to Vista. Cut the cord with legacy systems once and for all. With the new Intel and AMD virtualization technology coming out we can do some awesome stuff with virtualization, so why not run all our legacy apps (dos, 95, 98, XP, etc.) in a Virtual PC under Singularity. It seems to me that the transitioning from Vista to something as drastic as Singularity would not be as hard as in the past with this kind of technology. Most of the important applications would be rewritten in safe code within a few years, and anything not worth rewriting can just be run under the virtual machine. Anyways I know that’s not the point of the project I just hope people see the value in the system your building. Keep it up guys; I’ll be following your work.

Re: Singularity Revisited

Tonatiúh — Sat, 03 Dec 2005 11:50:31 GMT

David Tarditi wrote:
I'm the person on the far right, wearing the dark long-sleeve shirt. Manuel is sitting to the left, then Galen, then Jim.

Thank you a lot for such a rapid reply to a very superficial and personal question, derived only from my inability to hear English... Every time I've started the video, I've lost your names at the very presentation of your selves.

David Tarditi wrote:
[...] You might want to use a real-time collector for streaming video and a GC with higher throughput for a compute-intensive application. [...]

From the above quoted fragment, I understand that there is a mechanism at the implementation side as for tunning the timer of a GC... If it is true, it is there one of the discrete kind (i.e.: a fixed set of types amongst whose the implementor of the process targeted to run inside a SIP may choose), and/or it is there one of the opposite (i.e.: a value between a given range, which could be passed, or settled before calling, to the appropriate GC's method for timing)... Furthermore, I wonder if it is there, in addiition to one or both of the aforementioned types of mechanisms, one of an automatic type; as I think the GC timer could be tied to a relevant SIP runtime feature, which shall adjust cybernetically the former (i.e.: dynamically, as in a file's downloaded amount counter, by using as the GC's cybernetic feedback feature, the frecuency of the memory chunks left unused by the process at the SIP, been collected by the GC itself).

Makes this any sense?

Tonatiúh

Re: Singularity Revisited

David Tarditi — Sat, 03 Dec 2005 07:12:21 GMT

I'm the person on the far right, wearing the dark long-sleeve shirt. Manuel is sitting to the left, then Galen, then Jim.

Re: Singularity Revisited

Tonatiúh — Sat, 03 Dec 2005 06:50:31 GMT

Thank you David... I got blushy when read your name and suddenly became aware that you are one of the Singularity Project researchers... I've enjoyed this interview and the papers I have studied untill now.

Care tell me who of the four researchers in the video are you?... Perhaps you want spot me your shirt's color... That will suffice.

Tonatiúh

Re: Singularity Revisited

Tonatiúh — Sat, 03 Dec 2005 06:40:05 GMT

piersh wrote:
very interesting stuff.

i have a question, though: what was the reason for choosing to have a separate GC for each SIP? it would seem to me that in a fully trusted environment like this you could have a single GC for the whole system, integrated with the page manager, would eliminate the need for the exchange heap. memory in one SIP would automatically be 'available' to another process, the consistency of the channel being checked by the verifiable 'handing-off' mechanism you described.

Please, don't take me as any kind of literated guy about this matter... But as far as I've understood from the documents available at the Singularity Project site, Each SIP has its own GC because the goal of making Singularity an highly reliable OS... That is, and again, as far as I understood: When a process is runing in a SIP (aka Software Isolated Process), it is totally isolated in its own space and time frames; by design definition, it can't share its space or even its timming amongst the kernel or other SIPs... So if it becomes unstable or corrupted in any manner, or there is the need to shoot down it because any kind of malfunctioning, there will not be left more unrecovered memory than the still not released by the affected process, and that, only for the time the kernel gets aware of and empties the SIP... OS corruption and unstability arises mainly beacuse the resonance of a chain effect of non-released memory.

I hope this may help.

Tonatiúh

EDIT: Resonance of a chain effect of non-released memory is only posible in an OS running with a single GC... I wonder after observing the way Windows XP SP2 (under 256 KB RAM) behaves in serendipitous scenarios prone for unstability, as many of those I am quite used to set up in my daily work on design, programming and research (keeping several heavy applications opened up to the end of the day and not few times thru a three or four days span, without any problems); if such version wasn't a worthwile feed for the design requirements of the Singularity Project, as for to the point I've reached on the Spec# specification papers, there are many underlaying paradigms whose are, conceptually, very similar.

Re: Singularity Revisited

David Tarditi — Sat, 03 Dec 2005 06:31:00 GMT

Having a separate GC per SIP lets you choose the an appropriate garbage collector for each application. For example, some GCs are real-time (they have low pause times), whereas other GCs emphasize throughput, at the expense of higher pause times. You might want to use a real-time collector for streaming video and a GC with higher throughput for a compute-intensive application.

In addition, it allows each process to be separately garbage collected , increasing the scalability and robustness of the system. For example, you could run multiple garbage collections at the same time. You could also avoid a denial of service attack where someone is allocating lots of data, causing other processes to slow down because one garbage collector for the whole system can't keep up.

Finally, it simplifies tracking resource usage and reclaiming resources when a process ends.