Surendra Verma: Vista Transactional File System

deedubb wrote:CCR, I thought I was going to hear some good music...
extremely interesting stuff! and the video was really fun to watch, too! reading through the paper i'm wondering how the port based ccr solution compares to the active/future objects idiom Herb Sutter is thinking about for c++0x/c++cli. did the ccr team
and the guys around active<> objectives share some thoughts on the topic or are these two completely independent efforts?
again a worthy installment of going deep! nice work, Charles and Scoble!
best regards,
martin
Where are the bits to download? or is it not released to the public yet?
I'm excited about this.. release it
Great video!
We need the effective concurrent environments!
The free lunch is almost over, even though indium antimonide may realize 6GHz CPU.
Intel aims for faster, cooler transistors
http://news.com.com/2100-1008_3-5984635.html
iStation wrote:
We need the effective concurrent environments!![]()
The free lunch is almost over, even though indium antimonide may realize 6GHz CPU.
Comment removed at user's request.
Zeo wrote:And I loved how robert stayed quiet and did an awesome job while filming
scobleizer wrote:See, I'm trying to not do my stupid laugh so much.
CSP has a single-entry, single-exit paradigm, requiring proper nesting of concurrent code in fork/join pairs.
The pi-calculus and join-calculus work which inspired the CCR has no such requirement.
Short answer: Yap we take care of it. And yes, if you want to, you can specify in more detail how the scheduling will occur.
Long answer:
The CCR takes care of automatically distributing work across processors. All you do is associate receivers with ports, constrainted under some arbiter (choice, one-time/re-issue receive, interleave etc) and we will generate work items when messages arrive,
and schedule them across task queues with some number of OS threads in the back end picking them apart. Usually we use 2 threads per processor but you can set that number.
We can achieve near perfect load balancing across heterogenous tasks and keep all your CPUs hapilly busy. A CCR program just needs one instance of a dispatcher process, and if you set the number of threads to use to 0, if you want us to choose how many threads
per processor to utilize in the back end.
Dispatcher creation:
DispatcherPort MyDispatcher =
Dispatcher.Create(0,"FriendlyName");
// now we can use it to associate a piece of code
// that will execute in one of the CPUs, when message arrives
MyDispatcher.post(!MyMainPort.with(delegate(DoWork W)
{
// do work here. If you get N DoWork messages, we will
// run as many versions of this code concurrently as your CPUs!
}));
// you can also set the dispatcher as field in a base class
class MyFirstCcrProgram : CcrServiceBase
{
public MyFirstCcrProgram(DispatcherPort Dispatcher):
base (Dispatcher) {}
void Init()
{
// because we associated a dispatcher
// when we were constructed, we can just
// use the base member activate()
activate(!MyMainPort.with(delegate(DoWork W)
{
// do work here. If you get N DoWork messages,
// we will run as many versions of this code
// concurrently as your CPUs!
}));
}
}
If you want greater control you can create multiple dispatchers, and then each item you want to activate a receiver, use a particular receivers. This is very powerfull and unlike global threadpool schemes: By being able to associate a dispatcher per receiver,
you can interoperate very well with Single-Threaded-Apartment components, like COM, but your CCR code does not look any different!
We will give you at least one simple "adaptor", used for UI, where you post messages on ports, but using its own dispatcher, with just one thread, it does Invoke in the back, on your behalf, toa winform.
All this helps you go the next level, if you wish, in controlling execution, scheduling etc and interoperating with the OS.
Developing dynamic analyses for message passing programs is of course important (e.g. run-time debugging). However, I feel that for message passing programs that is not going to be enough. We will also need the help of more powerful static analyses to help catch problems like deadlocks. There is some interesting work involving contracts and behavioral types which looks like a promising approach. For exampe: https://www.research.microsoft.com/behave/index.htm
waltal wrote:A really great video about some fantastic technology!
Anyone got a better suggestion for sources on doing it "old style"?
waltal wrote:I have an immediate need for more threading info and a future need for this concurrancy and coordination runtime.
Staceyw, you will actually be able to service alot more than the number of OS threads in 32 bit systems, if you use asynchronous socket APIs, even from C#. The underlying OS will use overlapped I/O and will not block a thread per connection. Then, the total
number of simultaneous requests you can service will not be bound by OS threads. The key ofcourse is to use asynchronous APIs all the way through, so you dont consume a CCR or C# threadpool worker. YOur actuall throughput might not increase btw, but your system
will be able to service hundreds of thousands of packets, independent of each other so appear more responsive.
See one of the examples of iterators in this thread or the wiki.We actually have implemented a CCR based transport that does this (but cant post the code, yet , and it uses the C# asynchronous socket calls underneath. We just had to wrap them with CCR ports
and simple processes so we did not have to deal with Begin/End.
The CCR, using iterators allows you to get "blocking" behavior so your code does not look like the spaghetti of continuations we find ourselves writing again and again. You can actually encode a while loop , processing packets but with no blocking! One of the
examples in the wiki shows a for loop using iterators to encode asyncronous but logically sequentially req/rsp operations. It keeps it all in one routine.
So really what we are doing for you, is allow you to use the asynchronous OS APIs, through a much more readable front end, to get the logical behavior of threads, but without the overhead and limitations.
When threads become super lightweight one day( if ever), then the CCR can use alot more of them and get rid of iterators, but its syntax will not really change. The CCR can have blocking arbiters (join, choice etc)
hope this helps.
Any idea when we can play with some bits? TIA
I have to admit this looks amazing and is exactly what we need on my current project. I currently have a distributed server system that accesses data locally and on other machines, but I'm having difficulty adding parallelism to the system. I believe this would allow me to easily add this if I can massage the dispatcher to not only load balance across processors but across machines.
Georgio do you see this as being already built in or at least a workable implementation on my part? I already have a running system (in .net 2.0) capable of handling thousands of requests a second, and I believe I can easily shoehorn this into it. We are still months away from going live so hopefully the timing will be workable on my part to use this.
Hi! I have been playing with the CCR and I think it takes away some of the pain of working with concurrency, which is very good!
However I'm a little confused of how should one program with the CCR
For instance, is it correct to send ports to ITasks and post from within them (like in the following example)?
static void HandlerPorts(Port<int> pInts)
{
pInts.Post(42);
}
static void Main(string[] args)
{
using (Dispatcher dispatcher = new Dispatcher(0, "CcrDispatcher"))
{
DispatcherQueue queue = dispatcher.AddQueue("MainQueue");
Port<int> pInts = new Port<int>();
Port<Port<int>> pPorts = new Port<Port<int>>();
Arbiter.Activate(queue, Arbiter.Receive<int>(true, pInts,
delegate(int value){
Console.Out.WriteLine(value);
}
));
for(int i=0; i<2; i++){
Arbiter.Activate(queue, Arbiter.Receive<Port<int>>(false, pPorts, HandlerPorts));
pPorts.Post(pInts);
}
System.Threading.Thread.Sleep(1000);
}
}
This seems to work as I want, printing twice the value 42. But am I messing with the CLR in any way I shouldn't?
Also, is the API defined anywhere? I've heard that there are some quite neat ways of using the CLR (operator overloading, like the | for choice).
Thanks and keep up the good work!
Nuno
Hi again!
Can the Ports, Queues and Dispatchers be used as "globals" (class variables, for instance) from within the tasks?
Is it possible to extend a Port with a different strategy (like a LIFO or a priority queue instead of a FIFO)?
How are the objects passed by the ports to the tasks? By reference or copied?
Thanks!
Great! Those features really do simplify the implementation of our app.
What I meant in the the first question was if the Ports are threadsafe: for instance, can I post to them from different CCR Tasks, eventually running simultaneously, without having to worry about race conditions in the Port's object, right?
sumothecat wrote:Very enjoyable video! I have a simplistic question -- I thought a "runtime" was something that loaded code and managed its execution (e.g. CLR, Java runtime). This is a type library -- what makes it a "runtime"? Thanks!
Hi everyone! You are welcome to try my commit to the community and play with my framework samples at http://plugins.codeplex.com/">http://plugins.codeplex.com
Sincerely yours,
hack2root