Comments

In a database of the future, I'd like to be able to declare my taxonomy, interval (including temporal) properties of objects and their attributes and functions to derive new truths about objects based on the truths obtained so far. Then I might want to specify how much of those computational graphs I want to store precomputed (which would allow me to have OLTP, data warehouses, cubes and whatever funky terms are currently used to descripbe various stages of truth discovery/derivation). Are there any developments in those directions?

(I guess inclusion of MDM services into 2008 R2 may seem to hit that the answer is "soft of").

To the subject of the code running closer to the data, this was obvious for quite a while now that shipping bits over network to have a copy in application memory and process it using (more often than not) less sophisticated algorithms than those already present at the point of storage doesn't make much sense. Database is a computational engine, so it should have a powerful language to create computational expressions. Unfortunately SQL in general is not quite that, and T-SQL in particular isn't either.

Having F# inside database engine would be nice. Add "pure mode" for querying (which should be side effects free) and use imperative mode to produce side effects (data modifications).

Yeah, that's a hack and is a bad one too. Why not to make "done adding" semantics explicit at least?

Then instead of add one before, remove one after there will be explicit call to ce.NoMoreCounterIncrements()

It doesn't guarantee that programmer won't forget to add it into the code, but that's one change to one code line as opposed to two (and it's a better "pattern" than the alternative).

Plus, if this primitive constructor accepted nested lambdas, then AddCounter() and NoMoreCounterIncrements() could be hidden from the programmer all together. Just declare what you want to spawn and how many of those in a nested lambda and happily wait for completion.

On the subject of strict or linient evaluation.

It seems that an adanced enough run-time can and should use both, based on the accumulated "knowledge" (stats) about workloads being executed.

Expectation that something can be strictly evaluated in false in absolute sense, because each and every CPU instruction and/or memory read/write may fail because of faulty hardware. Yet, it can be statistically true. If hardware is somehow known to be 99.something% reliable, such assumption can be made safely (in statistical sense), otherwise nothing can be computed or done ever.

(I believe that proponents of strict evaluation are stuck because they base their reasoning on incorrect assumptions without explicitly stating what those assumptions are, which is a known issue that plagued physics for centuries, and most likely still does)

The same must apply to the algorithms as well. If algorithm is known to be predictable on a given workload (either statistically or by devine intervention of the mister human), it's OK to evaluate is strictly. If there is no prior knowledge, lazy evaluation is the way to go and please gather execution stats upon exit so it can be reused in the future evaluations/executions. And if it does not exit in the requested amount of time - abandon (preferrably kill first) the execution and and black list it (till the end of time or the next devine intervention).

From 10000 feet it looks like a nice logical schema with a feed back loop, which is statistically a necessity for each and every successful eco system (observe the nature).

Right, and old ideas are just facades on even older ideas and so on recursively till the big bang

In the end it will be about believes, whether one believe this or that language being "new" or not. Attempts to define "pure novelty" would end up nowhere.

Ah, "Once zero it cannot be incremented" - that's what was missing. Then it's all cool and dandy

I guess that the counter is NULL originally (if not set to any positive integer).

Actually, it's quite obvious. C#, Java, C++, C are sugar coated assembler. Reasoning about assembler, even sugar coated is a lost cause. Making those languages into something that can be reasoned about at compile and especially run-time would be practically impossible because of long hairy legacy that those languages carry around.

In order to run a program on a parallel hardware, run-time would have to reason about side effects to come up with some strategy to partition computational graph into work loads that have minimal interactions between each other.

If many core processors will have cores of different capabilities (which seems to be the case), run-time reasoning and JIT will be a necessity.

It seems like none of the existing imperative languages would survive transition to parallel era. Of course run-times are still be written in something that is sugar coated assembly, yet for general-purpose programming completely new languages would be required.

Declarative and richly typed presumably.

Also to the point of run-time reasoning and code generation, to provide fault tolerance computational graph might need to be re-evaluated if a computation node returns exceptional value or goes into non-termination state. That in theory would allow automatic remediation for run-away queries in databases and handling of non-responding services in the cloud (as well as mutating hardware - failed or hot plugged general and special purpose CPUs, failed or hot plugged memory and so on).

It probably will take another 10 to 20 years to get it right, but it looks like that's where things are going.

It seems to me that in order to have a race condition ce.AddCount() should be inside lambda in the last code sample.

As long as ce.AddCount() is inside foreach loop on the main thread there won't be any race conditions, so adding "one for the host" before and then taking it out after the loop is kind of pointless.

Also, guys, _No Sound_ in SL3 on XP SP3 _again_.

Cheers,

Seva.

Luke, for the sake of correctness, Debug.Writeline call was added to the MulTask() method, which was executed after the sequential and PFor multiplication. So, PFor loop ran without any blocking on screen output, and it was slower than sequential presumably because of all the extra work associated with priming up parallel execution environment.

If 90% of the input for my app on any given day happens to be small, it's better to process those 90% sequentially and use parallel execution only when appropriate, but for that it would be nice to know, where (approximately) is that cutting point.

I can run tests and collect some stats on what overhead of firing up parallel execution is, but assuming that this work might have been already done while developing the parallel framework, it would be preferrable for me to look at the stats collected by the PF development team than spend time and efforts myself.

Cheers,

Seva. 

Very nice.

One thing though, when matrix multiplication was executed on a small data set, sequential was actually faster than ParallelFor (@20:48). Are there any insights on estimating an overhead of setting up parallel execution machinery, so, application could attempt guessing whether to process data set sequentially or in-parallel (assuming that application knows the size of the data set)?

XPSP3, old Compaq Pressario, SL3, no sound on this particular video. Eric Meijer's functional programming part 3 is fine though.