How would you do this in .NET?

I have currently worked around this issue trivially by limiting the data size but I'd like to hear if you have better ideas than I do:

Lets say you have a simple local app which uses LINQ to do things with data. I don't want to use SQL server or other "heavy" solution, I just want to load the data into the process and then do queries.

Now, I want to edit and add LINQ queries which work with the data, without having to reload the data every time I change query. I understand VS11 has improved edit and continue when dealing with lambdas but I read that it still does not allow editing lambdas. I haven't tried if this could be worked around somehow by eg. creating new methods during e&c with the lambda queries or something like that.

Ideas (assume that my queries use/need all the data):

1) Keep data in a running process, then somehow rebuild/recompile the linq query code in a dll which is (re)loaded into that process after every modification of the query code.

2) Keep data in another process and the newly compiled/edited linq queries/lambdas are transferred to that process somehow instead of directly loading the newly compiled code.

3) Make the data loading much faster. Currently I use protobuf-net to deserialize the objects that LINQ is used to query. It would be much faster if the parts of the process memory containing the static dataset would be lightly(fast) compressed and saved (and loaded from) to disk in a way that allows efficient use of windows file buffering (load speed 2 GB/s on warm runs). This would probably require deep knowledge of how CLR works and might not be compatible across different CLR's.

4) build some sort of "linqpad" /query builder into the app with the data with IntelliSense and all. This would be nice for an end user app but I want something simple that works with console apps too and TBH I want to edit the code more than just the queries without reloading the data all the time.

But I'm looking for a way to do this in least lines of code possible and least amount of change to switch from existing simple code to this solution without having to change existing linq queries in the code or throw away the protobuf-net (de)serialization code.

A server process that 'owns' the pre-loaded data and a separate client process that runs the LINQ queries against them -- use remoting or WFC from the client to request the data from the server?

Would depend on how large the data is and the speed of the remote serialisation.

Herbie

EDIT : I've never tried sending delegates or closures through RPC, would that work (client creates delegate containing LINQ query to server and server executes and returns results)?

Why not SQLCE4.0 ? It is lite and you can adhoc whatever you like.

LINQPad supports custom data access drivers. Maybe the only piece of it that you have to build is the part that loads the data into memory and allows LINQPad to connect to it.

12 hours ago, androidi wrote

*snip*

1) Keep data in a running process, then somehow rebuild/recompile the linq query code in a dll which is (re)loaded into that process after every modification of the query code.

*snip*

What do you mean by rebuild/recompile here? Do you mean to rebuild/recompile from Visual Studio?

2 hours ago, kettch wrote

LINQPad supports custom data access drivers. Maybe the only piece of it that you have to build is the part that loads the data into memory and allows LINQPad to connect to it.

LINQ Pad !  it rocks and if you spend some time with it you might find out how to capture a linq expression tree and then serialize that to save it and then later re-load it.

i bet you can just store the expressions in strings or trees and do what it does to evaluate and run them.