Measure .Net object size

I'm storing a lot of objects in a List<T>. The number of objects can be very, very large, and is unknown beforehand. I also want to limit the memory usage of my application. Essentially the final goal is to sort the items.

What I want to do is this: when the total in memory size of the List<T> exceeds a certain size, I want to sort what I have so far, save them to disk, and continue with an external sorting algorithm. However, in order to do that, I need to know how much memory all the objects in the List<T> are using. Is that even possible in .Net? How would you solve this problem?

Do you know what the T you are going to be storing in your List<T> is. Can an object of Type T be of variable size or is it fixed?

KevinB said:

Do you know what the T you are going to be storing in your List<T> is. Can an object of Type T be of variable size or is it fixed?

Could be anything. It's part of a library, users of the code can create their own record types, so I don't know anything at all about the records a priori.

This can't really be done.  http://social.msdn.microsoft.com/forums/en-US/clr/thread/b871dee4-6eb5-4dca-be79-b9589a79f5e9/

wkempf said:

This can't really be done.  http://social.msdn.microsoft.com/forums/en-US/clr/thread/b871dee4-6eb5-4dca-be79-b9589a79f5e9/

 

I guess strictly speaking, it wouldn't be managed code anymore would it? There are things you relinquish when you start using .NET and it appears that this is one of them.

vesuvius said:

wkempf said:

*snip*

I guess strictly speaking, it wouldn't be managed code anymore would it? There are things you relinquish when you start using .NET and it appears that this is one of them.

Not really. You couldn't do this in C++ either, at least in general.

C# has sizeof(), but it's going to report 4 for reference types.  C++ has sizeof(), but it will report 4 for pointers.  In both cases, you'd be reporting something different from "how much memory is my list consuming". At least in .NET you can use reflection to find an answer, though I'm not sure this is a great idea in general, much less for your needs here.

Sven Groot said:

KevinB said:

*snip*

Could be anything. It's part of a library, users of the code can create their own record types, so I don't know anything at all about the records a priori.

Expose the threshold (in number of objects) in your API and let the library user determine an appropriate value for the task at hand.

If you are concerned about the amount of memory the application is using, why don't you measure that instead of the list?  You can compare the process memory to the available memory to avoid paging.  This is really just an approximation (another thread could be eating memory) but it may be good enough?  The advantage is that your memory limit automatically increases on new hardware or decreases if other processes are eating up memory (let's say SQL is running on the same machine).

Depending on your app, i would consider sorting in memory and flushing partitions to disk (then build another partition in memory and repeat).  Then you can just merge the partitions once your are ready instead of worrying about a disk based sort. 

kriskdf said:

If you are concerned about the amount of memory the application is using, why don't you measure that instead of the list?  You can compare the process memory to the available memory to avoid paging.  This is really just an approximation (another thread could be eating memory) but it may be good enough?  The advantage is that your memory limit automatically increases on new hardware or decreases if other processes are eating up memory (let's say SQL is running on the same machine).

 

Depending on your app, i would consider sorting in memory and flushing partitions to disk (then build another partition in memory and repeat).  Then you can just merge the partitions once your are ready instead of worrying about a disk based sort. 

The problem is that this has to work on Mono too. Mono's GC almost never actually releases memory from the managed heap, so the process's memory usage will pretty much never go down, making it an unreliable measurement.

It's not an immediate issue, I just figured I'd ask since if someone here knew a good solution I could use that. It seems obvious now there isn't one.

Sven Groot said:

kriskdf said:

*snip*

The problem is that this has to work on Mono too. Mono's GC almost never actually releases memory from the managed heap, so the process's memory usage will pretty much never go down, making it an unreliable measurement.

 

It's not an immediate issue, I just figured I'd ask since if someone here knew a good solution I could use that. It seems obvious now there isn't one.

Yep, I think I asked about this back in 2005 or so and had no luck.  We eventually went to using perfmon and actively monitoring memory usage...there might be a way to do this programmatically.

if you were especially clever and had the free time available, you could probably write something that reflected through the object until you reached the primitive types.  It wouldn't be perfect, especially because of string reuse, but it would give you an estimate.