Measure .Net object size | Tech Off

Sven Groot
My name has 9 letters. Coincidence? I think not...

I'm storing a lot of objects in a List<T>. The number of objects can be very, very large, and is unknown beforehand. I also want to limit the memory usage of my application. Essentially the final goal is to sort the items.

What I want to do is this: when the total in memory size of the List<T> exceeds a certain size, I want to sort what I have so far, save them to disk, and continue with an external sorting algorithm. However, in order to do that, I need to know how much memory all the objects in the List<T> are using. Is that even possible in .Net? How would you solve this problem?

KevinB

Do you know what the T you are going to be storing in your List<T> is. Can an object of Type T be of variable size or is it fixed?

Sven Groot
My name has 9 letters. Coincidence? I think not...

In reply to KevinB

#Jul 21st @ 9:16 AM

Could be anything. It's part of a library, users of the code can create their own record types, so I don't know anything at all about the records a priori.

wkempf

This can't really be done. http://social.msdn.microsoft.com/forums/en-US/clr/thread/b871dee4-6eb5-4dca-be79-b9589a79f5e9/

vesuvius
Das Glasperlenspiel

In reply to wkempf

#Jul 21st @ 10:35 AM

I guess strictly speaking, it wouldn't be managed code anymore would it? There are things you relinquish when you start using .NET and it appears that this is one of them.

wkempf

In reply to vesuvius

#Jul 21st @ 12:10 PM

Not really. You couldn't do this in C++ either, at least in general.

C# has sizeof(), but it's going to report 4 for reference types. C++ has sizeof(), but it will report 4 for pointers. In both cases, you'd be reporting something different from "how much memory is my list consuming". At least in .NET you can use reflection to find an answer, though I'm not sure this is a great idea in general, much less for your needs here.

DCMonkey
Monkey see, monkey do, monkey will destroy you!

In reply to Sven Groot

#Jul 21st @ 9:28 AM

Expose the threshold (in number of objects) in your API and let the library user determine an appropriate value for the task at hand.

kriskdf

If you are concerned about the amount of memory the application is using, why don't you measure that instead of the list? You can compare the process memory to the available memory to avoid paging. This is really just an approximation (another thread could be eating memory) but it may be good enough? The advantage is that your memory limit automatically increases on new hardware or decreases if other processes are eating up memory (let's say SQL is running on the same machine).

Depending on your app, i would consider sorting in memory and flushing partitions to disk (then build another partition in memory and repeat). Then you can just merge the partitions once your are ready instead of worrying about a disk based sort.

Sven Groot
My name has 9 letters. Coincidence? I think not...

In reply to kriskdf

#Jul 21st @ 11:57 PM

The problem is that this has to work on Mono too. Mono's GC almost never actually releases memory from the managed heap, so the process's memory usage will pretty much never go down, making it an unreliable measurement.

It's not an immediate issue, I just figured I'd ask since if someone here knew a good solution I could use that. It seems obvious now there isn't one.

ScanIAm
On a scale of 1 to 10, people are stupid.

In reply to Sven Groot

#Jul 22nd @ 1:47 AM

Yep, I think I asked about this back in 2005 or so and had no luck. We eventually went to using perfmon and actively monitoring memory usage...there might be a way to do this programmatically.

if you were especially clever and had the free time available, you could probably write something that reflected through the object until you reached the primitive types. It wouldn't be perfect, especially because of string reuse, but it would give you an estimate.