ASP.NET Out Of Memory Problems

If anyone here uses asp.net, I'd appreciate some advice. We are having problems with our main web application where internally we start getting a lot of OutOfMemoryExceptions and all our users start seeing a variety of error messages when the submit/navigate to a new page. Here s the message I proposed we send to our users this morning:

"We have identified the cause the problems in our web application and are testing of fixes is ongoing. As usage of our web application reaches a memory limit of a little over 1,400,00KB (or 1.4GB), users may begin seeing a variety errors. When this memory limit is reached, everyone will encounter errors and the web application on our web server will reset shortly thereafter. Any users of the system will continue to receive errors until they log in again after the reset has completed.

The last few days we have been developing and testing fixes to keep our web application from continually consuming excessive amounts of memory, which is the root cause of all these problems. Unfortunately this process involves a wait and see approach each day as we try different changes and settings while monitoring the results. We appreciate your patience while we work through this issue and we hope to have a permanent solution very soon."

Our web application is built on DevExpress XPO (an ORM system) and their XAF (eXpress Application Framework). The data in our system current has 38,000 Client entities, and 53,000 Transaction entities.

Using a memory profiler I see those are the two biggest groups of entities which stay "live" and are chewing up memory. I've also tried manually collecting the garbage, which helps a little, but the memory still keep building up and up.

I'm sure our users/customers are getting frustrated, but I'm no closer to a solution. I'm at the point where I am logging each page generation, recording the time, used memory, the current entity/view, but am so far unable to fix this problem.

Does anyone have any advice as to what I can try next to resolve this problem?

TIA

http://blogs.iis.net/webtopics/archive/2009/05/22/troubleshooting-system-outofmemoryexceptions-in-asp-net.aspx

@sysrpl: I have found that Out of Memory exceptions typically occur when there is too much reliance on Session objects.  In my ASP.NET apps I don't use Session at all because it uses up server memory and also times out.  I rely on View State, Query String paramters, and database calls.  I haven't had a single Out of Memory exception since.

Hope that helps.

@JeremyJ: Thanks Jerry. Unfortunately I am tied down to whatever method of session management this framework (DevExpress XAF) is using.

@sysrpl: Can you set the web app to store session state in the database? It would be a bit slower, but it would keep the web app from using up all of that memory.

If memory keeps going up, something somewhere is keeping a reference live when it shouldn't have.  Apparently, the only design pattern some devs know about is the singleton, and it is a good source of these kinds of issues.

If you can, run a differential profile where you run your app for a while, look at the objects that exist before 1 iteration of the app, and then look again after 1 iteration.  By iteration, I mean 'use the app once'.

If you look then at the counts of objects that exist before and after, you should then be able to cross-reference them to see if some stuff is still in memory when it shouldn't be.  And just because you see a bunch of memory being used by the 3rd party objects doesn't mean that memory isn't needed.  Memory leaks are never where you think they are

4 hours ago, sysrpl wrote

Our web application is built on DevExpress XPO (an ORM system) and their XAF (eXpress Application Framework). The data in our system current has 38,000 Client entities, and 53,000 Transaction entities.

Your client entities = session? Meaning it will use memory until client logs off? And transaction entities are? Not sure what this is. I mean, even in a concurrent situation, I thought rowversion would would be able to deal with that. You can easily compare the rowversion at HDD level to make sure the value you update is not stale. So, I am not sure why you would have so many transactions. Unless your transactions are stuck and keep piling up? 

@magicalclick: A client is something like this:

Transactions can be attached to Clients:

@sysrpl:

Looks like a databas entities, I don't think that would have impact on ram, unless you didn't do paging on the result and you have a long page cache period.

Other than that, I would think about stastic code on a page if they are managed controls. assuming the 3rd party control is managed.

The thing that has caught me out before was not removing event-handlers for an object before disposing it; event-handlers count as a reference to the object so if there is a centralised subscriber that lives a long time, all the instances it holds event subscriptions to are kept alive.

I always find memory leaks hard to figure out

Herbie