Website maxs out the cpus

I don't where to begin researching this problem, so I thought I'd ask you guys here where I could begin finding out what gone wrong with our asp.net application.

The problem: Occasionally our application gets into a state where the asp.net worker process pegs the CPU at 100% usage and stays there until IIS is shutdown/restarted. After the CPU is pegged users start getting asp.net errors all over the place.

Our application is built on top of a fairly complication DevExpress XAF (an application framework) and is deployed as an asp.net application. We've only recently started having this problem and only recently have had a lot of people (5 or more) doing their daily work using the application. I've checked the XAF application logs and don't see anything that tells me why or what causes the application to peg the CPU.

I am asking here because XAF is a complex framework, and in the past asking DevExpress for help (under the product we have with them) almost never resolves anything. They always want you to zip up you project and send it to them (our project is rather large at this point), and they rarely are able to reproduce problems or give good advice.

Has anyone here run into situations where your asp.net application uses all the CPU resources and won't let go? How can I prevent this from happening or where should I start looking (other than google and the XAF application logs which I've already done) to decide what to investigate next?

Thanks.

Usually we have dev use something like ANTS Memory Profiler to see what all the things are doing and then go from there.

first thing that jumps out at me is this:

" We've only recently started having this problem and only recently have had a lot of people (5 or more) doing their daily work using the application. "

so some things to look at possibly are:  what do you put in "Session" ?

could there be a problem with say SQL connections ??

for example years back when .net 2.0 first came out there was a problem with using data readers that could lead to filling up the sql connection pool.

how ,uch of the site is "statefull" ?   what does the  viewsource / viewstate look like ?

what is the normal number of users ?

oh also sometimes a database table / tables that need indexes can make for ugly perf issues.

as a table grows if it's not well indexed then select statements start generating table scans that take to long and hog up the cpu time.

Are you not able to shutdown the worker process without shutting down IIS? You should be able to recycle the application pool (or at the very least shutdown and restart the same) without having to touch the W3Svc.

8 minutes ago,figuerres wrote

first thing that jumps out at me is this:

" We've only recently started having this problem and only recently have had a lot of people (5 or more) doing their daily work using the application. "

so some things to look at possibly are:  what do you put in "Session" ?

could there be a problem with say SQL connections ??

for example years back when .net 2.0 first came out there was a problem with using data readers that could lead to filling up the sql connection pool.

how ,uch of the site is "statefull" ?   what does the  viewsource / viewstate look like ?

what is the normal number of users ?

Almost every time I've seen something like this happening, it's tied to the database connection. It can be something as innocuous as forgetting to wrap a SqlConnection instance inside a using statement.

That's where a good profiler starts. I think some of them will tell you what the function calls are that are doing th damage. You're never going to find the issue if it's just a "database thing", but if you see that it's the GetAllCompaniesByPartnerType() function, you know where to start.

2 thoughts come to mind:

1.  A logic loop caused by an unexpected data state.  The Profiler approach as Harlequin suggested would reveal that.

2. A database locking contention issue with some sort of database retry logic.  Use the database profiler to see who is deadlocking who and what the deadlocker is currently doing.  Or also look for unexpected repeated database calls.

Thanks all for you advice. I'll check the connection string and make sure it's pooling. I am pretty sure any SQL connection I am using uses the connection string from the web.config file. I am also pretty sure all my SQL reads (which use SQL Reader) wrap everything in using statements.

Q: How could excessively orphaned SQL connections leave the CPU maxed?

Also, the XAF Framework is built on top of the DevExpress ORM they call XPO, which in turn uses their own ObjectSpace and UnitOfWork objects to wrap connections. I do often create new ObjectSpace instances, but that's only because every single view needs it's own ObjectSpace. That is, when you want to create a view, you have to define a ViewController and when the controller is activated you create a ObjectSpace then call myNewObjectSpace.SetView(ClientDetailView) and then just forget about the ObjectSpace (i assume the thing that maintains the view controllers also maintains the ObjectSpaces I create).

Harlequin: Thanks. I'll look into the ANTS Profiler.

Just to be sure there are no SQL Server connections being left open, run the following script from SSMS:

SELECT DB_NAME(dbid) as 'DbName', * from master.dbo.sysprocesses

1 hour ago,sysrpl wrote

Thanks all for you advice. I'll check the connection string and make sure it's pooling. I am pretty sure any SQL connection I am using uses the connection string from the web.config file. I am also pretty sure all my SQL reads (which use SQL Reader) wrap everything in using statements.

Q: How could excessively orphaned SQL connections leave the CPU maxed?

Also, the XAF Framework is built on top of the DevExpress ORM they call XPO, which in turn uses their own ObjectSpace and UnitOfWork objects to wrap connections. I do often create new ObjectSpace instances, but that's only because every single view needs it's own ObjectSpace. That is, when you want to create a view, you have to define a ViewController and when the controller is activated you create a ObjectSpace then call myNewObjectSpace.SetView(ClientDetailView) and then just forget about the ObjectSpace (i assume the thing that maintains the view controllers also maintains the ObjectSpaces I create).

Harlequin: Thanks. I'll look into the ANTS Profiler.

well picture this, now this is really bad but....

imagin that say 100 users are going on the site, that they are hitting a number of pages that have badly done sql connection and sql datareader and that possibly the app has say 3 different versions of the connection string   and the data readers are not getting closed - in turn the connection is not closed .... then the connection pool is filled and grows and the sql connection objects are using .net memory and not getting GC'ed as they are open handles ...

if 100 users * 3 connections per page hit * 3 page hits == 900 open data readers + 900 open db connection BOOM !  ram goes up till you hit swap file then cpu goes nuts swapping pages and the server goes really really slow....

that's a really bad case but it has happened i have seen it.

the other thing is say if you have a lot of inserts to a table like orders and order details

the basic logic may be right and the connections may be closed but...

if the default locking and PK / clustered indexes are used then the inserts will create page locks.

the page locks will then escalate to table locks and you get a kind of "traffic jam" where 20 orders are in transactions that are waiting for other orders to commit and let go of locks....

BOOM  cpu and RAM go nuts!

or say you have a large table that gets a select but no good index is ther to filter out the rows?

then sql does a table scan to try and find the rows, if the tbale only has say 100 rows or 1,000 rows you do not see a problem with only 5 users.

but have say 100,000 rows and 100 users and BOOM!  you are cpu bound on checking all the rows for each users page of data ....

i manage a system where our data can grow to over 100 gigs in a 6 month time frame and i have tables with 20 - 30 million rows.   i have seen this stuff happen.

@sysrpl: it's hard to help without having all the facts, but the fact that the problem materializes as soon as a you reach a small number of users might indicate the presence of a deadlock or some kind of contention. If you have some code that needs resources that other threads generate or could be using, take a close look at how they get synchronized.

@sysrpl: Please email me at dennis@devexpress.com and provide more details on the issue you are having with XAF Web UI. I will be glad to help you. Thank you in advance!