Recommendations: IIS Website Log Analyser | The Coffeehouse

I’ve been trying to find the “perfect” log-file analysis and stats package for an IIS server, and am still searching.

On the off-chance that someone has already found it... I want:

Log Analyser engine that runs as an NT-Service on the Web-Server
Preferably store the analysis data into an SQL-Server
Access the reports via website, preferably ASP.NET based (see above SQL-Server preference)
Allow me to provide reporting to my customers on a secure site-by-site basis.

Basically, I host a number of sites which I develop for my customers, that are very specific to their needs, and for which I create specific campaigns on a regular basis.

I want them to be able to access the stats for their sites (so they have immediate feedback on what’s happening) via secure pages on their own site, and not require me to intervene or to manage the process on a day-to-day basis.

W3bbo

How is it going to store analysis data in a SQL Server? The in-memory representation of an analysis of records cannot be refactored into a series of rows.

I'm aware of only a handful of stats programs which enable you to persist in-memory state, chiefly Webalyzer, and the contents of those state files is pretty complicated.

And even so, the generated reports are going to be static HTML files, not dynamic ASP.NET sites, simply because the time to generate a report can be as long as 10 minutes (not forgetting huge memory utilisation if you're storing all of the logs in-memory).

No, the best way is like any other, read the logfiles in (or read them line-by-line to conserve memory usage at the expense of losing more complicated statistical anaylssi), work on the data, generate a report (with pretty graphics), persist it all to disk. Run the program all over again the very next day (usually controlled by a Service or Scheduled Task).

MB

I suspect I didn't do a good job of explaining "database" and asp pages.

This is the TYPE of thing I'm talking about

http://demo.logrover.com/login.asp?demohelp=1

They use a proprietory database to store the processed data, and a scheduler that updates from the most recent log records.

You can then generate dynamic reports over various periods, and compare periods, using the asp report pages.

I'm sure there must be similar packages out there.

W3bbo

Why not just work with the original IIS *.log files? They contain more information you can work with, including requests for non-asp pages.

MB

I don't want to turn this into an IIS log analysys discussion, and bore everyone stiff, as I was just after any experiences with analysers & stats packages, but since you asked...

Yes, as you say, the IIS logs have all the detail you want/need... and I have websites spewing out multi-gigabytes of this detail every day. Even compressed, this will rip through disk space quick smart.

Hence I'm looking for a product (Urchin-Like) that stores processed aggregated data, in a progressive manner, into a database, so that you can archive or even dispose of the raw data... and you can then work with the database to generate dynamic reports.

W3bbo

MB wrote:

Hence I'm looking for a product (Urchin-Like) that stores processed aggregated data, in a progressive manner, into a database, so that you can archive or even dispose of the raw data... and you can then work with the database to generate dynamic reports.

All of the programs I know of store the aggregate data in the filesystem, not a database and the reports aren't dynamic.

ZippyV

Can't you set the option in IIS to store the log directly in SQL server? Then you could use analysis and reporting services to process the data.

W3bbo

ZippyV wrote:

Can't you set the option in IIS to store the log directly in SQL server? Then you could use analysis and reporting services to process the data.

You can set IIS to store Request/Response logs to any ODBC Data Source, however the fields stored in the database is fixed and lacks many useful ones, like the HTTP Referrer field.

SlackmasterK

I use WebLog Expert. It's not nearly as in-depth as what you're looking for, but it's also not nearly as proprietary. I suppose the name is somewhat amusing as I use it to monitor my blog.

http://www.weblogexpert.com/

W3bbo

SlackmasterK wrote:

I use WebLog Expert. It's not nearly as in-depth as what you're looking for, but it's also not nearly as proprietary. I suppose the name is somewhat amusing as I use it to monitor my blog.

http://www.weblogexpert.com/

Weblog Expert reads all the logs into memory then works on it, this is inefficient and leads to massive memory usage; it also isn't as fast either.

ZippyV

Weird, I just looked at the logging options for IIS 7 and I can't set it to save to sql server.

blowdart

W3bbo wrote:

SlackmasterK wrote:I use WebLog Expert. It's not nearly as in-depth as what you're looking for, but it's also not nearly as proprietary. I suppose the name is somewhat amusing as I use it to monitor my blog.

http://www.weblogexpert.com/

Weblog Expert reads all the logs into memory then works on it, this is inefficient and leads to massive memory usage; it also isn't as fast either.