Screen scraping AskVille | The Coffeehouse

Steve411

Hey everyone,

I'm having a hard time with this one. Since AskVille does not provide an RSS feed I can work with, I need to write a regular expression which will retrieve all of the results of a search.

So for example, on askville.com if i were to search for "google", i'd get referred to this page:
http://askville.amazon.com/SearchRequests.do?search=google&open=true&closed=false

It returns about 10 individual results. I need to grab the URL and title of each result. Can anyone help me with the regular expression for this?

Thanks.
Steve

Sven Groot
My name has 9 letters. Coincidence? I think not...

Forget about regular expressions. Extracting stuff from HTML with regular expressions is a pain and extremely error prone.

Instead, I suggest you use the HTML Agility Pack, a free full-blown HTML parser (that can deal with malformed HTML because the page you linked to is not well-formed), and then use an XPath expression to get the elements you want.

It seems the results are contained in a <div> with id "ad_contained_1", so all you need is an XPath expression //div[@id='ad_contained_1']/a to get all the anchor elements for the results. Then it's a simple matter of reading the contents and the href attribute for each element to get the text and link.

Sven Groot
My name has 9 letters. Coincidence? I think not...

In reply to Sven Groot

#Jul 14th, 2008 @ 1:36 AM

Quick sample to get you started:

HtmlAgilityPack.HtmlDocument doc = new HtmlAgilityPack.HtmlDocument();

Uri baseUrl = new Uri("http://askville.amazon.com/SearchRequests.do?search=google&open=true&closed=false");

using( WebClient client = new WebClient() )
using( Stream s = client.OpenRead(baseUrl) )
{
    doc.Load(s);
}


HtmlAgilityPack.HtmlNodeCollection results = doc.DocumentNode.SelectNodes("//div[@id='ad_contained_1']/a");

foreach( HtmlAgilityPack.HtmlNode node in results )
{
    Uri url = new Uri(baseUrl, node.GetAttributeValue("href", ""));
    string text = node.InnerText.Trim();
    // Do something with it.
}