Tech Off Thread

4 posts

How do I get search engine results programatically?

Back to Forum: Tech Off
  • complete

    How do I get search engine results programatically?  Some search engines use AJAX to display content and so the URL does not change. This makes it difficult to get result content from a search engine programatically.

    Most search engines do not use AJAX.  But two popular ones that I am interested in is YANDEX and DUCKDUCKGO.  Both of these search engines use AJAX to advance beyond the first page of results.  How do I find a way to programatically access these pages?

  • complete

    I do not use screenscraping.  I use the c# WebBrowser class and I get the HTML after I use this class to navagate to a URL.  I was able to use yandex by using a "&p=" to navagate to a particular page.

    The duckduckgo search engine does not work this way, though.

    I have found out how to make a specialized query for YANDEX in a modified URL.  I will ask DUCKDUCKGO for advice.

    https://duckduckgo.com/api.html does not provide a solution.

    Running this, http://api.duckduckgo.com/?q=stuff+to+search&format=xml , in a browser's address bar does not provide anything as http://duckduckgo.com/?q=stuff+to+search does.

    This shouldn't be rocket science.  I am able to see the HTML code.  Shouldn't there be some sort of AJAX tag in the HTML that I can find to make use of?

  • ZippyV

    From DuckDuckGo:

    So is there then another API with all the Web links?

    Since we use many underlying APIs in our full results (merged with our own indexes), we do not have rights at this time provide an API with all our Web links. We do, however, provide search box integration.

  • ZippyV

    Yandex provides a XML api for you:

    http://api.yandex.com/xml/

Comments closed

Comments have been closed since this content was published more than 30 days ago, but if you'd like to continue the conversation, please create a new thread in our Forums, or Contact Us and let us know.