Jump to content

How to search Google and Bing in C#

+ 3
  JonUdell's Photo
Posted Oct 26 2010 09:02 AM

Introduction

In this week's companion article on the Radar blog, I discuss a feature of the elmcity service that enables curators to find web pages that mention recurring events. The elmcity service searches Google and Bing for a family of queries like every 2nd Tuesday at 9PM. But you can use the core search methods for any queries against the JSON APIs exposed by Google and Bing.

Encapsulating Google's search API

Here's a simple class to represent a search result along with its source, which might be Google or Bing or both.


  public class SearchResult
  {
    public string url;
    public string title;
    public string content;
    public FindingEngine engine;

    public enum FindingEngine { google, bing, google_and_bing };

    public SearchResult(string url, string title, string content, FindingEngine engine)
    {
      this.url = url;
      this.title = title;
      this.content = content;
      this.engine = engine;
    }
  }

Both searchers return lists of these objects. Here's the Google searcher:


    public static List<SearchResult> GoogleSearch(string search_expression, 
      Dictionary<stringobject> stats_dict)
    {
      var url_template = "http://ajax.googleapis.com/ajax/services/search/web?v=1.0&rsz=large \
        &safe=active&q={0}&start={1}"
;
      Uri search_url;
      var results_list = new List<SearchResult>();
      int[] offsets = { 0, 8, 16, 24, 32, 40, 48 };
      foreach (var offset in offsets)
      {
        search_url = new Uri(string.Format(url_template, search_expression, offset));

        var page = new WebClient().DownloadString(search_url);

        JObject o = (JObject)JsonConvert.DeserializeObject(page);

        var results_query =
          from result in o["responseData"]["results"].Children()
          select new SearchResult(
              url: result.Value<string>("url").ToString(),
              title: result.Value<string>("title").ToString(),
              content: result.Value<string>("content").ToString(),
              engine: SearchResult.FindingEngine.google
              );

        foreach (var result in results_query)
          results_list.Add(result);
      }

      return results_list;
    }

The JSON deserializer, JsonConvert.DeserializeObject, is part of James Newton-King's Json.NET library. Why use it here, rather than one of the deserializers built into the .NET Framework? Those deserializers -- JavascriptSerializer (in System.Web.Extensions) and DataContractJsonSerializer (in System.Runtime.Serialization.Web) expect to map JSON to .NET types. But here I have no other need for those .NET types, and I don't want to bother to define them, I just want to parse some values out of the JSON. Json.NET is a convenient way to do that.

Encapsulating Bing's search API

Here's the Bing searcher:


    public static List<SearchResult> BingSearch(string search_expression, Dictionary<stringobject> stats_dict)
    {
      var url_template = "http://api.search.live.net/json.aspx?AppId=8F9B...1CF6&Market=en-US& \
        Sources=Web&Adult=Strict&Query={0}&Web.Count=50"
;
      var offset_template = "&Web.Offset={1}";
      var results_list = new List<SearchResult>();
      Uri search_url;
      int[] offsets = { 0, 50, 100, 150 };
      foreach (var offset in offsets)
      {
        if (offset == 0)
          search_url = new Uri(string.Format(url_template, search_expression));
        else
          search_url = new Uri(string.Format(url_template + offset_template, search_expression, offset));

        var page = new WebClient().DownloadString(search_url);

        JObject o = (JObject)JsonConvert.DeserializeObject(page);

        var results_query =
          from result in o["SearchResponse"]["Web"]["Results"].Children()
          select new SearchResult
              (
              url: result.Value<string>("Url").ToString(),
              title: result.Value<string>("Title").ToString(),
              content: result.Value<string>("Description").ToString(),
              engine: SearchResult.FindingEngine.bing
              );

        foreach (var result in results_query)
          results_list.Add(result);
      }

      return results_list;
    }


Tags:
0 Subscribe


8 Replies

0
  garo77's Photo
Posted Nov 19 2010 07:26 AM

Hi there, sorry about my english, I'm new developer with c sharp. Im trying to do a search of any text and show the results into a datagrid into my windows app. I understan the code you put in here but i can not make it work cause throw an error. Maybe i don not make it well. Could you please help me with a funtional exmaple?? or any tip?? Thanks in advance.(Visual Studio 2010)

Best regards
+ 1
  jimibt's Photo
Posted Dec 24 2010 04:38 AM

Nice little article. took the liberty of doing an alternative version using interfaces etc. hope it gives another viewpoint.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Net;
using Newtonsoft.Json;
using Newtonsoft.Json.Linq;
using Test.Api;

namespace Test.Api
{
    public struct SearchType
    {
        public string url;
        public string title;
        public string content;
        public FindingEngine engine;
        public enum FindingEngine { Google, Bing, GoogleAndBing };
    }

    public interface ISearchResult
    {
        SearchType.FindingEngine Engine { get; set; }
        string SearchExpression { get; set; }
        List<SearchType> Search();
    }

    public class BingSearch : ISearchResult
    {
        public BingSearch(string searchExpression)
        {
            this.Engine = SearchType.FindingEngine.Bing;
            this.SearchExpression = searchExpression;
        }
        public SearchType.FindingEngine Engine { get; set; }
        public string SearchExpression { get; set; }

        public List<SearchType> Search()
        {
            // our appid from bing - 3F4313687C37F1....23A79E181D0A25
            const string urlTemplate = @"http://api.search.live.net/json.aspx?AppId=3F4313687C37F1...23A79E181D0A25& \
                                            Market=en-US&Sources=Web&Adult=Strict&Query={0}&Web.Count=50";
            const string offsetTemplate = "&Web.Offset={1}";
            var resultsList = new List<SearchType>();
            int[] offsets = { 0, 50, 100, 150 };
            Uri searchUrl;
            foreach (var offset in offsets)
            {
                if (offset == 0)
                    searchUrl = new Uri(string.Format(urlTemplate, SearchExpression));
                else
                    searchUrl = new Uri(string.Format(urlTemplate + offsetTemplate, 
                                        SearchExpression, offset));

                var page = new WebClient().DownloadString(searchUrl);
                var o = (JObject)JsonConvert.DeserializeObject(page);

                var resultsQuery =
                  from result in o["SearchResponse"]["Web"]["Results"].Children()
                  select new SearchType
                  {
                      url = result.Value<string>("Url").ToString(),
                      title = result.Value<string>("Title").ToString(),
                      content = result.Value<string>("Description").ToString(),
                      engine = this.Engine
                  };

                resultsList.AddRange(resultsQuery);
            }
            return resultsList;
        }
    }

    public class GoogleSearch : ISearchResult
    {
        public GoogleSearch(string searchExpression)
        {
            this.Engine = SearchType.FindingEngine.Google;
            this.SearchExpression = searchExpression;
        }
        public SearchType.FindingEngine Engine { get; set; }
        public string SearchExpression { get; set; }

        public List<SearchType> Search()
        {
            const string urlTemplate = @"http://ajax.googleapis.com/ajax/services/search/web?v=1.0& \
                                        rsz=large&safe=active&q={0}&start={1}";
            var resultsList = new List<SearchType>();
            int[] offsets = { 0, 8, 16, 24, 32, 40, 48 };
            foreach (var offset in offsets)
            {
                var searchUrl = new Uri(string.Format(urlTemplate, SearchExpression, offset));
                var page = new WebClient().DownloadString(searchUrl);
                var o = (JObject)JsonConvert.DeserializeObject(page);

                var resultsQuery =
                  from result in o["responseData"]["results"].Children()
                  select new SearchType
                  {
                      url = result.Value<string>("url").ToString(),
                      title = result.Value<string>("title").ToString(),
                      content = result.Value<string>("content").ToString(),
                      engine = this.Engine
                  };

                resultsList.AddRange(resultsQuery);
            }
            return resultsList;
        }
    }
}

class Program
{
    static void Main(string[] args)
    {
        //ISearchResult searchClass = new BingSearch("Asp.net mvc");
        ISearchResult searchClass = new GoogleSearch("Asp.net mvc");
        var list = searchClass.Search();
        foreach (var searchType in list)
        {
            Console.WriteLine(searchType.title);
        }
        Console.Read();
    }
}

0
  MarceCRodriguez's Photo
Posted May 02 2012 07:28 AM

Hello,

The JSON deserializer works really fine, and I was able to do searches from google APIs using this examples (.net with C# accessing to http://ajax.googleap...es/search/web).

The problem is, after some searches, google blocks me, and the application starts to fail. The error that prompts is:

"Error 403 'Suspected Terms of Service Abuse. Please see http://code.google.com/apis/errors" .I´ve added a user_ip in my searches, but the error is still there.

I´m wondering if there have been changes on google terms of service that disables this way of accessing to google APIs. Any help will be appreciated.

Best Regards,

Marcela
0
  alitanloo's Photo
Posted Dec 15 2012 03:31 PM

hi
i want to write a program with c#.net that search my word in search engine like google,bing or aol and find my web site after that brows my web site,
how can i write please help me.
0
  Alex9's Photo
Posted Dec 25 2012 11:18 PM

I have use first example and only 56 results in program causes on my query. Why so little list ?
0
  Brinda Arumugam's Photo
Posted Jan 05 2013 01:58 AM

i hav little doubt .. i did following code

private void button1_Click(object sender, EventArgs e)
{
//ISearchResult searchClass = new BingSearch("Asp.net mvc");
ISearchResult searchClass = new GoogleSearch("Asp.net mvc");
var list = searchClass.Search();
foreach (var searchType in list)
{
richTextBox1.AppendText(searchType.title);
}
}
}

public struct SearchType
{
public string url;
public string title;
public string content;
public FindingEngine engine;
public enum FindingEngine { Google, Bing, GoogleAndBing };
}

public interface ISearchResult
{
SearchType.FindingEngine Engine { get; set; }
string SearchExpression { get; set; }
List<SearchType> Search();
}

public class BingSearch : ISearchResult
{
public BingSearch(string searchExpression)
{
this.Engine = SearchType.FindingEngine.Bing;
this.SearchExpression = searchExpression;
}
public SearchType.FindingEngine Engine { get; set; }
public string SearchExpression { get; set; }

public List<SearchType> Search()
{
// our appid from bing - 3F4313687C37F1....23A79E181D0A25
const string urlTemplate = @"http://api.search.live.net/json.aspx?AppId=3F4313687C37F1...23A79E181D0A25& \
Market=en-US&Sources=Web&Adult=Strict&Query={0}&Web.Count=50";
const string offsetTemplate = "&Web.Offset={1}";
var resultsList = new List<SearchType>();
int[] offsets = { 0, 50, 100, 150 };
Uri searchUrl;
foreach (var offset in offsets)
{
if (offset == 0)
searchUrl = new Uri(string.Format(urlTemplate, SearchExpression));
else
searchUrl = new Uri(string.Format(urlTemplate + offsetTemplate,
SearchExpression, offset));

var page = new WebClient().DownloadString(searchUrl);
var o = (JObject)JsonConvert.DeserializeObject(page);

var resultsQuery =
from result in o["SearchResponse"]["Web"]["Results"].Children()
select new SearchType
{
url = result.Value<string>("Url").ToString(),
title = result.Value<string>("Title").ToString(),
content = result.Value<string>("Description").ToString(),
engine = this.Engine
};

resultsList.AddRange(resultsQuery);
}
return resultsList;
}
}

public class GoogleSearch : ISearchResult
{
public GoogleSearch(string searchExpression)
{
this.Engine = SearchType.FindingEngine.Google;
this.SearchExpression = searchExpression;
}
public SearchType.FindingEngine Engine { get; set; }
public string SearchExpression { get; set; }

public List<SearchType> Search()
{
const string urlTemplate = @"http://ajax.googleapis.com/ajax/services/search/web?v=1.0& \
rsz=large&safe=active&q={0}&start={1}";
var resultsList = new List<SearchType>();
int[] offsets = { 0, 8, 16, 24, 32, 40, 48 };
foreach (var offset in offsets)
{
var searchUrl = new Uri(string.Format(urlTemplate, SearchExpression, offset));
var page = new WebClient().DownloadString(searchUrl);
var o = (JObject)JsonConvert.DeserializeObject(page);

var resultsQuery =
from result in o["responseData"]["results"].Children()
select new SearchType
{
url = result.Value<string>("url").ToString(),
engine = this.Engine
};

resultsList.AddRange(resultsQuery);
}
return resultsList;
}
}

but i got a error :
-1
  Brinda Arumugam's Photo
Posted Jan 05 2013 02:07 AM

error occur
Cannot access child value on Newtonsoft.Json.Linq.JValue.

plz help!!!!!!
0
  zfarzaneh's Photo
Posted Dec 09 2013 02:12 AM

hi ,Offset you have written ,on what basis?