Jump to content

How to use the Google Suggest API to come up with topics for Answers

+ 2
  odewahn1's Photo
Posted May 24 2010 12:01 PM

Google has a nice little RESTful API that powers the auto-suggest function:

http://blogoscoped.c...-08-17-n22.html

The API returns the top 10 suggestions for the given query, plus the number of times it was searched for.

I wrote a quick Python script that allows you to put in a phrase like "How do i ..." and then pull the top 10 results for each letter in the alphabet (i.e., "How do i a," "how do i b," ... "how do i z"). In its infinite wisdom, Google converts the letter into the most used word that starts with that letter. The output is sent to the screen, so you should redirect it into a file if you want to save the results.

from sgmllib import SGMLParser
import urllib2
import urllib

# Define the class that will parse the suggestion XML
class PullSuggestions(SGMLParser):

   def reset(self):
      SGMLParser.reset(self)
      self.suggestions = []
      self.queries = []

   def start_suggestion(self, attrs):
      for a in attrs:
         if a[0] == 'data': self.suggestions.append(a[1])

   def start_num_queries(self, attrs):
      for a in attrs:
         if a[0] == 'int': self.queries.append(a[1])


# -----------------------------------------------------------
# Enter your phrase here.  Be sure to leave the %s at the end!
# -----------------------------------------------------------
base_query = "how do i %s"  #This is the base query

alphabet = "abcdefghijklmnopqrstuvwxyz"
for letter in alphabet:
   q = base_query % letter;
   query = urllib.urlencode({'q' : q})
   url = "http://google.com/complete/search?output=toolbar&%s" % query

   res = urllib2.urlopen(url)
   parser = PullSuggestions()
   parser.feed(res.read())
   parser.close()

   for i in range(0,len(parser.suggestions)):
      print "%s\t%s" % (parser.suggestions[i], parser.queries[i])


So, no more excuses: drop into your terminal, fire up Python, run this script, and start adding some Answers. The world cannot wait to learn how to delete a Facebook account, jailbreak an iPhone, become a vampire, or zest a lemon. Won't you please help these desperate, lemon-zesting-deficient people out?

Tags:
2 Subscribe


7 Replies

+ 1
  georgerix's Photo
Posted Sep 05 2010 04:48 AM

Can you suggest how this script can be modified to use 'past month' or 'past 2 weeks' to limit the search. Also, I am interested in how to enter a date range, for example, 1998 through 2008. Can the results be sorted by the numerical result? How do you direct the output to a text file?

Lots of requests I know but any help would be appreciated.

Thanks,
George
0
  odewahn1's Photo
Posted Sep 09 2010 03:47 AM

Hi, georgerix. I think the suggest API is pretty limited -- it always returns just 10 most recent results (unless there are other query parameters that I don't know about.) So, you can't really "automatically" get what you want. Obviously, you could just run the script every 2 weeks and start tracking the results over time, but that's probably not what you're after.

The closest equivalent thing I can think of would be to use the search api -- it won't give you what people are searching for, but it will at least tell you the top things people find for a given phrase. I wrote a piece on it called Simplify business research with Google Ajax Search API, but there are a lot of ther resources out there. That API has a wide set of parameter you can tweak.

Redirecting to a text file is pretty easy. Assuming you've saved the script using the name "suggest.py," you can drop into a terminal and enter this command:

python suggest.api > out.txt


This will redirect the output into a tab-delimited text file. From there, you can import it into excel and do the sorting and searching.

Hope this helps!

ps -- to run this, you'll need a Python interpreter. If you're on a Mac, it's built-in, but on a PC, you might need to install one. ActivePython, from ActiveState, is a pretty good all around choice because they've done a great job making a nice, installable system. Good luck!
 : Sep 09 2010 12:03 PM
Andrew,

Thanks for the help, much appreciated. I ran the script in Automator on the Mac (Snow Leopard) to play with it rather than from the command-line. I will need to learn a little more before I can make use of the APIs. Thanks for the suggestions.

One last thing, can the output be directed to a text file from within the script?

Thanks again,
George
0
  odewahn1's Photo
Posted Sep 10 2010 02:22 AM

George --

You can use basic file I/O to write the results out -- try this modified version should do it, which (should!) save the results to a file called "out.txt" on your Desktop.

Andrew

from sgmllib import SGMLParser
import urllib2
import urllib

# Define the class that will parse the suggestion XML
class PullSuggestions(SGMLParser):

   def reset(self):
      SGMLParser.reset(self)
      self.suggestions = []
      self.queries = []

   def start_suggestion(self, attrs):
      for a in attrs:
         if a[0] == 'data': self.suggestions.append(a[1])

   def start_num_queries(self, attrs):
      for a in attrs:
         if a[0] == 'int': self.queries.append(a[1])


# -----------------------------------------------------------
# Enter your phrase here.  Be sure to leave the %s at the end!
# -----------------------------------------------------------
base_query = "how do i %s"  #This is the base query
f = open( '~/Desktop/out.txt','w')

alphabet = "abcdefghijklmnopqrstuvwxyz"
for letter in alphabet:
   q = base_query % letter;
   query = urllib.urlencode({'q' : q})
   url = "http://google.com/complete/search?output=toolbar&%s" % query

   res = urllib2.urlopen(url)
   parser = PullSuggestions()
   parser.feed(res.read())
   parser.close()

   for i in range(0,len(parser.suggestions)):
      f.write("%s\t%s\n" % (parser.suggestions[i], parser.queries[i]))

f.close()

 : Sep 17 2010 11:10 PM
I have to say, your script is great! However, I just wonder how can I change the key word by sending it by get method? For example, it shows the result when I type /?q=how+do+i

thanks very much!
0
  odewahn1's Photo
Posted Sep 22 2010 03:44 AM

Hi, snail000. I'm not quite sure I understand your question, but here's a shot. If you want to change the keyword, you can change the string in the base_query variable. (It comes just before the big loop that does the work.) If that's not your question, can you clarify?
0
  aarans's Photo
Posted Dec 24 2010 12:49 PM

Nice work.

The script exits when it encounts a string that doesn't have any suggestions.

eg : if there is no suggestion for "how do i q" the output stops there.
Is there any way to fix that?