With a little SOAP, cleanliness is next to Googliness.
For the past few months, we've been looking at a number of Web services offered by Amazon, allowing us to search through its catalog with relative ease. Amazon decided several years ago to make its Web services largely free, on the assumption that this would raise the number of people eventually buying from its Web site. And indeed, a large number of developers now use Amazon Web services to create everything from custom bookstores to programs that can help with bookstore management.
Amazon isn't the only commercial Web site that has opened up its catalog to the outside world. Google, another 900-pound Internet gorilla, also released its Web APIs several years ago. These APIs make it possible to search through Google's extensive catalog of Web content. It's impossible to know whether this catalog is the largest in the world, but from my perspective, that's somewhat irrelevant. Google's catalog is large enough, and is updated frequently enough, for me to rely on it as my primary search engine most of the time.
Google has made a number of different APIs available over the last few years. This month, we look at the simplest of them, for performing basic searches of the Web archive. We examine how Google uses WSDL (Web service description language) to advertise its Web services and how we can make SOAP calls to search through Google's extensive library for our own purposes.
If you have worked with Amazon Web services, getting started with Google's APIs will not surprise you a great deal. To begin, both companies require that you register to use their services. Registration is free in both cases and provides you with an identification key that is placed in every request to the server.
To obtain a Google key, you first need to register for a Google account. Now, I've had a “Google account” for some time, for use with services such as Gmail and its personalized news page. However, it seems the APIs are linked to a different set of accounts. The fact that I had to register and log in to the API system, even after initially logging in to my “main” Google account, struck me as a bit odd.
That said, creating an account is simple and straightforward. Go to the main Google API page (www.google.com/apis), click on create a Google account, and fill out the form. Soon after submitting the HTML form, you will receive e-mail from Google confirming the creation of your account and containing your Google key, along with a URL to visit in order to confirm the account's creation. After confirming the creation of your account, you're ready to move forward with the use of your Google key, creating programs that take advantage of Google's Web services.
Before we do that though, we should consider the restrictions that Google places on the service and the data we retrieve through it. Amazon allows participants to make only one API call per second, which means a maximum of 86,400 calls in a given 24-hour period. Google, by contrast, allows users to make only 1,000 calls in a given 24-hour period.
Moreover, the way in which these maximums are defined indicates the way in which violations will be handled. Google will return an error message if you have made more than 1,000 queries in the previous 24 hours, whereas Amazon will complain only if a query comes within one second of a previous query. Neither service keeps track of these numbers before returning an error message, but it is obviously easier to recover from violating Amazon's restrictions (by sleeping for one second and retrying) than Google's (as the program might need to sleep for up to 24 hours before retrying).
There are a number of legal differences between the two sites' services. Amazon pioneered the idea of affiliate vendors on the Web, encouraging people to create commercial services around its database. By contrast, Google explicitly states that users are forbidden from creating a commercial service around its search results. (If you are interested in creating a commercial service based around Internet search data, consider looking at Amazon's Alexa Web search platform service, which doesn't have these restrictions. At the same time, it'll cost you 25 cents for every 1,000 requests, which can add up quickly for a popular site.)
Finally, there are some technical differences between the two sites. Amazon's APIs work via both SOAP and REST, allowing developers to choose between these two formats. Google, by contrast, provides only a SOAP interface to its search engine. So, in order to create our search system, we need to install and use a SOAP client library. Fortunately, most languages have high-level libraries that allow for SOAP calls.
SOAP, formerly the Simple Object Access Protocol, but now an acronym that officially doesn't stand for anything, provides a relatively easy method for sending an XML-encapsulated query to a server. The server then responds with an XML-encoded response. Over the years, SOAP has strayed far from its simple roots. Although SOAP is still easier to understand, implement and work with than some more complicated protocols (such as CORBA), it is more difficult than most people would like to admit. If I can get away with it, I personally prefer to use XML-RPC for Web services. Although XML-RPC doesn't offer all of the features of SOAP, it is far easier to work with.
That said, Google requires that we use SOAP, and with many good SOAP client libraries available nowadays, we should not be afraid to work with it. Perl programmers have a particularly strong implementation, known as SOAP::Lite, at their disposal. For the programming examples in this article, we use Perl and SOAP::Lite. Note that the Lite part of the module name describes the ease with which programmers can implement Web services, not a stripped-down version of SOAP. You can install the latest version of SOAP::Lite from CPAN by typing:
perl -MCPAN -e 'install SOAP::Lite'
The SOAP::Lite installation will ask you to indicate which tests, if any, you want to perform before installing the module. I normally accept the defaults, but you might want to add to or remove from these depending on your needs.
With SOAP::Lite installed, it's time to write a program that queries Google. But to do that, we need to know the URL of the service, as well as the method that we will be invoking on Google's computer, along with the names and types of any parameters we want to send. We could specify these by hand, but that would mean a lot of work on our part. Moreover, Google currently expects SOAP requests to be pointed at api.google.com/search/beta2. If Google ever decides to change that URL without warning, many people might be surprised and upset.
Luckily, Google has provided a WSDL file, describing the services offered via Google's APIs, as well as the request and response parameters the system accepts. It also describes the endpoint for queries, allowing Google (in theory) to make changes to the service without notifying developers in advance. Of course, this assumes that the WSDL file itself will remain in the same location. It also assumes that the names of the services will not change, and that each of them is documented somewhere, because the choice of which method to invoke still requires human intervention.
WSDL is written in XML, and it is fairly easy to understand, once you realize that it's describing nothing more than the various Web services available on a particular server, including the number, names and types of inputs. Thus, the WSDL entry for doGoogleSearch, which performs the basic Google search of Web content, is defined as follows:
<message name="doGoogleSearch"> <part name="key" type="xsd:string"/> <part name="q" type="xsd:string"/> <part name="start" type="xsd:int"/> <part name="maxResults" type="xsd:int"/> <part name="filter" type="xsd:boolean"/> <part name="restrict" type="xsd:string"/> <part name="safeSearch" type="xsd:boolean"/> <part name="lr" type="xsd:string"/> <part name="ie" type="xsd:string"/> <part name="oe" type="xsd:string"/> </message>
To use WSDL from within a Perl program using SOAP::Lite, we invoke SOAP::Lite->service with the WSDL file's URL. If the file resides on the local filesystem, make sure that the URL begins with file:. For example:
my $google_wsdl = "http://api.google.com/GoogleSearch.wsdl"; my $query = SOAP::Lite->service($google_wsdl);
SOAP::Lite is then smart enough to look through the WSDL and make all of the advertised methods dynamically available, such that we can do the following:
my $results =
$query->doGoogleSearch($google_key,
$query_string,
$starting_page,
$max_results,
$filter,
$geographic_restriction,
$safe_search,
$language_restriction, 'utf-8', 'utf-8');
Do you see what happened here? There is a one-to-one mapping between the inputs described in the WSDL and the parameters that we pass to $query->doGoogleSearch().
We have now seen the core of our Google search program written in Perl. All that's left is to review the input parameters and the contents of $results, which contains the results returned from Google.
The documentation for the API at www.google.com/apis/reference.html describes the input parameters. All of them are mandatory, but some of them are more important than others. In particular, the Google key and the query string typically will be set, and the others will be set with simple default values, as you can see in Listing 1.
Most people, including myself, typically want to query the widest possible number of Web pages with our queries; however, there are times when it is more appropriate to retrieve data only from servers in a particular geography or in a certain language. The fact that Google's API makes this possible and straightforward opens the door for many different interesting applications.
Just as we send a query to Google via SOAP-encoded XML, we receive a result in SOAP-encoded XML. But as SOAP::Lite shielded us from having to write even a tiny bit of XML for the query, we similarly will be insulated when it comes to the response. The $results variable provides a Perl interface to the data that we received in response.
And exactly what data will we receive? To know that, we can look at the WSDL file once again. It indicates (among other things) that we will receive responses as a set of results, each of which looks like this:
<xsd:complexType name="ResultElement">
<xsd:all>
<xsd:element name="summary" type="xsd:string"/>
<xsd:element name="URL" type="xsd:string"/>
<xsd:element name="snippet" type="xsd:string"/>
<xsd:element name="title" type="xsd:string"/>
<xsd:element name="cachedSize" type="xsd:string"/>
<xsd:element name="relatedInformationPresent" type="xsd:boolean"/>
<xsd:element name="hostName" type="xsd:string"/>
<xsd:element name="directoryCategory" type="typens:DirectoryCategory"/>
<xsd:element name="directoryTitle" type="xsd:string"/>
</xsd:all>
</xsd:complexType>
In other words, each search result we receive back from Google (up to a maximum of ten) will provide all of the information we need to create a results page that looks just like Google's. Moreover, we can pick and choose the elements we want to display, showing (for example) only the title and the dmoz directory category and title. Or we can show a short snippet from the searched page. Or all of these. Or none of these.
doGoogleSearch is not the only method described in the WSDL file. There also are other methods, such as working with Google's cached pages and checking the spelling of individual words. When Web services were first unveiled to the public, a common example was that a word processor would now be able to call a remote Web service for spell-checking, rather than coming with a built-in system. That day is still far off in the future, but you can imagine using Google's API for an experimental version of such a service.
Moreover, we can use these outputs as inputs into another Web service call, either locally or remotely. Combining data from multiple sites is an increasingly popular thing to do, especially when combined with Google's maps API. It's amazing to see what can happen when you combine services in this way—something that we will explore in the coming months.
This month, we took a brief look at Google's search API. Using some simple tools, including the SOAP::Lite module for Perl, we were able to build a simple command-line version of Google's search page. In coming months, we'll look at Google's map API and begin to see how we can create mashup services that combine multiple data sources.
The code for this article is available at ftp.linuxjournal.com/pub/lj/listings/issue145/8866.tgz.
Resources for this article: /article/8881.