Google Search Results in Clojure

This Clojure Library is meant to scrape and parse results from Google, Bing, Baidu, Yandex, Yahoo, Ebay and more using SerpApi.

The following services are provided:

• Search API
• Location API
• Search Archive API
• Account API

SerpApi.com provides a script builder to get you started quickly.

Installation

Clojure must be already installed.

Follow these instructions to set up Clojure

Quick start

(require '[serpapi.core :as sc])
(def edn-results (sc/edn-search {:q "coffee", :api_key "secret_api_key"})

This example runs a search about "coffee" using your secret api key.

The SerpApi.com service (backend)

• searches on Google using the search: q = "coffee"
• parses the messy HTML responses
• return a standardized JSON response The function sc/edn-search
• Format the request to SerpApi.com server
• Execute GET http request
• Parse JSON into EDN using the Cheshire library Et voila..

Alternatively, you can search on other search engines by including the :engine keyword and appropriate value in the to map provided as argument to edn-searchor any other search function. The following values are supported:

:bing :baidu :yahoo :yandex :ebay

All functions support both strings and keywords for map keys as well as any keyword sets mentioned in this document.

For example the usage below is also supported

(require '[serpapi.core :as sc])
(def edn-results (sc/edn-search {"q" "coffee", "api_key" "secret_api_key" "engine" "baidu"})

See the playground to generate your code.

Summary

• Google Search Results in Clojure
  • Installation
  • Quick start
• Summary
  • Guide
    • How to set the private API key
    • Search API capability for Google
    • Example by specification
    • Location API
    • Search Archive API
    • Account API

Guide

How to set the private API key

The api_key can be set globally using the set-api-key function.

(require '[serpapi.core :as sc])
(sc/set-api-key "secret-api-key")
(def result (sc/edn-search {:q "coffee"}))

or api_key can be provided for each search.

(def edn-results (sc/edn-search {:q "coffee", :api_key "secret_api_key"})

To get the key simply copy/paste from serpapi.com/dashboard.

Search API capability for Google

(def search-params  {
  :q "search",
  :google_domain "Google Domain", 
  :location "Location Requested", 
  :device "desktop", ;; #{"desktop" "mobile" "tablet"}
  :hl "Google UI Language",
  :gl "Google Country",
  :safe "Safe Search Flag",
  :num "Number of Results",
  :start "Pagination Offset",
  :api_key "private key", ;; copy paste from https://serpapi.com/dashboard
  :tbm "nws" ;; #{"nws" "isch" "shop"},
  :tbs "custom to be search criteria"
  :async true ;;  boolean, allow async
}

;; return search results in edn
(def search (sc/edn-search search-params))

;; return search results as raw html
(def html-search (sc/html-search search-params))


;; search as raw JSON format
(def json-search (sc/json-search search-params))

;; or use the base search function and set the options there
(def base-search (sc/search (assoc search-params :output "json")))

Full Documentation

More search APIs are documented on SerpApi.com.

You will find more hands on examples below.

Example by specification

We love true open source, continuous integration and Test Drive Development (TDD).

The directory test/ includes specification/examples.

To run the test:

bin/kaocha

Location API

(def location-list  (sc/locations {:q "Austin", :limit 3}))

it prints the first 3 location matching Austin (Texas, Texas, Rochester)

[{:id "585069bdee19ad271e9bc072",
  :google_id 200635,
  :google_parent_id 21176,
  :name "Austin, TX",
  :canonical_name "Austin,TX,Texas,United States",
  :country_code "US",
  :target_type "DMA Region",
  :reach 5560000,
  :gps [-97.7430608, 30.267153],
  :keys ["austin", "tx", "texas", "united", "states"]},
  ...]

Search Archive API

This API allows to retrieve previous search. To do so run a search to save a search_id.

(def search (sc/edn-search {:q "Coffee", :location "Portland"}))
(def search-id (get-in search [:search_metadata :id]))

Now let's retrieve the previous search from the archive.

(def archive-search (sc/search-archive search-id))
(println archive-search)

it prints the search from the archive.

Account API

(def account-search (account))
(println account-search)

it prints your account information.