{"id":442782,"date":"2024-12-24T15:07:18","date_gmt":"2024-12-24T15:07:18","guid":{"rendered":"http:\/\/savepearlharbor.com\/?p=442782"},"modified":"-0001-11-30T00:00:00","modified_gmt":"2024-12-24T15:07:18","slug":"","status":"publish","type":"post","link":"https:\/\/savepearlharbor.com\/?p=442782","title":{"rendered":"Parsing Google\u2019s Search Results by Keywords: A Detailed Guide to Building a Free Google Parser<\/span>"},"content":{"rendered":"
<\/div>\n
\n
\n
\n
\n

Any SEO expert knows the pain of collecting Google keyword data. It\u2019s one thing if you can count all the queries on one hand, but what if they number in the thousands? How do you check the search volume in Google for each keyword? Frankly, once you hit tens of thousands of keywords, it\u2019s enough to make your head spin. You\u2019ll be tempted to reach for outdated, familiar tools, only to find modern reality throwing a curveball: the old formula of Key Collector + Google Ads + a few proxies simply doesn\u2019t cut it anymore. We\u2019re entering a new era, and without direct access to the official API, things get grim and complicated fast. <\/p>\n

<\/figure>\n

But every cloud has a silver lining. If you\u2019re ready to dive into coding acrobatics, a bit of ritual dancing, and a dash of Google bureaucracy, you can likely snag direct access to the trove of data within the Keyword Planner API\u2014a goldmine for SEO enthusiasts craving a real \u201cGoogle Wordstat\u201d (admit it, we all secretly want that). In this article, I\u2019ll show you a step-by-step guide for ditching overpriced external services (300 bucks a month\u2014really?!) in favor of building your own tool on top of the Google Ads API. Let\u2019s roll!<\/p>\n

<\/figure>\n

And yes, there\u2019s no actual \u201cGoogle Wordstat\u201d out there\u2014those stories are SEO campfire tales. Sure, you could force-fit the Google Keyword Planner into that role, but they\u2019re too different to compare side by side\u2026<\/p>\n

A Google Parser, or Why You Need Access to the Google Keyword Planner <\/h2>\n

Let\u2019s broaden our view. If you\u2019re looking to assemble a complete Google semantic core, you\u2019ll need a decent batch of queries (it\u2019s definitely more than 20!). Key Collector once did the trick but is less helpful now. This situation often forces people onto paid equivalents like Keyword Tool\u2014costly services that don\u2019t really stand out, other than by replicating Google Ads data through their own interface or API. But we want direct access and the ability to automate our process however we see fit.<\/p>\n

Though Key Collector may be less relevant for extracting new data, you can still use it to process an existing semantic core: it does a solid job with sorting, cleaning, clustering, and integrating with external services. You likely won\u2019t need residential proxies or CAPTCHA-solving services for that; a tool like KeySo might be handy, but that\u2019s another story.<\/p>\n

<\/figure>\n<\/p>\n

Keyword Tool itself requires a hefty fee, and Ubersuggest is basically a lightweight spin on Keyword Tool\u2014lightweight because it doesn\u2019t offer an API.
Long story short: if you want an official solution that lets you code your own data-gathering logic, your best bet is the Google Ads API\u2014preferably doing it yourself rather than paying for middleman services that run $300\u2013$1,000 a month.<\/p>\n

Building a Google Parser via the API: Where It All Begins\u2014Your Developer Token and Ad Account <\/h2>\n

Want to pull in 40,000+ keywords? Go for it! But first, there\u2019s some formal stuff to handle. Sure, parts of this might feel like red tape, but once you get through it, you\u2019ll be rewarded with the keyword data stash you once only dreamed of.<\/p>\n

<\/figure>\n<\/p>\n

You need a Google Ads account\u2014not an empty one, but one that\u2019s actually spent at least some ad budget. Without real expenses, the Keyword Planner stays locked down. Getting a functioning ad account is beyond the scope of this article, but it\u2019s required if you want to access the Keyword Planner as the backbone of your parser.<\/p>\n

Create a Manager Account. Go to the Manager Accounts (MCC) page, sign up, and once you\u2019re in settings, check the left-hand sidebar for \u201cAPI Center.\u201d That\u2019s where you\u2019ll find your Developer token\u2014grab it and note it down.<\/p>\n

Google Cloud Console: Client ID and Client Secret\u2014You Can\u2019t Parse SERPs Without These<\/h4>\n

Moving on. You\u2019ll need to authenticate with the Google Ads API using OAuth, which means registering an application in the Google Cloud Console:<\/p>\n

<\/figure>\n
    \n
  1. \n

    Sign into the Google Cloud Console under the same Google account that manages your ad account.<\/p>\n<\/li>\n

  2. \n

    Create a new project (name it whatever you like).<\/p>\n<\/li>\n

  3. \n

    In the left-hand menu, go to APIs & Services \u2192 Library<\/strong>.<\/p>\n<\/li>\n

  4. \n

    Search for Google Ads API<\/strong>, then click Enable<\/strong>.<\/p>\n<\/li>\n

  5. \n

    Then go to APIs & Services \u2192 Credentials<\/strong>.<\/p>\n<\/li>\n

  6. \n

    Click \u201cCreate Credentials\u201d \u2192 \u201cOAuth client ID.\u201d<\/strong><\/p>\n<\/li>\n

  7. \n

    Choose the application type \u201cWeb application.\u201d<\/strong><\/p>\n<\/li>\n

  8. \n

    Give it a name (maybe \u201cmy-ads-api,\u201d or anything else).<\/p>\n<\/li>\n

  9. \n

    Under Authorized redirect URIs<\/strong>, add:<\/p>\n<\/li>\n<\/ol>\n

    http:\/\/localhost:8081\/ http:\/\/localhost:8081<\/code><\/pre>\n
    10. Hit \u201cCreate.\u201d<\/strong> You\u2019ll get two important values: Client ID<\/strong> and Client Secret<\/strong>.    <\/p>\n
    Concerning those two nearly identical redirect URIs: I experimented with multiple setups. The one with the trailing slash worked for me, but I kept the version without the slash just in case it might work differently for someone else.<\/p>\n
    Getting a refresh_token for Google Ads\u2014The Key to a Functional Google Parser  <\/h2>\n
    So your code can automatically call Google Ads, it needs a refresh_token<\/strong>\u2014a long-lived key that updates session tokens behind the scenes without manual logins.
    Let\u2019s generate it (and by the way, this took me the longest, aside from Google\u2019s official approval process, but more on that shortly).  <\/p>\n
    <\/figure>\n
      \n
    1. \n
      Install the Python library:  <\/p>\n<\/li>\n<\/ol>\n
      pip install google-ads<\/code><\/pre>\n
      Create a .py file (e.g., get_refresh_token.py<\/code>) and paste this code:  <\/p>\n
      import logging from google.auth.transport.requests import Request from google_auth_oauthlib.flow import InstalledAppFlow  logging.basicConfig(level=logging.DEBUG)  def generate_refresh_token(client_id, client_secret):     scopes = [\"https:\/\/www.googleapis.com\/auth\/adwords\"]      flow = InstalledAppFlow.from_client_config(         {             \"installed\": {                 \"client_id\": client_id,                 \"client_secret\": client_secret,                 \"auth_uri\": \"https:\/\/accounts.google.com\/o\/oauth2\/auth\",                 \"token_uri\": \"https:\/\/oauth2.googleapis.com\/token\",                 \"redirect_uris\": [\"http:\/\/localhost:8081\/\"]  # Notice the trailing slash             }         },         scopes,     )      auth_url, state = flow.authorization_url(         access_type=\"offline\",         include_granted_scopes=\"true\",         prompt='consent'     )     print(f\"Authorization URL: {auth_url}\")      credentials = flow.run_local_server(port=8081, state=state)     print(f\"Your Refresh Token: {credentials.refresh_token}\")  generate_refresh_token(     \"Client ID\",     \"Client secret\" )<\/code><\/pre>\n
      Don\u2019t forget to insert your Client ID<\/code> and Client Secret<\/code> at the bottom of the file.<\/p>\n
      Run:  <\/p>\n
      python get_refresh_token.py<\/code><\/pre>\n
        \n
      1. \n
        In the console, you\u2019ll see an authorization link. Open it in your browser (it might open automatically, but if not, just copy and paste). Choose the ad account you want, and when it\u2019s done, your console will show a refresh_token<\/strong>. Copy and save it somewhere safe.<\/p>\n<\/li>\n<\/ol>\n
        The google-ads.yaml Config File\u2014The Core of a Functional Google Parser  <\/h2>\n
        What was all the fuss for? To feed the data into a config file that your keyword-collection script will rely on. Let\u2019s build it with the info we gathered. Create or edit a file called google-ads.yaml<\/code>:<\/p>\n
        developer_token: \"YOUR_DEVELOPER_TOKEN\" client_id: \"YOUR_CLIENT_ID\" client_secret: \"YOUR_CLIENT_SECRET\" refresh_token: \"YOUR_REFRESH_TOKEN\" login_customer_id: \"YOUR_MANAGER_ACCOUNT_ID\"  # Ad account ID<\/code><\/pre>\n
        Keep in mind: login_customer_id<\/code> should be your actual ad account ID (the one where your ads run), not your MCC\u2019s ID. Don\u2019t mix them up!
        I haven\u2019t tested using the manager account ID. If you do that, make sure it\u2019s the same ID used in your main script (also without dashes, just digits in a row).<\/p>\n
        Getting a Higher Access Level (Otherwise, Your Google Parser Remains a Pipe Dream)<\/h4>\n
        Now for the main gotcha: even if you do everything above, you\u2019ll run into Test Access<\/strong> restrictions. To retrieve real keyword data, you need to bump it up to Basic Access<\/strong>. Here\u2019s how:<\/p>\n
          \n
        1. \n
          Go back to the API Center<\/strong> of your manager account (MCC).<\/p>\n<\/li>\n
        2. \n
          You\u2019ll see you have Test Access<\/strong>. Click that status\u2014there should be an option to Apply for Basic access<\/strong>.<\/p>\n<\/li>\n
        3. \n
          Fill out the form, explaining that you\u2019re using the API for your own advertising analysis or internal automated data gathering.<\/p>\n<\/li>\n
        4. \n
          Wait for approval (up to 3 days).<\/p>\n<\/li>\n<\/ol>\n
          If Google finds your request valid, you\u2019ll get access. In most cases, if your ad account is genuinely active, you\u2019re golden.
          One more important note: your manager account must be linked to your ad account. Head to \u201cAccounts\u201d in the left menu; if your ad account isn\u2019t there, click Add<\/strong> and enter the account number you want to link. You\u2019ll get an email notification\u2014confirm it, and you\u2019re all set.<\/p>\n
          At Last, Collecting Keywords: A Python Script for Parsing Google  <\/h2>\n
          Below is a snippet for gathering data via the official Google Ads Keyword Planner API. Briefly, the script reads a list of keywords from a CSV, splits them into batches of 10, sends them to the API for monthly traffic stats (average searches, competition, bid ranges), then saves everything into another CSV file:<\/p>\n
          import csv import time from google.ads.googleads.client import GoogleAdsClient  def chunk_list(lst, n):     for i in range(0, len(lst), n):         yield lst[i:i+n]  def main():     # Load the client from the config file     client = GoogleAdsClient.load_from_storage(\"google-ads.yaml\")          # Keyword ideas service     keyword_plan_idea_service = client.get_service(\"KeywordPlanIdeaService\")          # Put your Customer ID here (no dashes, e.g. \"1234567890\")     customer_id = \"YOUR_CUSTOMER_ID\"          # Read keywords from a CSV     keywords = []     with open(\"keywords.csv\", \"r\", encoding=\"utf-8\") as f:         reader = csv.DictReader(f)         for row in reader:             kw = row['keyword'].strip()             if kw:                 keywords.append(kw)          chunk_size = 10      with open(\"keyword_data.csv\", \"w\", newline=\"\", encoding=\"utf-8\") as outfile:         writer = csv.writer(outfile)         writer.writerow([             \"keyword\",             \"avg_monthly_searches\",             \"competition\",             \"low_top_of_page_bid_micros\",             \"high_top_of_page_bid_micros\"         ])                  for chunk in chunk_list(keywords, chunk_size):             # Build the request             request = client.get_type(\"GenerateKeywordIdeasRequest\")             request.customer_id = customer_id              # Add language and geo:             # France             request.geo_target_constants.append(\"geoTargetConstants\/2250\")             # French             request.language = \"languageConstants\/1010\"              # Insert the keywords             request.keyword_seed.keywords.extend(chunk)              # Send the request             response = keyword_plan_idea_service.generate_keyword_ideas(request=request)                          for idea in response.results:                 text = idea.text                 metrics = idea.keyword_idea_metrics                 avg_searches = metrics.avg_monthly_searches if metrics.avg_monthly_searches else 0                 competition = metrics.competition.name if metrics.competition else \"UNSPECIFIED\"                 low_bid = metrics.low_top_of_page_bid_micros if metrics.low_top_of_page_bid_micros else 0                 high_bid = metrics.high_top_of_page_bid_micros if metrics.high_top_of_page_bid_micros else 0                                  writer.writerow([                     text,                     avg_searches,                     competition,                     low_bid,                     high_bid                 ])              # Pause to avoid hitting API limits             time.sleep(1)  if __name__ == \"__main__\":     main()<\/code><\/pre>\n
          How It Works  <\/h2>\n