How does the Google search algorithm work? From crawling to ranking

Have you ever wondered what exactly happens in the milliseconds after you type a search query into Google and before you receive a perfectly sorted list of answers? It seems like magic, but it’s the result of one of the most complex systems in the digital world: the Google search algorithm.

Think of it as the brain of the internet, managing the world’s largest library. Before you can find a book, the librarian needs to know which books exist, what they’re about, and where they are. That’s exactly what Google does - only at a speed and scale unimaginable to humans.

In this article, I decode this process step by step. I’ll show you how Google evolved from a pure keyword machine to an AI-powered semantic search engine. Through the Google API Documentation Leaks 2024 - first published by Rand Fishkin (SparkToro) and technically analyzed by Mike King (iPullRank) - we now have more insight than ever before. Get ready to open the black box.

The Google search algorithm is not a single piece of code but a complex system of countless individual algorithms and machine learning processes. Each part of this system has a specific task that contributes to finding the best possible answer to your question from billions of web pages.

The Google API leaks at Search Engine Land provided the first look behind the curtain, revealing internal system names like Ascorer, NavBoost, Mustang, and Twiddler - terms Google had never used publicly. Key interpretation work on these systems was done by Mike King (iPullRank), Julian Redlich (rankmeamadeus.com), and Mario Fischer (Search Engine Land).

The system processes over 8.5 billion search queries daily and must deliver relevant results from an index of hundreds of billions of web pages in fractions of a second. A technical feat that is constantly being refined.

The entire Google Search process is divided into five clearly defined main phases. Each phase builds on the previous one and is critical for what you see on the search engine results page (SERP).

How heavily Google now intervenes in how results are displayed is clear from the AI-generated titles in search results - and the direction search is heading was spelled out most recently at Google Search Central Live Toronto 2026.

For detailed information on Google’s complete search process, see the official Google guide: How Search Works.

What Is Crawling?

Crawling is the process where automated programs (crawlers or spiders) systematically search the internet. Google’s primary crawler is the Googlebot. It follows links from already known pages to discover new ones - much like how you click from one article to the next.

With over 62% of website traffic coming from mobile devices (Source: Statista 2025), the mobile user experience has become the standard. Googlebot primarily uses the smartphone version for crawling (Mobile-First Indexing).

How Does Googlebot Discover URLs?

• Links from already known pages: When page A links to page B, Googlebot follows that link.
• XML Sitemaps: A list of all important URLs on your website that you can submit in Google Search Console.
• URL submission: Manually requesting indexing of individual URLs through Search Console.
• External links (backlinks): References from other websites to your page accelerate discovery.

What Is the Crawl Budget?

Not every page gets crawled equally often. Google distributes its resources according to the crawl budget - a combination of crawl capacity (how many URLs Google can crawl without overloading your server) and crawl demand (how important Google considers your URLs).

Large, high-quality websites with fast servers receive more budget. Factors like duplicate content, soft 404 errors, or slow load times waste your budget.

Deep dive: You can find a detailed guide on optimization in my guide: Crawl Budget Optimization: How to Get Your Content Indexed Faster

Indexing is the process where information found during crawling is analyzed, understood, and stored in a massive database - the Google Index. Only indexed pages can appear in search results. You can find an in-depth analysis of both processes in my article: Crawling & Indexing: How Google Finds and Stores Your Content.

What Is Caffeine?

Caffeine is Google’s indexing infrastructure, introduced in 2010. It is not a ranking change but a complete architecture redesign. Before Caffeine, the index was updated in large batches - today this happens continuously and incrementally.

“Caffeine provides 50 percent fresher results for web searches than our last index, and it is the largest collection of web content we have offered.”

- Google Search Central Blog (2010)

The Index Tiers: Base, Zeppelins, Landfills

The Google API leaks revealed that the index is structured in three tiers - a detail Google had never publicly communicated. The tier names themselves (Base, Zeppelins, Landfills) come from the leaked API documents. The mapping to specific storage media is an interpretation by Mike King (iPullRank), based on references to Google’s TeraGoogle infrastructure in the leak documents:

• Base (likely flash storage, per King’s analysis): The most important, frequently updated content. Fastest access, highest priority. This is where you want to be!
• Zeppelins (likely SSDs): Content of medium importance and freshness. Solid performance, but not top priority.
• Landfills (likely HDDs): Rarely updated or less important content. Slowest access, lowest priority.

The days when Google was a pure keyword machine are long gone. Today, the search engine wants to understand what you really mean - your intent, the context, and the meaning behind your words.

The Shift from Keywords to Concepts

This fundamental shift was driven largely by the Hummingbird update in 2013. Since then, Google understands that someone searching for “best pizza place nearby” doesn’t just want a list of websites, but concrete, local restaurant suggestions with reviews, opening hours, and directions.

What Is the Google Knowledge Graph?

The Knowledge Graph is Google’s massive knowledge database about entities (people, places, things, concepts) and their relationships. Google doesn’t just know that “Albert Einstein” is a string - it knows he was a person, a physicist, born in Ulm, connected to the theory of relativity, and won the Nobel Prize.

This networked knowledge enables direct answers in Knowledge Panels, Featured Snippets, and AI Overviews - without the user needing to visit a website.

How AI Revolutionized Language Understanding

BERT (since 2019)

BERT (Bidirectional Encoder Representations from Transformers) revolutionized language understanding. It reads text bidirectionally - in both directions simultaneously - and thereby understands the context of small words like “for,” “without,” or “not” that were previously often ignored.

Example: For the search query “can I pick up medication FOR someone,” BERT understands that you want to know if you can pick up medication for another person - not for yourself. Before BERT, the small word “for” was often ignored, leading to irrelevant results. (This example comes from Google’s official BERT announcement by Pandu Nayak, 2019.)

Gemini 2.5 (2025)

The latest milestone is Google’s Gemini 2.5 AI, used in both the new AI Mode and traditional search results. Gemini can process multimodal queries (text, image, and video simultaneously) and has advanced reasoning capabilities that enable complex multi-step thinking processes.

What About MUM?

Important: MUM is NOT used for general ranking! Despite many claims in the SEO community, MUM (Multitask Unified Model) is currently only used for specific applications - such as COVID vaccination information and certain Featured Snippets. Google itself confirms in its official Ranking Systems Guide: “MUM is not currently used for general ranking in Search.” This is one of the most common SEO myths of 2025.

Ranking is the heart of Google Search - this is where it’s decided which pages appear at position 1, 2, or 3 and which disappear to page 10.

What Is Ascorer? The Primary Ranking Algorithm

Ascorer likely stands for “Amit’s Scorer” - named after Amit Singhal, the former Head of Search. This derivation goes back to the analysis by Julian Redlich (rankmeamadeus.com, 2023), who first described the connection between the internal variable name and Singhal in the SEO community. Ascorer is the fundamental ranking algorithm that scores pages BEFORE all other adjustments by RankBrain, BERT, or Twiddlers are applied.

Ascorer is embedded in the Mustang system - Google’s central infrastructure for scoring, sorting, and serving search results (cf. Mike King’s leak analysis at iPullRank). Mustang processes hundreds of ranking signals, including:

• Content relevance to the search query
• Content quality and depth
• Backlink profile and PageRank
• Technical factors (load time, mobile-friendliness, Core Web Vitals)
• User signals and engagement metrics

The AI Layer: RankBrain and Neural Matching

RankBrain (since 2015) was Google’s first machine learning system in ranking. It translates words and phrases into mathematical vectors (word embeddings) to understand concepts rather than keywords. For a search query like “apex predator food chain,” RankBrain can recognize that you’re probably looking for related concepts. According to Google, around 15% of all daily search queries are completely new - RankBrain helps understand these too.

Neural Matching (since 2018) extends this approach. Danny Sullivan (Google) described it as a “super-synonym system”: it understands the conceptual connection between search queries and page content - even when the words don’t match. A page about “why is my TV acting weird” can thus rank for “TV color problems fix.”

A complete overview of all active ranking systems is available in Google’s official guide to ranking systems.

After Ascorer and the AI systems have done their work, the final adjustment comes through the Twiddler framework. This phase was only revealed through the API leaks and shows how complex the system truly is.

What Are Twiddlers?

Twiddlers are specialized re-ranking modules that run after the initial Ascorer evaluation. They act as the final authority and can:

• Adjust a document’s information retrieval score up or down
• Change a result’s position (promote or demote)
• Completely filter results or control SERP diversity
• Restrict categories (e.g., max 3 blog posts or 2 videos per SERP)

Known Twiddler Systems

Through the API leaks, we know some of the most important Twiddlers:

• NavBoost: Adjusts rankings based on user behavior and click logs (more in the next chapter)
• FreshnessTwiddler: Rewards fresh, current content for time-sensitive queries like news or events
• QualityBoost: Improves rankings for high-quality content based on quality signals
• SiteBoost: Promotes or demotes entire websites based on site-wide quality signals
• RealTimeBoost: Prioritizes breaking news and current events in real-time

Isolation principle: According to the leaked internal Twiddler Quick Start Guide, each Twiddler operates in isolation - without knowledge of other Twiddlers’ decisions. The framework then consolidates all recommendations into a final ranking. In 2018, over 65 Twiddlers were already in production - today likely well over 100, though the exact number is not publicly known.

The most important overarching ranking factor is content quality, which Google evaluates using the E-E-A-T concept. Practical implementation is crucial, so I recommend my detailed E-E-A-T: The Ultimate Guide for More Trust and Top Rankings.

What Do the Letters in E-E-A-T Stand For?

• Experience: Does the author demonstrate practical, lived experience with the topic? Have they actually tested the product, visited the location, experienced the situation?
• Expertise: Is the author a recognized expert in their field? Do they have relevant education, certifications, or demonstrable competence?
• Authoritativeness: Is the website and author considered an authoritative source for the topic? Are they cited and linked by other experts?
• Trustworthiness: Is the site secure, transparent, and trustworthy? Is there an imprint, privacy policy, secure connection?

“Trust is the most important member of the E-E-A-T family because untrustworthy pages have low E-E-A-T no matter how Experienced, Expert, or Authoritative they may seem.”

- Google Quality Rater Guidelines

The complete evaluation criteria can be found in the Google Quality Rater Guidelines (PDF). I analyzed the key changes from the latest update: Google Quality Rater Guidelines Update September 2025.

The Google API leaks confirmed something Google publicly denied for years: click data and user behavior play a massive role in ranking. You can find a detailed analysis of these revelations in my article: Google Leak: Why User Signals Matter More Than Everything You Thought You Knew About SEO.

What Is NavBoost?

NavBoost is a re-ranking system based on click logs and user behavior. The existence and importance of NavBoost was revealed not only through the API leaks but also confirmed by Pandu Nayak’s testimony in the US antitrust case against Google (2023). The leaks show that Google stores and analyzes 13 months of click data. The following metrics factor in:

• Click-Through Rate (CTR): How often is a result clicked for a specific query compared to other results?
• Long Clicks vs. Short Clicks: Does the user stay on the page for a long time (positive) or return to the SERP within seconds (negative)?
• Pogo-Sticking: Does the user bounce between multiple results? This is a strong negative signal.
• Dwell Time: How long does the user spend on the page before returning to search?
• ChromeUX data: Real user data from the Chrome browser also feeds into the evaluation.

What Does siteAuthority Mean?

The leaks also reveal a field called siteAuthority - although Google publicly denied for years that anything like “Domain Authority” exists. This signal apparently evaluates the overall quality and trustworthiness of a website at the domain level - not just individual pages.

Google updates its algorithm thousands of times per year to improve search results, adapt to new user behavior, and combat manipulation attempts. Most changes are small and barely noticeable - but a few “Core Updates” can shake up entire industries.

The Most Important Updates of 2025

My detailed analyses of the 2025 updates:

• Google June 2025 Core Update: Everything You Need to Know
• Google August 2025 Spam Update: What Webmasters Need to Know
• Google Core Update December 2025: Gift or Nasty Surprise?

You can track all official updates at the Google Search Status Dashboard. How to interpret SERP volatility and respond to updates is covered in: How the Semrush Sensor Helps You Master SERP Volatility and Google Updates.

What Is Google’s AI Mode?

In 2025, Google introduced AI Mode - a completely new way to search, powered by Gemini 2.5. AI Mode uses a “Query Fan-Out” technique: Google breaks down complex questions into subtopics, searches multiple sources in parallel, and creates AI-generated, comprehensive answers. The term “Query Fan-Out” was coined by Google’s VP of Search Liz Reid at Google I/O 2025.

The differences and SEO implications in detail:

• Google AI Overviews: How They Work Explained
• Google AI Mode & AI Overviews: The Difference Explained
• SEO in the Age of AI Browsers: Framework for the Citation Economy

What Changes with Core Web Vitals?

Since March 2024, Interaction to Next Paint (INP) is officially part of Core Web Vitals, replacing First Input Delay (FID) (Source: web.dev). The key difference: INP measures your website’s responsiveness to all user interactions throughout the entire visit - not just the first one.

Core Web Vitals Benchmarks 2026

More at the official Web Vitals documentation.

llms.txt - Preparing for AI Crawlers

With the rise of AI search systems, the question of how to optimize your content for AI crawlers becomes relevant. The llms.txt format was proposed in September 2024 by Jeremy Howard (Answer.AI) and provides a structured way to give AI systems information about your website. It works similarly to robots.txt, but specifically for Large Language Models: llms.txt Explained: Step-by-Step Guide & Practical Example.

Understanding the Google algorithm means recognizing that there are no simple tricks anymore. The days when keyword stuffing and link spam could manipulate rankings are definitively over. Modern SEO success in 2026 is built on four fundamental pillars:

1. Technical Excellence: Ensure flawless crawlability and indexability, pay attention to Core Web Vitals (especially INP < 200ms), and use structured data. A technically sound website is the foundation.
2. Semantic Relevance: Create content that covers topic areas deeply and comprehensively. Answer your target audience’s questions better, more completely, and more helpfully than anyone else.
3. Absolute Trustworthiness: Build your personal brand, demonstrate your real expertise and experience, and become an authority that both users and Google trust (E-E-A-T).
4. AI Optimization: Prepare your content for AI Mode and AI Overviews. Structure information clearly, answer questions directly and comprehensively, and become a citable source.