How does the Google search algorithm work? From crawling to ranking

Have you ever wondered exactly what happens in the milliseconds after you enter a search query into Google and before you receive a perfectly sorted list of answers? It seems like magic, but it is the result of one of the most complex systems in the digital world: the Google Search Algorithm.

Think of it as the brain of the internet, managing the world’s largest library. Before you can find a book, the librarian must first know which books exist, what they are about, and where they are located. This is exactly what Google does—but at a speed and scale unimaginable to humans.

In this post, I will decode this process step-by-step. I’ll show you how Google evolved from a pure keyword machine into an AI-driven semantic search engine. Thanks to the 2024 Google API Documentation Leaks, we have more insight today than ever before. Get ready to open the black box.

The Google Search Algorithm is not a single piece of code, but a complex system of countless individual algorithms and machine learning processes. Each part of this system has a specific task that contributes to finding the best possible answer to your question among billions of web pages.

The Google API Leaks shared by Search Engine Land allowed a behind-the-scenes look for the first time, revealing internal system names like Ascorer, NavBoost, Mustang, and Twiddler—terms Google had never used publicly.

The system processes over 8.5 billion search queries daily and must deliver relevant results from an index of hundreds of billions of web pages in fractions of a second. A technical masterpiece that is constantly being refined.

The entire Google Search process is divided into five clearly defined main phases. Each phase builds on the previous one and is crucial for what you see on the Search Engine Results Page (SERP).

Detailed information on Google’s entire search process can be found in the official Google Guide: How Search Works.

Before a page can rank, Google must know it exists. This first phase is the absolute foundation for online visibility.

What is Crawling?

Crawling is the process by which automated programs (called crawlers or spiders) systematically search the internet. Google’s primary crawler is the Googlebot. It follows links from already known pages to discover new ones—much like you click from one article to the next.

With over 62% of website traffic being mobile (Source: Statista 2025), the mobile user experience has become the standard. Googlebot primarily uses the smartphone version for crawling (Mobile-First Indexing).

How does Googlebot discover URLs?

• Links from known pages: If Page A links to Page B, Googlebot follows that link.
• XML Sitemaps: A list of all important URLs of your website that you can submit in the Google Search Console.
• URL Submission: Manually submitting individual URLs for indexing via Search Console.
• External Links (Backlinks): References from other websites to your page accelerate discovery.

What is Crawl Budget?

Not every page is crawled with the same frequency. Google distributes its resources according to the Crawl Budget—a combination of crawl capacity (how many URLs Google can crawl without overloading your server) and crawl demand (how important Google considers your URLs to be).

Large, high-quality websites with fast servers receive more budget. Factors like duplicate content, soft 404 errors, or slow loading times waste your budget.

Indexing is the process by which information found during crawling is analyzed, understood, and stored in a massive database, the Google Index. Only indexed pages can appear in search results. You can find an in-depth analysis of both processes in my article: Crawling & Indexing: How Google Finds and Stores Your Content.

What is Caffeine?

Caffeine is Google’s indexing infrastructure introduced in 2010. It is not a ranking change, but a complete redesign of the architecture. Before Caffeine, the index was updated in large batches—today, this happens continuously and incrementally.

“Caffeine provides 50 percent fresher results for web searches than our last index, and it is the largest collection of web content we have offered.” – Google Search Central Blog (2010)

The Index Tiers: Base, Zeppelins, Landfills

The Google API leaks revealed that the index is structured into three levels (Tiers)—a detail Google never publicly communicated:

• Base (Flash Memory): The most important, frequently updated content. Fastest access, highest priority. This is where you want to be!
• Zeppelins (SSDs): Content of medium importance and freshness. Solid performance, but not top priority.
• Landfills (HDDs): Rarely updated or less important content. Slowest access, lowest priority.

The days when Google was a pure keyword machine are long gone. Today, the search engine wants to understand what you really mean—your intent, the context, and the meaning behind your words.

The Shift from Keywords to Concepts

This fundamental shift was largely driven by the Hummingbird Update in 2013. Since then, Google has understood that someone searching for “best pizzeria nearby” doesn’t just want a list of web pages, but concrete, local restaurant suggestions with reviews, opening hours, and directions.

What is the Google Knowledge Graph?

The Knowledge Graph is Google’s massive knowledge base about entities (people, places, things, concepts) and their relationships to each other. Google doesn’t just know that “Albert Einstein” is a string of text—it knows he was a person, a physicist, born in Ulm, associated with the theory of relativity, and won the Nobel Prize.

This networked knowledge enables direct answers in Knowledge Panels, Featured Snippets, and AI Overviews—without the user needing to visit a website.

How AI Revolutionized Language Understanding

BERT (since 2019)

BERT (Bidirectional Encoder Representations from Transformers) revolutionized language understanding. It reads text bidirectionally—meaning in both directions simultaneously—and thus understands the context of small words like “for,” “to,” or “not,” which were often ignored in the past.

Find more details in the official BERT announcement from Google.

Gemini 2.5 (2025)

The latest milestone is Google’s Gemini 2.5 AI, which is used both in the new AI Mode and in classic search results. Gemini can process multimodal queries (text, image, video simultaneously) and possesses advanced reasoning capabilities that enable complex, multi-step thought processes.

What about MUM?

Ranking is the heart of Google Search—this is where it’s decided which pages appear at position 1, 2, or 3, and which ones disappear on page 10.

What is Ascorer? The primary ranking algorithm

Ascorer (short for “Amit’s Scorer,” named after Amit Singhal, the former Head of Search) is the fundamental ranking algorithm that evaluates pages BEFORE all other adjustments by RankBrain, BERT, or Twiddlers occur.

Ascorer is part of the Mustang system—Google’s primary scoring, ranking, and serving system. It processes hundreds of ranking signals, including:

• Relevance of content to the search query
• Quality and depth of content
• Backlink profile and PageRank
• Technical factors (loading time, mobile-friendliness, Core Web Vitals)
• User signals and engagement metrics

The AI Layer: RankBrain and Neural Matching

RankBrain (since 2015) was Google’s first machine learning system in ranking. It translates words and phrases into mathematical vectors (word embeddings) to understand concepts instead of keywords. For a search query like “apex predator food chain,” RankBrain can recognize that you are likely looking for information related to top-tier carnivores.

Neural Matching (since 2018) is Google’s “super-synonym system.” It understands the conceptual connection between search queries and page content—even if the words don’t match. A page about “why is my TV acting weird” can thus rank for “fix TV color issues.”

A full overview of all active ranking systems is provided by Google’s official guide to ranking systems.

After Ascorer and the AI systems have done their work, the final adjustment comes through the Twiddler framework. This phase only became widely known through the API leaks and shows how complex the system really is.

What are Twiddlers?

Twiddlers are re-ranking functions that run after the primary Ascorer algorithm. They act as the final authority and can:

• Adjust the information retrieval score of a document up or down
• Change the position of a result (promote or demote)
• Filter results entirely or control SERP diversity
• Limit categories (e.g., max 3 blog posts or 2 videos per SERP)

Known Twiddler Systems

Through the API leaks, we know some of the most important Twiddlers:

• NavBoost: Adjusts rankings based on user behavior and click logs (more on this in the next chapter)
• FreshnessTwiddler: Rewards fresh, up-to-date content for time-sensitive queries like news or events
• QualityBoost: Improves rankings for high-quality content based on quality signals
• SiteBoost: Promotes or demotes entire websites based on site-wide quality signals
• RealTimeBoost: Prioritizes breaking news and current events in real-time

The most important overarching ranking factor is content quality, which Google evaluates using the E-E-A-T concept. Practical implementation is crucial, so I recommend my detailed E-E-A-T: The Ultimate Guide for More Trust and Top Rankings.

What do the letters in E-E-A-T stand for?

• Experience: Does the author show they have practical, real-life experience with the topic? Did they actually test the product, visit the place, or live the situation?
• Expertise: Is the author a recognized expert in their field? Do they have relevant education, certifications, or demonstrable competence?
• Authoritativeness: Are the website and the author considered an authoritative source for the topic? Is it cited and linked by other experts?
• Trustworthiness: Is the site secure, transparent, and trustworthy? Is there an imprint, privacy policy, and secure connection?

“Trust is the most important member of the E-E-A-T family because untrustworthy pages have low E-E-A-T no matter how Experienced, Expert, or Authoritative they may seem.” – Google Quality Rater Guidelines

Full evaluation criteria can be found in the Google Quality Rater Guidelines (PDF). I analyzed the most important changes from the latest update here: Google Quality Raters Guidelines Update September 2025.

The Google API leaks confirmed something Google publicly denied for years: Click data and user behavior play a massive role in ranking. A detailed analysis of these revelations can be found in my article: Google Leak: Why user signals are more important than anything you previously knew about SEO.

What is NavBoost?

NavBoost is a re-ranking system based on click logs and user behavior. The leaks show that Google stores and analyzes 13 months of click data. The following metrics are included:

• Click-Through-Rate (CTR): How often is a result clicked for a specific query compared to other results?
• Long Clicks vs. Short Clicks: Does the user stay on the page for a long time (positive) or return to the SERP within seconds (negative)?
• Pogo-Sticking: Does the user jump back and forth between multiple results? This is a strong negative signal.
• Dwell Time: How long does the user spend on the page overall before returning to search?
• ChromeUX Data: Real user data from the Chrome browser also flows into the evaluation.

What does siteAuthority mean?

The leaks also show a field called siteAuthority—even though Google publicly denied for years that anything like “Domain Authority” existed. This signal apparently evaluates the overall quality and trustworthiness of a website at the domain level—not just individual pages.

Google updates its algorithm thousands of times per year to improve search results, adapt to new user behavior, and combat manipulation attempts. Most changes are small and barely noticeable—but a few “Core Updates” can shake up entire industries.

Key Updates 2025

My detailed analyses of the 2025 updates:

• Google June 2025 Core Update: Everything you need to know now
• Google August 2025 Spam Update: What webmasters must know
• Google Core Update December 2025: A gift or a nasty surprise?

In the Google Search Status Dashboard, you can track all official updates. How to correctly interpret SERP volatility and react to updates is explained in: How Semrush Sensor helps you master SERP volatility and Google updates.

What is Google’s AI Mode?

In 2025, Google introduced AI Mode—a completely new way of searching powered by Gemini 2.5. AI Mode uses a “Query Fan-Out” technique: Google breaks down complex questions into subtopics, searches various sources in parallel, and generates comprehensive AI-driven answers.

Differences and SEO implications in detail:

• Google AI Overviews: Functionality explained simply
• Google AI Mode & AI Overviews: The difference explained
• SEO in the age of AI browsers: Framework for the citation economy

What’s changing with Core Web Vitals?

Since March 2024, Interaction to Next Paint (INP) has been an official part of Core Web Vitals, replacing First Input Delay (FID). The crucial difference: INP measures your website’s responsiveness to all user interactions throughout the entire visit—not just the first one.

Core Web Vitals Benchmarks 2026

More on this in the official Web Vitals documentation.

llms.txt – Preparing for AI Crawlers

With the rise of AI search systems, the question of how to optimize your content for AI crawlers is becoming relevant. The new llms.txt format provides a structured way to provide information about your website to AI systems: llms.txt explained: Step-by-step guide & practical example.

Understanding the Google algorithm means recognizing that there are no more simple tricks. The days of manipulating rankings with keyword stuffing and link spam are long gone. Modern SEO success in 2026 is based on four fundamental pillars:

1. Technical Excellence: Ensure flawless crawlability and indexability, pay attention to Core Web Vitals (especially INP < 200ms), and use structured data. A technically sound website is the foundation.
2. Semantic Relevance: Create content that covers topics deeply and comprehensively. Answer your target audience’s questions better, more completely, and more helpfully than anyone else.
3. Absolute Trustworthiness: Build your personal brand, demonstrate real expertise and experience, and become an authority trusted by both users and Google (E-E-A-T).
4. AI Optimization: Prepare your content for AI Mode and AI Overviews. Structure information clearly, answer questions directly and comprehensively, and become a citation-worthy source.

How often does the Google algorithm change?

Several times a day. Google confirms thousands of small changes per year. Larger, noticeable Core Updates are rolled out 3-4 times annually. In 2025, there were Core Updates in March, June, and December, plus a Spam Update in August.

What are the most important Google ranking factors in 2026?

The key concepts are: Content quality and E-E-A-T (Experience, Expertise, Authoritativeness, Trustworthiness), Mobile-First optimization and Core Web Vitals (especially INP), semantic relevance and comprehensive topic coverage, as well as technical SEO (crawlability, indexability, page load time). User signals like NavBoost also play a major role.

What is Ascorer?

Ascorer (short for “Amit’s Scorer,” named after Amit Singhal) is Google’s primary ranking algorithm that evaluates pages BEFORE RankBrain, BERT, or Twiddlers make their adjustments. It is part of the Mustang system and first became public through the 2024 Google API leaks.

Is MUM used for ranking?

No, not for general ranking. Contrary to many claims in the SEO community, MUM (Multitask Unified Model) is currently only used for specific applications, such as COVID vaccine information and certain Featured Snippets. Google officially confirmed: “MUM is not currently used for general ranking in Search.”

What are Twiddlers?

Twiddlers are re-ranking functions that run AFTER the primary ranking (Ascorer). They adjust the final search results based on specific criteria—e.g., NavBoost for user signals, FreshnessTwiddler for up-to-dateness, or QualityBoost for high-quality content. By 2018, over 65 Twiddlers were already in production.

What is the difference between Core Updates and Spam Updates?

Core Updates are broad adjustments to the entire ranking system to improve the relevance and quality of search results. They affect all websites and can lead to winners and losers. Spam Updates specifically aim to identify and penalize web pages that violate Google’s spam policies (link spam, cloaking, keyword stuffing, etc.).

How long does it take for ranking changes to become visible?

This varies greatly depending on the situation: New content on authoritative pages can be indexed and rank within hours. Content updates on existing pages usually take 1-4 weeks for a full re-evaluation. After Core Updates, it can take 2-4 weeks for the full effects to stabilize.

Can you trick the Google algorithm?

No, not sustainably. Manipulative tactics (Black-Hat SEO) are increasingly recognized by the algorithm—especially with new AI systems like Gemini 2.5 and the expanded SpamBrain system. Short-term gains often lead to long-term penalties. A strategy based on genuine quality and user value is the only sustainable path.

What did the 2024 Google API leak reveal?

The leaks revealed internal system names and functions that Google had never publicly communicated: Ascorer as the primary ranking algorithm, NavBoost for user signals (with 13 months of click-log data), Index Tiers (Base/Zeppelins/Landfills) for storage prioritization, siteAuthority as a domain-wide quality signal (though Google had always denied “Domain Authority”), and the Twiddler framework for final re-ranking.