Welcome to another insightful week with The SEOs Diners Club newsletter. Leveraging over a decade of SEO expertise, I aim to distill the week's most critical developments into a concise read that will take less than seven minutes. Let's dive into the insights that matter.

Advanced Screaming Frog Crawling Techniques for E-Commerce Sites

Effective crawling techniques on e-commerce sites with Screaming Frog! URL exclusions, segments, and more.

Every SEO professional's nightmare is moderating e-commerce sites with nearly endless pages. Fortunately, the Screaming Frog SEO Spider is invaluable for overcoming this challenge. But to fully utilize this powerful ally, you must explore advanced techniques beyond its essential functions.

There are excellent resources for detailed information, such as crawling large sites, memory, and database storage modes. Therefore, we will not cover these basics in this article. Instead, we'll go over advanced crawling tips to help you master Screaming Frog and audit your e-commerce sites at scale.

This guide by Maria Camañes Forés shows you how to make the most of Screaming Frog on e-commerce sites.

Some methods that are important in terms of content and efficiency, especially on e-commerce sites, stand out:

1.URL Exclusion:

You can exclude unwanted sections so Screaming Frog can crawl the most critical content. For this, you can reduce the scan size by adding sections that should not be included in the site from the Configuration > Exclude section.

2.Creating Crawl Segments:

Screaming Frog's “Segments” feature categorizes your site browsing. You can segment your site into different types, such as Product, Category, and Blog. You can start defining your segments from Configuration > Segments > Add. This way, you can see crawling issues specific to certain page templates.

3.Determining Out-of-Stock Products:

Screaming Frog's "Special Sticker" feature lets you quickly identify out-of-stock products on your site. To add the URL of an out-of-stock product, go to Configure > Custom > Extraction > Add > Globe icon. Then, you can review the HTML of your “Out of Stock” message and create a custom sticker rule.

4. Scraping Category Page Content:

E-commerce sites often include content at the top or bottom of category pages. Screaming Frog can extract this text from each page using the “Custom Extraction” process. You can then transfer these texts to a chart and identify the sections that need to be optimized.

5. Finding Products That Don't Receive Reviews:

Using unique stickers, you can find products that do not have reviews in scale. Using the web page extraction feature, you can find the star review section on product pages and extract the number of reviews. This way, you can quickly see products with few or no reviews.

While Google's Search Engine Market Share Is Falling, Its Competitors' Is Increasing

Google's decline in search engine market share presents an opportunity to rethink your SEO and content strategies. Learn about the rise of competitors and market dynamics.

I want to cover a significant development regarding the decline of Google's search engine market share and how its competitors are rising.

According to GS Statcounter data, Google's search engine market share has fallen to 86.99%, the lowest recorded since 2009. This coincided with rivals Microsoft Bing and Yahoo increasing their market share in the US and the global market. Bing, in particular, reached 13% of the market share in the US and 5.8% globally, which is its highest level since 2009.

So why is this important?

I wrote what I wrote below, considering the possibility that Statcounter data reflects the actual situation and the possibility that they made a miscalculation.

Google has long been a leader among search engines, which is essential in determining strategies for businesses and marketers. However, with Google's market share declining and other search engines increasing in popularity, publishers and SEO professionals may need to review their strategies and optimize for multiple search platforms.

These changes show that SEO experts need to develop new strategies. Diversifying SEO efforts in accordance with each search engine's algorithms and ranking factors, following best practices, and focusing on high-quality search results are crucial for successful SEO campaigns. Producing informative and reliable content that meets users' expectations is also critical.

This decline in Google's market share shows that other search engines must prioritize user satisfaction and how they must act to maintain and increase their market position. Increasing competition in this field offers users and industry professionals new opportunities. Following these dynamics closely and adapting will be one of the keys to success in the SEO world.

How Often Does Google Crawl URLs?

Google's URL recrawl rates vary; some are scanned several times daily, and others are scanned months later.

Google's senior analyst, John Mueller, gave very descriptive information on this subject. Google constantly crawls web pages and documents to index them, and as these pages are updated over time, it re-crawls URLs periodically.

According to Mueller, search engines crawl URLs at different speeds. Some URLs are crawled several times daily, while others are crawled again every few months. New news, in particular, is scanned frequently and several times daily. However, it may take several months to crawl old news on the site. This is normal, and Google does not expect old news to be updated frequently.

In response to questions about removing pages, Mueller noted that a page with a 404 status code added could take several weeks to months for Google to recrawl and remove it. To speed up this process, the Google Search Console URL removal tool is recommended.

This information can help you develop strategies for managing your content and determining how often Google will crawl it. Remember, new and frequently updated content is subject to a faster re-scanning process.

Google: Helpful Content Update Recovery May Take Much Longer Than Others

In his statement on this issue, John Mueller said that some sites may take months to recover.

I want to share a significant development regarding the Google Helpful Content Update with you in this week's newsletter. As you know, this update was announced in October last year and hurt many websites.

Google said the sites affected by the update will recover over time and return to their old ranking positions in search results. However, some sites still have not made any progress in this regard.

So, why does this happen?

In his statement on this issue, John Mueller stated that it may take months for some sites to recover or even until the next update cycle.

Mueller also emphasized that Google does not hold a grudge against the sites and that all sites can return to their previous ranking positions once the problems are resolved.

Some sites affected by the Google Helper Content Update have still not recovered. The recovery process may take months or until the next update cycle.

This week, experienced SEO experts Lily Ray, Glenn Gabe, and Marie Haynes reproached Google on this issue. They stated that sites with high-quality content cannot recover due to helpful content updates.

Let's see how Google responds to these reactions. I observe that Google is evolving from a search engine developed by engineers to a search engine focused on financial gains. As long as their income continues to increase, the age difference from the exchange rate will not be a big deal for them. For site owners who cannot recover after this update, this is not the time to cry and whine but to try different things.

116 Best Free SEO Tools

Discover free tools that will improve your SEO! This guide includes ideal tools for keywords, site speed, and more.

Check out this article from the Search Engine Journal to learn about the best free tools for improving your SEO strategies and website performance.

The article contains user-friendly and productivity-enhancing tools, providing comprehensive information on how to use each.

These tools are designed to meet SEO needs, from keyword research to site speed tests. They are ideal for both beginners and experienced SEO experts.

OpenAI is Working on a Search Engine to Rival Google

OpenAI is developing a search engine powered by Microsoft Bing. Why is this development important for SEO and digital marketing? The article provides details.

The news that OpenAI is working on a new search engine rivaling Google is among the significant developments in the SEO and digital marketing fields.

OpenAI is currently known as the creator of popular AI tools like ChatGPT and is now developing a web search product with support from Microsoft Bing. GPTBot, a web browser from OpenAI, and the fact that ChatGPT Plus users can search the web with Bing show why this new product is noteworthy.

This is something to watch to see if OpenAI will be a significant source of organic traffic for brands and businesses if it actually launches its search engine. However, the fact that even a giant company like Microsoft has not been able to gain a significant market share against Google raises doubts about whether OpenAI can be successful in this field.

Why is it important?

If OpenAI can truly offer an alternative to Google, this will open up a new field of competition in the search engine market and could lead to changes in SEO strategies. However, given Google's dominance and user habits, it remains unclear whether this new initiative will be successful.

Recommended Tools & Services

• WhitePress: WhitePress is a content marketing platform that enables accessible article publishing across 90,571 websites in 30 languages, offering native copywriting and automated global publication services.

• Cannibalization Explorer Looker Studio Report is a tool for identifying keyword cannibalization and providing insights into how keywords perform across different pages.

• SurferSEO is a tool that helps you create and optimize content for SEO. It analyzes the top-ranking pages for your target keywords and gives you suggestions on how to improve your content. Surfer AI also has a feature that can generate ready-to-rank articles in minutes.

• Copilot Pro: Copilot Pro is available for $20/month for cloud-based GenAI models and $27/month with Microsoft 365. It is integrated with Microsoft 365 applications and provides access to GPT-4 Turbo and advanced AI rendering tools.

• AlsoAsked is a helpful tool that can boost your visibility in the “People Also Ask” section of Google's search engine results pages.

• Semrush Map Rank Tracker is a game-changing tool for local businesses looking to dominate their niche and attract more customers. I've been using the free version for a while now, and I'm impressed with its ease and effectiveness.

• ContentShake AI is an intelligent writing tool that combines AI with real-life competitor insights. It guides you from ideation to publishing directly to the blog, generates SEO-friendly articles, and helps you optimize them for organic traffic and engagement.

• Microsoft Clarity has recently introduced a feature called Copilot, which incorporates Generative AI through Large Language Models (LLMs) into the analytics tool. This integration aims to make data more accessible and understandable for users by leveraging the same technology created by OpenAI, which powers the new Bing or ChatGPT.

• InLinks is an SEO tool that uses a knowledge graph to optimize content for search engines. It automates internal linking and schema markup, simplifying SEO processes and content strategies.

Subscribe to my newsletter to stay informed about the latest in SEO and digital marketing. Next week, I'll continue to share the newest insights, tactics, and strategies for navigating the ever-evolving digital marketing landscape.

Support My Newsletter:

If you find my content valuable, consider buying me a coffee to support my work. Your contribution will help me continue sharing the latest SEO developments. Let's achieve more together!

Best,

Mert