• Skip to primary navigation
  • Skip to main content
  • Skip to primary sidebar
  • Skip to footer
  • Home
  • Good Books
  • Tools
  • Blog
  • Darren
  • Contact
  • Hire Me
Small Biz Geek

Small Biz Geek

Small Business Design, Marketing & Technology Journal

Solving small business design, marketing & tech problems

  • The Internet
  • Marketing
  • Graphic Design
  • Web Dev
  • Web Design
  • Social Media
  • Privacy & Security
  • Tech
  • Misc

Screaming Frog – Website Page Crawler and SEO Auditor 📋

You are here: Home / Darren’s Small Business Tools / Screaming Frog

An outstanding bit of free software essential for every webmaster wanting to understand how their webpages appear to search engines and what issues, if any, exist.

Screaming Frog SEO Tool

How does you website appear to search engines? Have you crafted proper titles and descriptions? Do you need a birds eye view of webpage information?

Screaming Frog SEO Spider is a website crawler tool used for analyzing on-page SEO elements. It helps users identify issues such as broken links, duplicate content, missing meta tags, and more, allowing for better optimisation and improved search engine rankings.

Enter any URL and the software fetches all onsite elements, presenting the information in rows and columns, like a spreadsheet. There are loads of uses for the tool even under the free plan.

High level Overview of Webpages

Sometimes, it’s handy to crawl a competitor’s website to see what they’re doing, or if they’ve missed something from their website. It’s worth doing if you suspect they have an advantage over you, perhaps to do with the way they write their SEO title tags, for example.

It’s not just for public websites either. Do you build your websites offline in a localhost for development and testing?

I use WAMP to develop sites and yet I use Screaming Frog to comb the site for errors such as dead links producing 404 pages. It will also show missing meta information. It will also remind me that I might need to noindex certain pages before launching the website.

Introduction Screaming Frog SEO Spider

Watch video on YouTube

On Page Criteria

Enter the address of the website you’d like to crawl and click Start.

Once you’ve finished crawling a website you can scroll down to see how many pages have been included in the analysis.

Screaming Frog Crawl Screenshot

Scroll to the right and you’ll see various columns about the website page address, the type of content it is, status code, page title, meta description, headings, meta robots, canonical links, size of page, word count, internal links and external links.

There’s an official user guide on the Screaming Frog site and I suggest you refer to it if you want the latest documentation from the horse’s mouth.

Resource Info Summary

Each of the rows represent “resources”. Resources include, among other things, webpages and images.

If you click a row, the lower bottom pane of the Screaming Frog interface displays information about that resource.

Screaming Frog SEO Spider Lower Window Pane

The same information is located in the horizontal columns at the top. I’ve broken down each column one by one, below…

Address Column

The address column is the URI, which is an acronym for uniform resource identifier. This is computing lingo for something uploaded to your server, whether it be text, image, video, audio or a script.

The screenshot below shows rows containing each URI:

Screaming Frog Address Column

The most common type of URI also known as URL, is a HTML webpage. Screaming Frog comes equipped with another column to the right of the Address column showing what type of URI we are looking at.

Content Column

The Content column sits to the right hand side of the Address column. Each row indicates what type of content is on each URI.

In the screenshot below we have a mixture of HTML webpages (with character set information), images (JPEG files, in this case) and applications (javascript).

Screaming Frog Content Column

Tip: If you just want to crawl webpages, select HTML only, as the free version of Screaming Frog has a 500 limit crawl, and because by default it crawls all website assets, you’ll be wasting that 500 crawl quota on everything else besdies actual pages.

Status Code & Status Column

The next two columns return the status of each URI. A status code of 200 means that everything is functioning successfully and that search engines are able to fetch those particular resources.

Screaming Frog Status Column

A 404 (Not Found) suggests resources from a particular URI are unavailable.There are several possible reasons.

A webpage might have been deleted, or the URI might have been changed to something else but there could be page on your site still trying to link to that resource.

Other reasons for seeing a 404 may be because the URI is included in your website sitemap or exists only as a canonical link with no actual page.

Click the row and use the lower pane to look for clues as to the cause of the 404:

Screaming Frog 404 Info

Clicking the Inlinks and Outlinks buttons will normally reveal where the error is originating:

Screaming Frog 404 Inlinks

The cause of the errors shown above was simply because I had changed the page slug on a blog post and forgotten to update the corresponding canonical link. Tut tut for me! But I found the error thanks to this tool.

One more example: An eBook download link was broken. I could tell instantly by looking at the Inlinks information that I must be linking from an image, because there was no Anchor Text.

There was only ALT text (which I add to all my images) and that was the clue I needed to check the images on that particular webpage to locate the broken link and fix it:

Screaming Frog Broken Image link 404

By the way, I recommend looking at this useful list of HTTP status codes if you want to know what the most common ones mean.

Title Column

The next column is perhaps on of the most important, since webpage titles are the headlines uses in search engine snippets.

Screaming Frog Title Column

Page titles are what search engines use to help make decisions about what results to return, and is users see when choosing what results to click.

SERP title snippet

If you’re using WordPress, do not confuse the page title with the post/page title. The title, as seen by search engines, can also be seen in the top of your browser by hovering the mouse cursor over one of the tabs.

Web page browser page title

Title Length Column

A good page title must ideally fit into search engine results snippets without becoming truncated. The reason you don’t want page titles to be truncated is because the people reading them won’t get the whole title, and might not click.

Keeping page titles shorter is a good practice and if you keep them to around 55 characters, it is likely to fit.

The column for title length in Screaming Frog is invaluable because it lets you see where you can make improvements.

In 2014 Moz released a handy tool for inputting title tags and generating realistic Google preview snippets. They also emphasised the importance of understanding that there really is no “magic number”, but that on average, 55 characters gives some wiggle room. You can also a tool like the free Mangools SERPs simulator.

Screaming Frog Title Length Column

Meta Description Column

If meta descriptions haven’t been crafted for a particular webpage, you’ll know about it when you run the software.

When we were discussing page titles earlier, I showed a screenshot from the Google search engine results page. Now let me highlight the meta description of that page:

Meta description in SERPs

Each description should be unique and provide a summary of what each page is about. Assuming you have added these, they’ll show in the corresponding Screaming Frog column.

Screaming Frog Meta Description Column

Like page titles, meta descriptions should be kept to an ideal length, and that’s what the next column is for.

Meta Description Length Column

I suggest a meta description length of no more than 160 characters. As you can see in the screenshot below, some of mine have exceeded the recommended length but not my much.

Screaming Frog Meta Description Length

H1 Column (First Occurrence)

Heading 1 tags are also known H1. This is usually the main heading for the page, but it could also be your site title. Screaming Frog looks for two occurrences of the H1 tag and denotes them as H1-1 and H1-2.

The screenshot below shows that the words “Small Biz Geek” are used inside every page.

Screaming Frog Heading 1(1) Column

The reason “Small Biz Geek” is used multiple times is because by default WordPress has assigned a H1 tag to the title of my blog.

WordPress H1 & H2

The Site Title and Tagline are used for H1 & H2 respectively

H1 Column (Second Occurrence)

The second occurrence of the H1 tag is denoted as H1-2. Not all websites have multiple uses of the H1 tag and I would say any more than three H1 tags is heading into risky territory. 

The title of my posts and pages are using H1 tags since that is what each page is about. 

Screaming Frog Heading 1(2) Column

H2 Column (First Occurrence)

Heading 2 tags are denoted as H2-1 in their first occurrence. In this case I’ve used the phrase “My passion for small business design, marketing & tech” as my WordPress website tagline.

WordPress has then automatically wrapped the website tagline inside the H2 tag.

Screaming Frog Heading 2(1) Column

Canonical Link Element Column

If you’ve included canonical links for each of your pages they will show up here. As I mentioned earlier, any 404 errors could be a typo with a canonical link.

I forgot to correct one of these links a few weeks back and it was showing as a sort of phantom page and therefore returned the 404.

Basically, all you need to know is that a canonical link for a webpage must match the URI exactly.

Screaming Frog Canonical Link Column

Word Count Column

The word count is all the words between the <body> and </body> HTML tags. This can be a bit misleading because dynamic websites usually have headers, sidebars and footers.

I’m more interested in the word count of the main content area on a page.

Screaming Frog Wordcount Column

Inlinks Column

Number of internal inlinks to the URI. ‘Internal inlinks’ are links pointing to a given URI from the same subdomain that is being crawled. This is handy, because you can detect which pages on your sites don’t have enough backlinks (both internal or external) pointing at them.

If you produced a new page and published it, the chances are, it’s an orphan page, or at the very least, needs to be linked to from other relevant pages on your website.

For me, this is one of the handiest things about Screaming Frog.

Screaming Frog Inlinks

Click a URI row containing a wepbage with Inlinks and you can view those links in the lower pane:

Screaming Frog Inlinks Lower Pane

Outlinks Column

Number of internal outlinks from the URI. ‘Internal outlinks’ are links from a given URI to another URI on the same subdomain that is being crawled.

Screaming Frot Outlinks Column

External Outlinks Column

Number of external outlinks from the URI. ‘External outlinks’ are links from a given URI to another subdomain.

Screaming Frog External Outlinks

Response Time Column

Time in seconds to download the URI. Other tools exist online for checking webpage load speed such as Google Lighthouse.

Screaming Frog Response Time Column

Last Modified Column

Read from the Last-Modified header in the servers HTTP response. If there server does not provide this the value will be empty. 

Screaming Frog Last Modified Column

Redirect URI Column

If you’ve created a 301 redirect for an old link, the Redirect URI column shows where it is pointing.

Screaming Frog Redirect URI Column

The example below shows that I originally published a page but later redirected the link:

Screaming Frog Redirect Reaarange Columns

 

500 URI Crawl Limitation

Screaming Frog is limited to 500 URIs under the free/lite version. What it actually means is that if the website you crawl exceeds 500 pages (500 URI’s) then you’ll need to upgrade to a paid licence in order to get all the data.

This might be a moot point since many small business websites or indeed small websites. You might have as little as 5 pages (5 URI’s). However, if you have an eCommerce store, the product listings could run into the thousands of URI’s, especially if you have lots of images.

This is why you might want to select to only crawl HTML assets if it’s just webpages you’re wanting to analyse. No need for all the other assets, which will run into the thousands.

Summary: Worth Having Even Without a Paid Licence

This is an impressive piece of software and a must have tool for anyone running a website. The level of insight offered under the free/lite version is simply fantastic because it offers up errors that are glaringly obvious from a search engine perspective but invisible to the naked eye.

Get started with Screaming Frog for free.

Reader Interactions

Comments

  1. SiteAnalyzer says

    January 13, 2019 at 7:59 pm

    For a technical audit, I recommend to look towards SiteAnalyzer – a free program for auditing and technical analysis of the site. At the same time, the set of functions is practically not inferior to paid counterparts

    Reply

Add Your Thoughts Cancel reply

Your email address is safe and will NOT be shared with anyone else.

Hateful, spammy or abusive comments will not be tolerated.

For more information please see the comment policy.

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Primary Sidebar

Search Website

Latest Blogs

  • Is Generative Engine Optimisation (GEO) the Answer to the Future of Search? 🎨
  • How Hiding Your Phone Number and Using a Booking System Repels Time Wasters and Frees Up Time 🗓️
  • “Fucking Good Content” – Dan Kelsall (Book Review) 📘
  • Scam Poetry: The Time I Was Recruited to the Cult of Amway ⚠️
  • A Story of Seductive Social Media Success and Neglected Email Strategy 🏚️
  • The Classic Social Media Trend Destined to Ruin Your Business 😭
  • Looking for Online Work? Don’t Fall for this Fake Hays Recruitment WhatsApp Scam ⚠️
  • Cream of the Crap: How Fake Reviews, Bad Businesses and Dangerous Products Rise to the Top 💣
  • Exposed: The Hong Kong Investment “Fraud Recovery” Scammer Deceiving Victims 🤑
  • Computer Says No: Does Your Website Work for People with Disabilities? ♿
  • Chinese Tinder Profiles Are Using Photos of Pretty Girls to Scam “Investors” 💋
  • Small Biz Owners “Trapped” Using Email Addresses Belonging to Internet Service Providers 🪤
  • In the Pandemic, QR Codes are Finally Proving Worthy in the West 😷
  • Small Businesses Finally Start Marketing… and All It Took Was a Global Pandemic 🏁

Footer

Primary Navigation

Home
About Darren
Good Books
My Tools
Blog Posts
Hire Me
Contact Me

Derby & Nottingham Area

The Internet
Marketing
Graphic Design
Website Development
Website Design
Social Media
Technology
Miscellaneous
Privacy & Security

Let’s Connect

Twitter
YouTube
LinkedIn
RSS

Small Business Blog · Privacy Policy · Commenting Policy · Website Disclaimer · ICO number: ZA305900


Written and designed by Ilkeston Web Design

  • Home
  • Good Books
  • Tools
  • Blog
  • Darren
  • Contact
  • Hire Me