One of the first and most important steps in signing on a client for SEO services is performing a crawl of their website in its current state. This entails running a scan of every URL on the site to identify each HTML page, image, Javascript code, CSS sheet, and so on in order to create a map of how the website is structured, identify technical errors, and determine what needs to be adjusted to impact performance and rankings.
Sure, it’s simple enough to input a website into a crawling service, have it spit out a spreadsheet of URLs, and then start combing through it to identify problem areas. But time is precious.
This is where a more in-depth site crawling tool is of immense help. A good crawler will de-clutter the information and make it easier to pick out the exact things you’re looking for rather than having to test them back-and-forth on the site itself endlessly or piece together a variety of tools to accomplish the same thing.
There are many site crawlers out there, any of which will get the job done. But at SEOM, there’s one that we prefer due to its speed, flexibility, and simplicity of use: The Screaming Frog SEO Spider.
Developed by the UK-based search marketing firm Screaming Frog, the SEO Spider has become a staple tool in the industry, and with good reason. While there are countless helpful features in SEO Spider, I’ve whittled them down to a list of the features and attributes that I use most commonly – and the ones that set this tool apart from other similar site crawling services.
Speed
Before we dive into the features that SEO Spider offers, I must point out how fast it performs its crawls. Watch this SEO Spider site crawl in action:
It took about 20 seconds to crawl every one of the 579 URLs on the site I entered. Not bad, right? This especially comes in handy if you need to perform frequent crawls of a larger site and utilize the SEO Spider’s scheduling tool to do so.
Titles, Metas, and Headers
Now that we’ve gone through the site crawl, let’s look through the site’s title tags and meta descriptions. The SEO Spider has helpful, straightforward navigation for viewing all of this information separately by URL – down to more detailed information such as H1 or H2-tagged content on the page.
The program automatically checks for common areas of concern, such as the length of the tags and their similarity to other content on the site. You can also filter the crawl, which is a huge time-saver when you’re looking to make improvements to tags.
Finding Broken Links
When crawling a site, one of the most important things to look out for is links that lead to 404 errors. You don’t want to direct users to dead end pages. Plus, 404 errors can negatively impact a site’s organic search rankings.
The SEO Spider makes it incredibly easy to identify 404s wherever they exist on a site. How? Run your crawl, filter it by 4xx response codes, click on the offending link, and then navigate to “inlinks” on the bottom panel. You’ll be able to see the page where the broken link is located and its anchor text.
Structured Data Tool
Structured data, or schema, can be highly beneficial for a page’s organic ranking. It provides search engine crawlers with a better idea of that page’s contents, thereby making it easier to match it with relevant search queries. However, a common frustration when setting up structured data is ensuring there are no errors in the code and making sure the data will be correctly read by search engines, as it can take time to be indexed.
This is another place where the SEO Spider can save you. It has a built-in preview window that can be used to identify schema and, if need be, troubleshoot it for any page within the site that you’ve crawled.
SERP Snippet
The SEO Spider also comes with a built-in preview window that allows you to see how any given page will appear in organic search results. Even more helpful is the fact that this SERP Snippet preview gives you the ability to edit the title and meta description as it appears in the preview. This enables you to experiment with new title tags and meta descriptions to see how well they fit and if they get the message across right there in the tool. Rich snippets, including schema, will also appear in this preview, offering yet another avenue for troubleshooting. Select the HTML page that you want to see a preview for and select “SERP Snippet” on the bottom to access this tool.
Interested?
If you’d like to test any of these features for yourself, Screaming Frog offers a free package for the SEO Spider that offers basic auditing features and allows you to crawl up to 500 URLs at once. Upgrading to the paid package gives you a host of advanced features such as structured data analysis, scheduled crawling, Google Analytics integration, and more.
Better yet, the paid package also removes the limit on the number of URLs you can crawl on a single site, which is crucial for larger clients and e-commerce sites. The paid package starts at $192 per year per license (or £149, since Screaming Frog is based in the United Kingdom), with incremental price decreases for obtaining a larger number of licenses at once.
We use many different tools for various tasks on a daily basis here at SEOM. When we have to frequently cycle through so many different UIs and programs, each with their own quirks, it’s refreshing to be able to depend on something as functional and straightforward to use as the Screaming Frog SEO Spider. Especially for a time-consuming (and let’s be honest, sometimes monotonous) task like site crawling.
December 10, 2019