How To Get Found On Google – A Hands On Guide To Crawling And Indexing

Table Of Contents

  • What are crawling, indexing and ranking?
  • Getting Google’s Spider To Crawl Your Website
  • Yoast Setup and Sitemap Submission For WordPress

Google crawl index rank sphere

When it comes to ensuring your business can be found on Google search there are just three steps. It doesn’t matter who you are, whether you’re big or small, if your budget is thousands or millions; at its core, Google works the same no matter who you are.

The steps that Google go through before your website can be found are crawling, indexing and ranking. Today we’re going to discuss the basics of the first two and how to give your site the best possible chances of ranking really well in Google.



What are crawling, indexing and ranking?

Three steps, sounds simple right? Let’s go into a bit more detail and discuss what these three ‘simple’ steps actually mean:


  1. Crawling – To get a site ranking on Google they must be able to find your content. Google does this automatically all day every day using their crawler Googlebot.

    Googlebot is like a person who endlessly visits websites and clicks links to other websites, navigating the internet through a never-ending interconnecting sequence of links. They will also know to crawl around your website if you tell them it’s there, something we’ll go into later.

    When it comes to technical factors crawling is something you can improve a variety of ways including by providing a sitemap, robots.txt, meta tags and header responses.

  2. Indexing – Once a page is discovered (crawled) Google must then take that webpage and process it to pull out all the information it requires about the page to rank it.

    Indexing is the process of storing all of this information about every single one of your website’s pages. This includes things like the words from your content, their frequency and relationships along with their location on the page.

    This is done whilst also compiling other important information about every page on your website such as metadata, structured data and alt tags from images.

  3. red and green arrowsRanking – The third and most discussed stage of how to get found on Google, ranking. It’s also the most complex and not worth investing time in until you have the basics of crawling and indexing mastered.

    Ranking first involves finding pages in the index that match the users’ search query, then using over 200 different ranking factors to determine which page is most useful and should appear at the top of the search results.

You’re probably starting to notice that this is probably going to be a bit more involved than just doing three things to keep Google happy. Their process may only be three steps long but as website owners our investment in making their job as easy as possible is much bigger.

Let’s now take a deep dive into the first two stages – crawling and indexing. Once you finish reading this article you should know the common mistakes to avoid so your business has the best chance to be found on Google.

Getting Google’s Spider To Crawl Your Website

google spider at night with moonRandomly stumbling across your website isn’t easy especially when you consider that in 2019 we are expected to hit the 2 billion websites on the internet milestone. That’s a big world wide web for Google to navigate, to put that into perspective if you were Google it would mean that you know 1 in every 4 people on the planet. I struggle to remember the names of 1 in 4 people on my street, never mind globally.

So how do we make this easier for Google?

The simple place to start is by ensuring that Google knows your website exists, this means you should:

  1. Create a Google Search Console account – If you have a website and want to receive Google search traffic then you really should be setup with Google search console. You can sign into Google search console using an existing gmail email address or by registering an account using any other personal email address.

    Once registered you’ll need to add your site to search console, if you need help then follow this guide from Google or watch the video below that shows how to setup Google Search Console on your WordPress website from WP Beginner.

  2. Create and submit a Sitemap – The easiest way to add a comprehensive sitemap to your website is using the Yoast SEO WordPress plugin. You’re going to need this plugin if you want to improve your rankings in Google there are plenty of alternatives but Yoast SEO is the go to choice for a variety of different SEO tasks.

    We’ll walk through the setup of Yoast SEO and how to submit this sitemap to Google Search Console in just a moment.
  3. Check against our cheat sheet – There are a bunch of other technical things you need to check, so we’ve created a cheat sheet. Make sure everything is working as expected and if you don’t tick any of the boxes don’t panic we’ve also included where you can go to resolve any remaining problems you may have.

    The cheat sheet includes things like ensuring your robots.txt is setup correctly and checking to ensure your site is unique and only loads on one domain.

Get Your Indexing Cheat Sheet - FREE

Subscribe to our mailing list to get your free indexing cheat sheet. You can unsubscribe from our emails at any time!

Yoast Setup and Sitemap Submission For WordPress

The first thing you’re going to need to do is install the Yoast SEO plugin which is a pretty easy process, the video below from Yoast themselves will walk you straight though it.

Once you’ve got this out of the way you need to go to the plugin settings and run through the plugin configuration, start the configuration and walk through the steps with us below;

  • Start up the configuration tool, simply click configure Yoast SEO.

Yoast SEO for WordPress screenshot

  • Select Option A and click next, unless you’re working on a site created just for testing.yoast for wordpress walkthrough screenshot
  • Next, select the most appropriate option for your website and click next.

yoast for wordpress walkthrough screenshot step 3

  • Select company and then type in the name of your business exactly how you want it to appear online.
  • You should also upload a logo for your business at this point.
  • Next you’ll want to fill in any social media profiles your business has and click next.

Indexing yoast set up step 5

  • If you’re working on a new site without much content I suggest that you set only posts and pages to yes and the rest to no. Over time you may want to index other content such as taxonomies but let’s start by giving Google visibility of your most important content.

yoast for wordpress indexing step 6

  • Is all the content on your website going to be created by yourself and published under a single name, if so click no. Otherwise if you’re going to have multiple writers click yes.

yoast for wordpress indexing step 7

  • Next you can link your Google search console, this isn’t essential but it can be helpful. Once you’re all done just click next.

yoast for wordpress indexing step 8

  • The final step that requires any work is setting up your title settings, I recommend you set the website name to your brand name and use the dividing line as your title separator.

yoast for wordpress indexing step 9

  • You can just skip through step 10 and 11 by clicking next no work is required.

Yoast Indexing Setup complete

Whats Next? 

You’re all done congratulations you’ve setup Yoast SEO and are ready to submit your sitemap to Google Search Console. You’re almost there, just a few minutes work left to do. Let’s walk through that now.


  1. Login to Google search console
  2. Select your site
  3. Click sitemaps on the left hand side
  4. Type sitemap_index.xml into the box
  5. Hit submit


It really is as simple as that you’ve done it your site now has a full sitemap of all pages and blog posts on your site and you’ve submitted it to Google. This tells Google all of the content that exists on your website and encourages them to crawl it quicker, effectively ensuring that Google can later index and rank your content.

See your website through Google’s eyes

If you’re really committed to managing a website in a professional manner then you can use tools to help you see your website through Google’s eyes, my personal favourite is called Screaming Frog. What’s even better is that if your site is pretty small then it’s completely free to use.

If you want a great beginners guide to Screaming Frog then take a look at this video from Chase Reiner SEO.

Helping Google to index your website

So you’ve got everything covered when it comes to crawling and Google is finding all of your relevant content. There are however several things that should be done to ensure that your content is indexed in a way that gives it the greatest possible chance to rank:


  1. Don’t hide your content in images – Even if you have beautiful graphics to explain things step by step on your website it doesn’t mean it will rank. If you’ve put important content in images then Google simply can’t read it. If something is important then don’t hide it from Google ensure that your most important content is readable by search engines and users.
  2. Use headers in your content – Break your content up using headers this makes it clear to Google and your readers what the content is about, making it super easy to skim read too. It’s especially useful to use <h1>, <h2> and <h3> tags rather than simply increasing the font size or changing the styling in another manner.
  3. Use schema markup where appropriate  – Markup tells Google what parts of your content means, for example if you have a star rating then it gives Google the data it needs to know that it is your product review. You can also use markup to tell Google that your content is a product or a blog post, which is really useful.

    If you’re creating product reviews but aren’t using markup then you’re missing out on an opportunity to give Google deeper understanding of your content. In the case of reviews this can also mean missing out on star ratings in search results so it’s really important to keep yourself up to date with what markup exists and to use it appropriately.
  4. Describe your images – Google can’t understand your images so where you need to use them then ensure you have an accurate alt tag written in natural language, don’t force it full of keywords. Describe your image in a manner that makes sense to users as if you are describing it for a blind person using a screen reader to view your website. If your keyword is appropriate then use it, just don’t force it into every stock image.
  5. Latent semantic indexing – One of the frequent tactics used by SEOs was to stuff a page full of the keywords they wanted to target to rank higher; Google combatted this tactic by making use of LSI.

    Simply put LSI means that Google looks for synonyms and related words to your title within your content to support the fact your article matches the content. Using varied supporting LSI terms within your content ensures that your contents theme is correctly understood and gives you the best chance to rank for your actual target terms. A writer with plenty of knowledge on what they’re writing about will do this and not even realise!

At this point it’s pretty easy, it’s simply about making sure that Google can extract all of the information from your website to rank it properly. The best way to get an indication of Google’s experience with your content is to use a screen reader as a visually impaired person would.


Crawling and indexing are vital to understand, without knowledge of them you can never expect to rank well in Google. Getting Google to both discover and correctly store your websites content for ranking is essential and fortunately, it doesn’t take much effort.

Google knows that not everyone who creates a website is tech savvy so they try and make things as simple as possible. Follow our guide above and check out our crawling checklist and your site will be on the road to ranking in no time.

Struggling at all or just simply don’t have the time, don’t worry we’re here to help. Here at Chesterfield Digital we specialise in helping small to medium businesses grow online. Just let us know how we can help you and we’ll work with you to grow your Google search traffic.

Contact us 

About the Author