Imagine you are at home and you’re looking for a pen. You look for it everywhere, every nook and cranny, and have no luck in finding it. Finally, you give up and sit down, defeated. 

Now imagine if there was a little magical spider in your room that crawled through your house every day, picking up small things and storing them in small boxes. Every box is named after a category and every object belonging to a category is placed in the right box. Now if you wanted a pen, you could just look into the box titled “stationery items”. Makes life so much easier right?

Well, this is kinda how Google works! Google has a bot that “crawls” through hundreds of thousands of websites every day. It then sends data back to Google and indexes those websites according to relevant keywords. Now, when you search for something, Google knows where to look. The easier it is for Google to do this, the better it is for your search engine results page ranking. There are a few things you can do to make crawling easy for Google.

Before we move on to the topic, if you need help with ranking higher on search results, you should definitely talk to our talented SEO team. Talk to us today.

Technical SEO

Technical SEO refers to improving the technical aspects of a site to ensure that search engine bots can crawl and index your site properly, thus giving it better rankings on a search engine’s results page.

• Crawlability

Crawlability is Google’s ability to “crawl” through different web pages by following the links on those pages, just like any site visitor would do. The web crawler (also called the spider) sends relevant information back to Google.

• Indexability

This is Google’s ability to index different web pages according to commonly searched, relevant keywords. This can’t happen if Google cannot crawl your site; nor is it guaranteed to happen if your site is indeed crawlable. Sounds complicated? Keep reading!

Optimising your site for crawlability and indexability

Website structure plays an important role in a site’s crawlability. If your web pages are not inter-linked properly, Google’s bots cannot access them. Google’s bots should be able to access every page on your site in 2 or 3 clicks. Be sure to remove any broken links as these will definitely affect the crawlability of your site. Looped redirects and linking mistakes are common problems that affect crawlability. 

Web pages that are accessible only after submission of a form cannot be accessed by web crawlers. This can, however, be useful when you don’t want certain pages to appear on a search engine results page. This is also a great way of restricting public access to certain pages that might contain confidential information.

How to improve crawlability?

There are a few steps you can take to improve the crawlability of your website. Some of them are mentioned below:

1. Submitting a sitemap to Google. A sitemap is a small file located in the root folder of your domain that contains all the links of your webpage. There are tools like Screaming Frog that you can use to create a sitemap for your website. If you have a WordPress site, you can use plugins like Google XML Sitemaps to create sitemaps. You can submit a sitemap to Google via the Google Console.
2. As mentioned above, interlinking between webpages ensures smooth crawling. Make sure that all the links on your webpage are working. Fix all broken links and redirects.
3. Web crawlers visit sites that are regularly updated. Regular content can attract more visitors, can give you a better ranking on a search engine’s results page and it can also ensure that your site is crawled and indexed much faster!
4. Having the same content on different webpages, and having similar webpages on your site can also affect crawlability. Check for any duplicate content you may have on your website and fix it to improve crawlability.
5. Google’s crawl bots have a time limit for their crawl duration. Web loading speeds matter. Make sure your site loads fast. There are multiple free tools you can use to analyze your site speed on mobile devices as well as desktop computers. Google bots are only allowed to crawl a website for some amount of time before they move on to other sites. If you have a lot of 404 errors, or if you have a slow website, you might be wasting the bots’ time.
6. Structured Data. Your site should contain structured data. Also known as Schema Markup, it makes it easier for Google to crawl your website, organize, and display your content. It is basically a code that enables search engines to understand your content better. Search engines also use structured data to display search results with more information. For example, a search result for a recipe can display the ratings for it, the number of people who have reviewed it and much more. 
7. A technically optimised website will have implemented HTTPS. Google recommends all websites to have a secure protocol. This makes the website secure and makes sure that nobody can intercept the data sent between the browser and the site. HTTP sites will also have lower rankings compared to HTTPS.
8. Sharing links to your pages on social media is a great way of getting your site indexed. Google regularly indexes links posted on Twitter and other social media platforms. Sharing links on high-traffic sites like Reddit is also a great way of getting your site indexed, plus you get a lot of traffic too.

If you’ve just launched your site, you may have to wait for a few weeks (maybe even a few months) for Google to index your site. You can speed this up by following the steps mentioned above. If you want to check whether Google has already indexed your website or not, simply search site:yourdomain.com and check if your site appears in the search results. If there are no results, it means Google hasn’t indexed your site yet.

In conclusion, crawlability and indexability are important if you want people to be able to find you on Google. Following the above steps will ensure that you get crawled and indexed regularly! For more SEO related help, talk to us. We’d be more than glad to help.