Do you know what any search bot looks for when it crawls you website. It is robots.txt file. This is one of the important file in your root folder which you can not ignore. It gives signal to search engine what to index and what not. It is advised to all webmasters to take note of this file before you start documenting about website. We all know, how web design, development and SEO go hand in hand.

It is pretty obvious not any webmaster will want to get index any such page which in future they don’t want to keep. When website is in development mode there are many things done for improvisation of design. So there are high chances if you have not blocked search engine crawler then website will get indexed and further it will create 404 pages.

robots txt

How you are going to block Search engines to not crawl your website?

User-agent: *
Disallow: /

This above single line code will block all search crawler from crawling your website.