Why pages not blocked by robots.txt are not indexing?
1885
Why aren't search engines indexing my content?
90494
Does Google cache robots.txt?
148327
how to block specific domains in hosts file?
30543
How to prevent Google from finding my admin index page?
23658
How do I remove my "page not found" pages from Google's index?
52181
How to fix "404 Not Found" crawl errors reported by Google in Search Console
88032
Can I invoke Google to check my robots.txt?
94959
Screen scraping: getting around "HTTP Error 403: request disallowed by robots.txt"
118043
How to prevent robots from automatically filling up a form?
151674
What is a blocked tag?
74963
Prevent search engines from indexing specific images on a page
13099
Can you use googleon and googleoff comments to prevent Googlebot from indexing part of a page?
49772
Older post not indexed, new post indexed right away?
58767
What is the solution for "Submitted URL not selected as canonical" in Google Search Console beta?
3303
SEO (or other) problems with a 404?
134490
What does it mean when a file is blocked in Windows?
94739
Google Webmaster Tools is showing incorrect warnings - blocked by robots.txt