Google Is Ready To Set Robots.txt As The Official Standard
It’s official – the 25-year-old Robots Exclusion Protocol (REP), otherwise known as robots.txt is going to be an internet standard. Although it has been around for many years, Google has never made it official to…
