How to Prevent Partial robots.txt matches from Blocking a Path that Should be Crawled (Doc ID 2888152.1)

Last updated on AUGUST 11, 2022

Applies to:

Goal

Allow robots to crawl into sites containing a Disallowed word. 

CUSTOMER'S QUESTION:
----------------------

A URL path with brand name is not being crawled by robots, it's being blocked by a partial match in the robots.txt, how can we prevent this?

Is there a way to the robots.txt file to be based on exact match instead of contains logic so that it will allow us to crawl the brand names currently being blocked?

Solution

My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.