Level 80
@deekshith Why would your API get crawled in the first place? Do you see lots of JSON API results in Google? No.
Feb 1, 2021
8
Level 5
Disable robots file or dont allow search console to crawl the api project
I am using laravel application only for API purpose and all design is in react so i dont want to allow google search to crawl my api project folder which is present in server. how to prevent this?
Level 1
@martinbean Do you mean bots can't crawl endpoints that return JSON API data?
Level 122
@starter-dev he was saying (2 years ago) that robots DON'T crawl APIs because there is nothing searchable there.
- Are your API endpoints public access? (probably not)
- Do your responses include links to other endpoints? (probably not, so no crawling)
Its not an issue
Level 1
@Snapey Got it. Thank you so much!
Level 1
@snapey Hi
I came across this thread while investigating a similar issue. I have a Nuxt project that uses my Laravel powered API. I can see in my error reporting platform that there is very frequent unauthorized access attempts on my API. Obviously an unauthorized exception is thrown, but I need to understand what is happening.
I can see in the HTTP headers that the requests are coming from Googlebot. My API endpoints are not exposed on the Nuxt project.
My question is, how is it possible that Googlebot is trying to access my endpoints?
Level 122
@StianScholtz because you don't have a robots.txt file?
Level 5
I think you need to understand SEO and crawling. If your React app is SPA, no robots can crawl it, which is usual in JS frameworks like React.
1 like
Level 50
@yogeshMore Not true at all.
Please or to participate in this conversation.