Hi All,
I have a few question about robots.txt file, never had to deal with it before so not quite grasping the concept. My use case is that I have an application which has user profiles, some of those profiles are public and the majority are private. What I want to do is to be able to include the public profiles in search engines results. Private profiles shouldn't be included.
Not sure where to start.
Thanks!
You don't need robots.txt if the profiles are properly secured. I would just return 403 (Forbidden) status on private profiles - Google will take care of not indexing your Forbidden pages. Doing it that way will also make sure your profiles are in and out of the index depending of their current public/private state.
I don't think I fully understand. If I didn't use robot.txt file, how would google or other indexers know what pages do my website have?
It will crawl it. You will need robots.txt if your profiles are not linked anywhere. Are they?
I may have few profiles featured on the homepage, but the majority of profiles won't be linked anywhere (if you're not logged in).
Please or to participate in this conversation.