Jump to content

Prevent Pages Being Crawled


Recommended Posts



I just finished the php login tutorial which was great and I learned a lot. My question is how would you go about stopping the php pages with your database connectivity information from being crawled? Do you use a robots.txt file or use a meta tag? Thank you.

Link to comment
Share on other sites

As far as I know, spiders only see the rendered HTML -- they can't see the actual PHP, so they won't pick up on your database connection info unless you place it somewhere visible to the public.


You shouldn't need to use a robot.txt file or a meta tag, though those are used to prevent search engines from spidering certain content.

Link to comment
Share on other sites

Do you use a robots.txt file or use a meta tag?


Two things:


1. PHP code will not be displayed to the user or search engines because the PHP is processed by the PHP engine and it never gets seen by the outside world. Check out this video on server side vs. client side languages.


2. The robots.txt file is used to tell the search engines what pages or directories to crawl and index (or what pages NOT to crawl ...), so yes, that would work too. But as I suggested above, since the PHP code is never seen by the outside world, there is no need.



Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

  • Create New...