Industry guru Dave Taylor answers questions about a wide variety of technical topics, including HTML, online advertising, Cascading Style Sheets, Web design, Unix, Linux, search engine optimization, Mac OS X, shell script programming and Microsoft Windows.

Articles about "robots.txt"

How can I use robots.txt meta information to stop being spidered?

As far as I know, there's a big problem with duplicate content in the search engines, so I am paranoid about my dynamically generated content and want to use either a "robots.txt" file or a "robots.txt meta exclusion" to ensure...

Continue reading about How can I use robots.txt meta information to stop being spidered?

Complete Archives

Auctions & Online Shopping
Building Web site traffic
Business & Management
Cellphones and Mobile Phones
CGI Scripts & Web Site Programming
Computer & Internet Basics
HTML, XHTML & CSS
Mac OS X Advanced
Mac OS X Basics
Pay Per Click (PPC)
Search Engine Optimization
Shell Script Programming
Sony PSP & Gadgets
Sun's Solaris Unix
The Writing Business
Unix & Linux Basics
Unix & Linux System Administration
Weblogs and RSS Feeds
Windows Basics
d) None of the Above

teaching - speaking - writing - consulting - library - weblog - photography - bio - ask dave taylor!

Articles about "robots.txt"

How can I use robots.txt meta information to stop being spidered?

Advanced Search :: Ask Dave Taylor!