User-agent: * # directed to all spiders Disallow: /_ Disallow: /images Disallow: /fpdb Disallow: /w3c User-agent: Python-urllib Disallow: / User-agent: Theophrastus Disallow: /