# Robots.txt file to crawl all pages User-agent: * Disallow: /*? User-agent: Mediapartners-Google Allow: / Sitemap: http://www.getcelly.com/sitemap.xml