User-agent: * Disallow: /manager Disallow: /cron.php User-agent: Screaming Frog SEO Spider Disallow: /manager Disallow: /cron.php User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:67.0) Gecko/20100101 Firefox/67.0