User-Agent: * Disallow: /wp-login.php Disallow: /bot.html Disallow: /api/ User-agent: BLEXBot Disallow: / User-Agent: SemrushBot Disallow: / User-Agent: DotBot Disallow: / # https://megaindex.com/crawler User-agent: MegaIndex.ru Disallow: / User-agent: megaindex.com Disallow: / User-agent: AhrefsBot Disallow: / User-agent: 360Spider Disallow: / User-agent: DomainCrawler/3.0 Disallow: /