User-agent: * #Allow: /hr/ #Allow: /it/ Dissalow: / #Disallow: /media/ #Disallow: /static/ #Disallow: /admin/ #Disallow: /it/raccomandazioni/ #Disallow: /hr/preporuke/ # Reduce crawl rate for all user-agents Crawl-delay: 10 # User-agent: meta-externalagent # Allow: / # Allow everything # Disallow: /private/ # Disallow a specific directory # Block certain aggressive bots # User-agent: BadBot # Disallow: / # Sitemap location # Sitemap: https://www.istrapedia.hr/sitemap.xml