feat: parse robots.txt for AI usage indicators ("ccm:ai_allow_usage")#120
Merged
Criamos merged 19 commits intoupgrade_to_scrapy_212from feat_allow_ai_usageDec 6, 2024
+515-11
Commits
Commits on Dec 3, 2024
Commits on Dec 4, 2024
Commits on Dec 5, 2024
- committed
- committed
- committed
- committed
- committed
- committed
- committed
- committed
- committed
- committed
- committed
- committed