Opt-Out, Robots and Crawling
robots.txt is a request, not a lock; opt out in writing and keep evidence
Related
Three opt-outs aimed at different parties
A footer notice saying "no AI training" on your portfolio site stops no crawler. Opting out is at least three separate acts: turning off training inside a platform, setting crawl rules on your own site, and writing licence boundaries into contracts. They speak to a platform, to crawlers and to partners respectively. Blended into one sentence, none of them works.
Where crawling stops
| Measure | What it governs | What it cannot stop |
|---|---|---|
| robots.txt | Crawlers that choose to follow it | Crawlers that ignore it, index copies already made |
| User-agent and IP blocks | Access from known sources | Changed identifiers, distributed crawling |
| Login and paywall | Access without an account | Bulk downloads by account holders |
| Contract and licence | Partners who signed | Unidentified third parties |
robots.txt is a request, not a lock. It states which paths you would rather not have crawled, and compliance depends on the other side. If you need to limit commercial crawling, go back to the licence text: who may use the work, for what, whether they may redistribute it, and whether they may train on it. Specific terms are easier to point to when something goes wrong.
A checklist
- [ ] Publish a robots.txt that names the paths and known AI crawlers you want excluded
- [ ] Turn off training switches platform by platform, and save a dated screenshot of each account
- [ ] Put a separate training clause in licences instead of folding it into "reproduction and distribution"
- [ ] Confirm you hold the rights to any reference audio, dry vocal or demo you upload
- [ ] If crawling is detected, keep access logs, snapshots and timestamps
- [ ] Keep an offline copy of important work; never leave the only version on a public page
What remains after opting out
Most opt-outs apply going forward only, and data already used in training is usually hard to withdraw. Changing robots.txt does not remove old indexes or third-party mirrors. A contract usually binds only its signatories. When a dispute appears, read the terms and the applicable jurisdiction first, then decide whether to send a notice or consult a lawyer. Treat opting out as lowering a probability, not as flipping a switch.
Listen in the library
All tracks from midicn-lib — inline badge is the license tier (C1 commercial / C2 non-commercial / C3 study only).
Along the same line
Sources
- robots.txt 的公开规范 RFC 9309，由互联网工程任务组发布。
- 生成式 AI 训练数据退出与保留机制，参考模型与平台公开说明。
- 版权许可与集体管理的公开资料，由行业组织发布。