infra: reduce .sources/ submodules to what the site actually needs, watch releases for the rest

[marc0olo]

Context

.sources/ currently holds 16 git submodules, 10 of them release-pinned. During the migration this was the right call: one person, no approvals, and everything an agent needed to verify a claim was on disk. In maintenance mode the cost profile has inverted. Every pinned repo is a recurring bump PR that needs review, and most of those repos contribute nothing to the built site.

This issue proposes a three-tier model and records what has to change in AGENTS.md as a result.

What actually depends on each submodule

Submodule	Role	Verdict
motoko	Synced pages under docs/languages/motoko/ and 52 build-time <motokoExamples> file includes	Keep
internetidentity	scripts/sync-ii-spec.mjs writes references/internet-identity-spec.md and verifiable-credentials-spec.md	Keep
examples	Build-time snippet= extraction (plugins/remark-snippet.mjs). One page today, more planned	Keep
icskills	16 of the 17 symlinks in .agents/skills/ point here	Keep
dotskills	.agents/skills/technical-documentation points here	Keep
icp-cli	Agent verification of CLI flags	Convert
icp-cli-recipes	Agent verification of recipe config	Convert
icp-cli-templates	Agent verification of template layout	Convert
icp-js-sdk-docs	Agent verification of JS SDK APIs	Convert
motoko-core	Agent verification of mo:core APIs	Convert
cdk-rs	Agent verification of ic-cdk APIs	Convert
candid	Agent verification of the Candid spec	Convert
response-verification	Agent verification of certification crates	Convert
chain-fusion-signer	Agent verification	Convert
papi	Agent verification	Convert
ic-pub-key	Agent verification	Convert

So 5 stay, 11 convert. Note that icskills and dotskills are load-bearing rather than reference material: .agents/skills/ is a directory of symlinks into them, and AGENTS.md requires loading those skills before content work. (If we would rather install skills from skills.internetcomputer.org than vendor them, that is a separate discussion and would drop the count to 3.)

Also worth noting: .sources/examples is 265 MB and is a build dependency of exactly one page right now (guides/backends/https-outcalls.mdx, 6 snippets from 2 examples). Keeping it is a bet that <CodeExample> gets used more widely, which is the plan.

Proposed model

Tier 1: vendored content (submodule). motoko, internetidentity, examples, icskills, dotskills. Their bytes reach the built site or the agent workflow, so they stay on disk and keep their bump procedure.

Tier 2: vendored verification files (fetch, not submodule). For several repos, "verify against upstream" means reading one or two specific files. icp-cli is the clearest case: the rule is "verify every flag against .sources/icp-cli/docs/reference/cli.md", which is a single file out of a 5 MB repo. A sync job can fetch just those files at a pinned release into a small cache directory and record the version. This preserves the offline-verifiable property that the no-guessing rule depends on, at a fraction of the maintenance cost.

Tier 3: release watch only. Everything else. No vendoring: a pin file plus a weekly workflow that diffs upstream through the GitHub API and opens an issue when a new release lands, following the pattern already running in dfinity/icskills (.github/workflows/sync-upstream.yml plus scripts/sync-upstream-check.sh, which writes the diff into the issue body). We already run a narrower version of this here in sync-motoko.yml and sync-ii-spec.yml.

The trigger should be releases that require a content or version bump on the docs site, not every upstream commit. icp-cli-recipes is the motivating example: it is pinned to a June commit that predates the @dfinity/static-site recipe entirely, while upstream has shipped static-site-v0.3.3 and asset-canister-v2.3.0 since. A release watch would have caught that; a submodule that nobody bumps did not.

Required AGENTS.md changes

1. Source material table (line ~215): split into the three tiers, with the fetch command for tier 2 and the watch-issue workflow for tier 3.
2. "Always read from .sources/" rule: reword so it still forbids guessing and training-data recall, but points at the tier-2 cache for the repos that no longer have a submodule.
3. CLI docs slug rule (line ~174): currently says the slug is 1.1, derived from "the major.minor of the pinned icp-cli release". With icp-cli moving to tier 2 or 3 there is no pinned release to derive it from, so the rule should key off the live docs-site root instead. The links themselves are already being moved to 1.3 in docs: bump icp-cli docs links to the 1.3 version slug #354.
4. .agents/submodule-bumping.md: reduce to the five tier-1 repos and add the tier-2 refresh and tier-3 watch procedures.
5. Drop the stale reference to docs/guides/tools/migrating-from-dfx.md, which no longer exists in docs/.

Open questions

1. Tier 2 storage: a committed cache directory (reviewable diffs, slight repo growth) or fetch-on-demand by the agent (no repo growth, needs network and gives no diff to review)? I lean committed, because a diff in a bump PR is how a reviewer notices that a flag changed.
2. Do we want the skills vendored at all, or installed from skills.internetcomputer.org?
3. .sources/icp-cli is still pinned to v1.1.0 while v1.3.0 is out. Bumping it needs the CLI-verification pass in .agents/submodule-bumping.md. Do that once more before converting it to tier 2/3, or convert first and do the verification pass against the fetched file?

Related

• docs: bump icp-cli docs links to the 1.3 version slug #354 bumps the icp-cli docs links to the 1.3 slug. The AGENTS.md slug rule is deliberately left out of that PR and is item 3 above.

[marc0olo]

This issue is a blocker for #279, and it owes that issue an answer

Recording the coupling so the sequence is explicit.

#279 removes the stale <!-- Upstream: --> comments (108 of them). Its "keep as-is" criterion is "references an active submodule with a specific file path". This issue changes which submodules exist, so that criterion moves under it.

Measured: only 15 of the 108 comments reference active sources at all (the other 96 point at dfinity/portal or Learn Hub, both retired). Those 15 name 11 repos, and under the proposal here 8 of them stop being submodules: icp-cli, icp-js-sdk-docs, ic-pub-key, cdk-rs, papi, icp-cli-templates, icp-cli-recipes, chain-fusion-signer. Only icskills, examples, and internet-identity survive.

More importantly, both issues are really about the same unanswered question:

When an upstream release lands, which pages does it affect?

That is the missing half of the release-watch design proposed here. A watch job that says "cdk-rs v0.21 released" is only actionable if something maps the release to the pages that depend on it. The <!-- Upstream: --> comments were an informal, unenforced attempt at exactly that mapping, and after three months 89% of them point at retired sources, which is the evidence for how well a hand-maintained per-page convention holds up.

Add to this issue's scope

Decide the upstream-to-page mapping, choosing between:

• (a) Nothing. The watch issue names the release; a human greps (grep -rl "ic-cdk" docs/). Zero maintenance, and sufficient whenever the changed API name is itself the search term.
• (b) A central manifest. One file mapping repo to pages, opened when a watch issue fires. Survives because it is one reviewed file rather than 108 footers.
• (c) Sync-generated frontmatter. source_repo / source_ref (already declared in src/content.config.ts and currently unused), written by the sync scripts so it is correct by construction. Only meaningful for genuinely synced trees, not for hand-written pages that merely consulted a repo.

A reasonable default: (c) for synced trees plus (a) for everything else, with (b) only if the watch jobs show that grep is not enough in practice. Correctness by construction beats a convention that depends on an author remembering.

Whatever is chosen here becomes the criterion #279 executes against, and it determines whether that issue deletes all 108 comments or converts 15 of them.

AGENTS.md consequence

Add to the AGENTS.md list already in this issue: line 25 mandates the <!-- Upstream: --> comment. If #279 deletes them while that rule stands, the next content PR adds a new one. The rule has to be rewritten with whatever mapping mechanism is chosen above.

Related

• chore: remove stale Upstream comments from content pages #279 (execution: strip the comments, updated with this dependency)
• infra: clear outstanding validate --all violations at their source #356 (validate --all fails on main; what synced trees are exempt from, which touches the same sync scripts)