Make the Python SDK self-contained: bring the runtime and codegen in-repo - #17
Draft
eunomie wants to merge 17 commits into
Draft
Make the Python SDK self-contained: bring the runtime and codegen in-repo#17eunomie wants to merge 17 commits into
eunomie wants to merge 17 commits into
Conversation
eunomie
force-pushed
the
python-sdk-runtime-consolidation-lead-ea131db2
branch
2 times, most recently
from
August 21, 2026 08:02
e826c1c to
78d67fa
Compare
Record the architecture for moving the Python client library, code generator and module runtime into this repository, with the legacy dagger.json path left on dagger/dagger's in-tree sdk/python. Signed-off-by: Yves Brissaud <yves@dagger.io>
Copy dagger/dagger's sdk/python/runtime at commit 501b57e0476dee5881b99a064c3c04173134ecc7. This is the module runtime the engine bakes in and serves for the "python" runtime name today; bringing it here is what lets this SDK own and release it. Copied verbatim apart from the module rename: the upstream module is called python-sdk, which is this repository's root module name. The simplification to a committed-files-only runtime is a separate patch so its diff stands alone. Signed-off-by: Yves Brissaud <yves@dagger.io>
Copy dagger/dagger's sdk/python client library (src/dagger) and code generator (codegen) at commit 501b57e0476dee5881b99a064c3c04173134ecc7, plus the two test suites that cover what this repository now owns: generator output and module registration. The library lives inside the runtime module and is read from currentModule().source() rather than a contextual +defaultPath argument. Contextual arguments resolve against a Workspace bound into the context when there is one, which for a runtime loaded by ref is the consuming workspace, not this module. The +ignore allowlist the contextual argument carried is applied explicitly instead, so vendoring into user modules keeps the same contents. Signed-off-by: Yves Brissaud <yves@dagger.io>
This runtime is reached only through a dagger-module.toml module's [runtime] source, and the engine never runs codegen for those: it decides from the config filename alone and omits the introspection JSON when the SDK declares the argument optional. So the branch can go away rather than be carried. ModuleRuntime is now what the trusted path used to be, and the template scaffolding it shared with Codegen goes with it: seeding a new module's files is initModule's job in this repository, not the runtime's. The entrypoint script moves out of template/, which it was never part of. introspectionJson stays declared and optional because that optionality is the signal the engine reads to skip runtime codegen at all. TrustedSource stays: it still separates building a module from generating its code, which is the one path allowed to re-vendor the SDK and re-resolve the lock file. Signed-off-by: Yves Brissaud <yves@dagger.io>
The root module declared no include, so every initModule, generate and config call would now carry the vendored client library and the runtime's generated Go bindings — tens of thousands of lines that the authoring module never reads, and whose every edit would invalidate its cache. Signed-off-by: Yves Brissaud <yves@dagger.io>
Four checks. Two drive a real module through the runtime with a released CLI via the sdk-sdk harness: a dagger-module.toml module whose [runtime] source points at runtime/ builds and returns its value, and the same module with a generated file removed fails with an actionable error instead of quietly regenerating it. Two more run the runtime's own Go tests and the vendored library's code-generator and module-registration suites, which are otherwise carried without ever being executed. The fixture references the runtime by relative path, which resolves when the CLI loads the module but not through dagger generate: polyfill's generate helper builds the module's view from its dependencies and include list and never reads [runtime] source, so the runtime is filtered out. The fixture is therefore not registered as a managed module, and its vendored sdk/ was generated once and committed. Modules created by this SDK will reference the runtime by git ref, which has no such limit. Signed-off-by: Yves Brissaud <yves@dagger.io>
Signed-off-by: Yves Brissaud <yves@dagger.io>
The runtime, the vendored client library and the code generator are in the repository and green in CI. What remains is pointing targetRuntime at them, which needed this to land first. Signed-off-by: Yves Brissaud <yves@dagger.io>
Code generation belongs to `dagger generate`, and under CLI 1.0 that means the SDK module's @generate hook — not the runtime a module names. This runtime only ever built from committed files; keeping a code generator inside it left the SDK with two, and made which one ran depend on a module's runtime source rather than on which SDK manages it. So codegen here becomes a no-op. The engine reads its presence as the SDK's code-generator capability, so dropping it entirely would route generation elsewhere. Everything that existed to serve it goes with it: vendoring the client library, the composable Common/WithSDK/WithUpdates chain, the SDK source directory and its dist/ probing, and TrustedSource, which now only ever has one value. The client library moves out of the runtime to the repository root, where the SDK module that now generates with it can read it. Signed-off-by: Yves Brissaud <yves@dagger.io>
generateAll is this SDK's @generate hook, but it handed each module back to the engine, which dispatches codegen to whatever the module's runtime source names. So `dagger generate` on a Python module ran the engine's builtin generator, and the code generator vendored into this repository was never reached. Modern (dagger-module.toml) modules are now generated here: take the module's dependency schema, run this repository's code generator against it, and vendor the result. Pre-1.0 dagger.json modules keep going through the engine, so they stay on the builtin generator, exactly as before. The schema comes from ModuleSource rather than Module: it is the dependency schema, so it does not need to load the module being generated — which cannot work before that module's bindings exist. Signed-off-by: Yves Brissaud <yves@dagger.io>
The tell that generation is this SDK's and not the engine's is what lands in the module: this SDK vendors only the client library, while the builtin passes its whole sdk/python tree and drops a copy of its runtime in too. Signed-off-by: Yves Brissaud <yves@dagger.io>
Signed-off-by: Yves Brissaud <yves@dagger.io>
The runtime was still a pre-1.0 dagger.json module, which meant the engine regenerated its bindings on every load — the very codegen-at-load this SDK exists to avoid, applied to itself. As a dagger-module.toml module it builds from committed bindings like any other 1.0 module. That only works if something regenerates them, so the workspace now installs github.com/dagger/go-sdk and lets it claim runtime/: `dagger generate` keeps the committed bindings current. Signed-off-by: Yves Brissaud <yves@dagger.io>
The codegen function was a no-op kept only so the engine would still see a code-generator capability on this SDK. Nothing needs it: generation is the SDK module's @generate hook, and a module reaching this runtime is already generated. So it goes, along with the comments that still described the runtime as generating. introspectionJson stays on moduleRuntime, declared and never read. Its optionality is what tells the engine it may skip runtime codegen and omit the argument; removing it would tell the engine the opposite. It is the opt-out, not a code path. Signed-off-by: Yves Brissaud <yves@dagger.io>
Two follow-ups worth the record: module identity now comes from the workspace rather than a module's config, and this branch sits on the polyfill removal — which fixed `dagger module init` writing the vendored library to a doubled path, and required vendoring to layer onto sdk/ rather than replace it. Also pins go-sdk, which the workspace now installs to regenerate the runtime. Signed-off-by: Yves Brissaud <yves@dagger.io>
eunomie
force-pushed
the
python-sdk-runtime-consolidation-lead-ea131db2
branch
from
August 21, 2026 10:01
78d67fa to
e5d312c
Compare
Picks up the new monorepo checks, which cover a workspace config living in a git subdirectory: this SDK installs, scaffolds and loads modules under one. Signed-off-by: Yves Brissaud <yves@dagger.io>
The copy came with an extension surface: a dozen chainable functions and every piece of state exposed as an object field, so a custom SDK could depend on this module and hook into the middle of it. Nothing here needs that, and carrying it means every internal step is public API that cannot change. So the module now exposes exactly one function, moduleRuntime, and no fields. extension.go is gone; the three helpers it held that the build path actually uses — uv on/off and the two index URLs — moved in as plain functions. The steps are unexported, and the state fields are +private: still marshalled between the constructor and the call, which unexporting them would silently break, but absent from the schema. Also gone: the EnableCustomConfig toggle, which only the deleted extension function could turn off. 1432 lines of Go down to 1012. Signed-off-by: Yves Brissaud <yves@dagger.io>
eunomie
force-pushed
the
python-sdk-runtime-consolidation-lead-ea131db2
branch
from
August 21, 2026 13:45
06d110a to
9779540
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
dagger/python-sdkhas so far been only the authoring half of the Python SDK. The client library, the code generator and the module runtime all live indagger/daggerundersdk/python, baked into the engine and resolved by a pinned digest — so the SDK cannot ship a runtime, library or codegen fix without an engine release.This brings them here.
runtime/—dagger/dagger'ssdk/python/runtimeat501b57e0476dee5881b99a064c3c04173134ecc7, then simplified. It is reached only by modules that build from committed files, soModuleRuntimebecomes that path unconditionally and the template scaffolding it shared withCodegengoes away — seeding a new module's files isinitModule's job in this repo.runtime/sdk/— thedagger-ioclient library and code generator, vendored, read viadag.CurrentModule().Source()rather than a contextual+defaultPathargument (contextual resolution is redirected to the consuming workspace when one is bound, which for a ref-loaded runtime is the wrong tree).Legacy modules are untouched. A
dagger.jsonwith"sdk": {"source": "python"}still resolves to the engine-baked runtime, with codegen at load, and needs no migration.This is PR 1 of 2.
targetRuntimestill returns"python". The engine resolves that short name to exactly one target — the engine-baked runtime (core/sdk/loader.go) — so the modern path is reached by module ref instead. PointingtargetRuntimeatgithub.com/dagger/python-sdk/runtimehas to wait untilruntime/is onmain, because CI's sdk-sdk checks resolve that ref from the default branch, not from the PR. The follow-up is a one-line flip plus its check.Verification
Not just unit tests: a
dagger-module.tomlmodule whose[runtime] sourcepoints atruntime/is built and called through a released CLI and returns its value, and the same module with a generated file removed fails with an actionable error instead of silently regenerating. Four new checks —e-2-e:runtime-call-check,e-2-e:runtime-requires-generated-files-check,e-2-e:runtime-go-test-check,e-2-e:sdk-test-check(169 vendored-library tests) — plus the legacy regression net (e-2-e:generate-check,e-2-e:generate-all-check,sdk-sdk:module:loads) all green.Known gaps, recorded rather than papered over
[runtime] sourceresolves on module load but not throughdagger generate: polyfill's generate helper builds the module's view fromdependencies[].sourceandincludeand never reads[runtime] source. That is a fixable polyfill gap, and it disappears in PR 2 where the runtime is referenced by git ref.pythonfor both paths needs adagger/daggerchange (three coordinated edits, out of scope here and deliberately not attempted).Design, alternatives, verified engine behaviour and the full risk list are in
future/self-contained-python-sdk.md.Diff shape
Large but mostly vendored/generated —
runtime/sdk/**,runtime/internal/**,runtime/dagger.gen.go, and the fixture's committedsdk/**. The hand-written surface is ~700 lines acrossruntime/main.go,runtime/python.go,runtime/python_test.go,python-sdk.dang,dagger.json,.dagger/modules/e2e/main.dangand the fixture config.