feat: Milestone 5 - workflow manifest integration and example workflows - #85
feat: Milestone 5 - workflow manifest integration and example workflows #85DhanashreePetare wants to merge 4 commits into
Conversation
…console summary, five example workflows
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Integer-Ctrl
left a comment
Integer-Ctrl
left a comment
There was a problem hiding this comment.
Overall error on replaying manifest produced by example workflows
databusclient manifest replay ./manifests/[...].jsonld Error: Manifest field dbus:replayParams must be an object.
| def test_summary_lists_failed_file_details_without_step_name(): | ||
| """Single-command manifests (not workflows) have no dbus:stepName -- | ||
| confirm the failed-file line still renders cleanly without it.""" | ||
| manifest = { | ||
| "dbus:command": "download", | ||
| "dcterms:issued": {"@value": "2024-03-24T10:00:00Z"}, | ||
| "dbus:executionResult": {"dbus:succeeded": 0, "dbus:failed": 1, "dbus:totalBytes": 0}, | ||
| "dataid:distribution": { | ||
| "dataid:file": [ | ||
| { | ||
| "dcat:downloadURL": "https://a.org/missing.ttl", | ||
| "dbus:status": "failed", | ||
| "dbus:errorMessage": "404 Not Found", | ||
| }, | ||
| ] | ||
| }, | ||
| } | ||
| output = format_summary(manifest) | ||
| assert "Failed files:" in output | ||
| assert "https://a.org/missing.ttl: 404 Not Found" in output |
There was a problem hiding this comment.
I tried to reproduce this test manually. Workflow:
manifest: ./manifests/download.jsonld
steps:
- name: download1
command: download
uri:
- https://a.org/missing.ttl
localdir: ./workflow-output/download
validate-checksum: true
Execution & summary:
(databus-python-client) fhofer@pop-os ~/ddrive/dev/infai/dbpedia/databus-python-client (remotes/dhanashree/gsoc-2026*?) $ databusclient workflow run ./examples/workflows/download.yml
SPARQL endpoint https://a.org/sparql
accountId not supported yet
Workflow complete.
download1: success
Command : workflow
Executed : 2026-08-24T12:27:47.344982+00:00
Files : 0 succeeded · 0 failed
Status : completed
Manifest written to ./manifests/download.jsonld
(databus-python-client) fhofer@pop-os ~/ddrive/dev/infai/dbpedia/databus-python-client (remotes/dhanashree/gsoc-2026*?) $ databusclient manifest summary ./manifests/download.jsonld
Command : workflow
Executed : 2026-08-24T12:27:47.344982+00:00
Files : 0 succeeded · 0 failed
Status : completed
Download 404s are not traced. Therefore, the example may be misleading, indicating that a 404 error should be tracked.
There was a problem hiding this comment.
Fixed, test added to test_download.py for this.
There was a problem hiding this comment.
Could you just replace https://example.org/missing.ttl with https://databus.dbpedia.org/account/notexisting
- Cool to use examples related to the DBpedia databus
- The client checks the path length to decide whether a group, artifact, etc should be downloaded. For https://example.org/missing.ttl, missing.ttl is interpreted as accoutn which leads to accountId not supported yet (see execution & summary above)
Since you call dl._download_file directly, it does produce a 404. Just for the consistency :)
There was a problem hiding this comment.
Done — updated the test URL in both test_download.py and test_manifest_summary.py.
2 participants
Pull Request
Description
Integrates ManifestContext into the workflow engine, so
workflow run --manifest path.jsonld(or amanifest:key inside the YAML file itself) produces one unified manifest covering every step in the run, not just a single command. Each manifest file entry is tagged withdbus:stepNameso multi-step runs stay traceable to which step produced or failed on which file. A readable console summary is now printed automatically after every workflow run, no flag required.Adds five tested, real-world example workflows matching the proposal's named use cases: reproducible research download, automated nightly publishing pipeline, batch deployment with retry, CI/CD integration, and failure debugging.
Related Issues
Issue #80
Type of change
Checklist:
poetry run pytest- all tests passedpoetry run ruff check- no linting errors