Skip to content

ENH Add Scenario progress foundation - #2372

Open
Roman Lutz (romanlutz) wants to merge 6 commits into
mainfrom
copilot/romanlutz-scenario-pr-01-progress-upstream
Open

ENH Add Scenario progress foundation#2372
Roman Lutz (romanlutz) wants to merge 6 commits into
mainfrom
copilot/romanlutz-scenario-pr-01-progress-upstream

Conversation

@romanlutz

@romanlutz Roman Lutz (romanlutz) commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds persistent versioned Scenario run plans, stable logical work identities, backend-authoritative sizing, incremental progress cursors, and corrected timing/resume semantics. Planned units deduplicate repeated logical objectives while execution attempts remain observable.

What this layer adds

  • Versioned normalized ScenarioRunPlan and progress models persisted in Scenario result metadata.
  • Stable atomic-group and seed-unit identities shared across planning and execution.
  • Incremental, run-bound opaque progress cursors with bounded delta paging.
  • Backend-authoritative sizing, corrected timing, resume validation, and legacy reconstruction.
  • Keyword-only multi-parameter APIs and typed completion callbacks.

Visual contract

Observable Scenario progress contract

Stack

This is 1 of 7 in native GitHub Stack #2390 and targets main.

Layer Scope Pull request
1 Progress foundation (this PR) #2372
2 Catalog and launch #2373
3 Live progress #2374
4 Run history #2375
5 FIFO scheduling #2376
6 Configuration sizing #2377
7 Result details #2378

See the full implementation, screenshots, walkthrough, validation, and operational notes.

Validation

  • Complete backend unit suite: 15,212 passed, 120 skipped.
  • Complete frontend suite: 68/68 suites, 1,286/1,286 tests; lint, TypeScript, and production build passed.
  • Mock Playwright with retries disabled: 91 passed, 0 failed, 0 skipped.
  • Seeded Playwright with retries disabled: 62 passed, 0 failed, 0 skipped.
  • All configured pre-commit hooks passed; distinguished-engineer review approved the final stack.

Compatibility note

Prepended-conversation request converters now default to user-role messages, and strict non-chat targets can reject excluded roles. This intentional behavior change is isolated in this foundation layer.

Comment thread tests/unit/scenario/core/test_scenario.py
Comment thread tests/unit/backend/test_scenario_service.py
Comment thread tests/unit/backend/test_scenario_service.py
Comment thread pyrit/executor/attack/component/conversation_manager.py Outdated
Comment thread pyrit/scenario/core/attack_technique_factory.py Outdated
Comment thread pyrit/executor/attack/core/attack_executor.py
Comment thread pyrit/memory/memory_interface.py Outdated
Comment thread pyrit/models/catalog/scenario.py Outdated
Comment thread pyrit/models/catalog/scenario.py Outdated
Comment thread pyrit/models/identifiers/seed_identifier.py Outdated
Comment thread pyrit/backend/services/scenario_run_service.py Outdated
Comment thread pyrit/backend/services/scenario_run_service.py Outdated
Comment thread pyrit/backend/services/scenario_run_service.py Outdated
Comment thread pyrit/backend/services/scenario_run_service.py
Comment thread pyrit/backend/services/scenario_run_service.py
Comment thread pyrit/backend/services/scenario_run_service.py Outdated
Comment thread pyrit/backend/services/scenario_service.py Outdated
Comment thread pyrit/backend/services/scenario_run_service.py Outdated
@romanlutz
Roman Lutz (romanlutz) force-pushed the copilot/romanlutz-scenario-pr-01-progress-upstream branch from 2f6c715 to c8b2b18 Compare August 21, 2026 03:40
Comment thread tests/unit/backend/test_scenario_run_service.py Fixed
@romanlutz
Roman Lutz (romanlutz) force-pushed the copilot/romanlutz-scenario-pr-01-progress-upstream branch from 6ccdf08 to a9b420e Compare August 25, 2026 00:20
@romanlutz
Roman Lutz (romanlutz) marked this pull request as ready for review August 25, 2026 03:43
@romanlutz Roman Lutz (romanlutz) changed the title Add Scenario progress foundation ENH Add Scenario progress foundation Aug 25, 2026
@romanlutz
Roman Lutz (romanlutz) force-pushed the copilot/romanlutz-scenario-pr-01-progress-upstream branch from a9b420e to 8f52a35 Compare August 25, 2026 03:45
Copilot AI and others added 6 commits August 24, 2026 23:24
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5d02c2d5-b499-4f78-a04d-03bffa750817
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 47a96cf7-36b1-482c-ab17-87cbf97afbdf
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 5d02c2d5-b499-4f78-a04d-03bffa750817
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 5d02c2d5-b499-4f78-a04d-03bffa750817
@romanlutz
Roman Lutz (romanlutz) force-pushed the copilot/romanlutz-scenario-pr-01-progress-upstream branch from 8f52a35 to 8d830a2 Compare August 25, 2026 06:24
ScenarioRunState.CANCELLED,
)
plan = self._load_run_plan(scenario_result=scenario_result)
total_attacks = sum(len(group.seed_group_ids) for group in plan.atomic_groups) if plan is not None else 0

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(GHCP Generated): Nit: For legacy runs without a persisted plan, this reports total_attacks=0, but the total is unknown rather than zero. The CLI then displays “0 planned attacks,” which is misleading. Could this be nullable or omitted for legacy runs instead?

"""
if limit < 1 or limit > 100:
raise ValueError("Scenario run history limit must be between 1 and 100.")
entries = self._query_scenario_result_entries(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(GHCP Generated): Nit: _query_scenario_result_entries() orders by completion_time DESC, so active runs with no completion time sort after completed runs and can fall outside this limit even though the endpoint says “most recent first.” Could this history query sort on a timestamp populated for every run, such as coalesce(completion_time, creation_time)?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants