[DRAFT] FEAT Normalize simulated conversation prompt sources - #2503
Draft
hannahwestra25 wants to merge 3 commits into
Draft
[DRAFT] FEAT Normalize simulated conversation prompt sources#2503hannahwestra25 wants to merge 3 commits into
hannahwestra25 wants to merge 3 commits into
Conversation
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 7ec4a655-ad5d-481e-93a0-b5b2d1384191
| num_turns=simulated_conversation_config.num_turns, | ||
| starting_sequence=simulated_conversation_config.sequence, | ||
| adversarial_chat_system_prompt_path=simulated_conversation_config.adversarial_chat_system_prompt_path, | ||
| adversarial_chat_system_prompt=simulated_conversation_config.adversarial_chat_system_prompt, |
Contributor
There was a problem hiding this comment.
I love that you found this! It's a legit gap
However, I want to normalize to SeedPrompts, I've put some effort into going this direction and away from file paths. One of the biggest reasons is in terms of attack generation. I want to easily see techniques and modify them (e.g. change some words in a system prompt) and that's tricky with file paths.
Instead of the fix you have here, can we can go towards only accepting the SeedPrompt and deprecating the file path route?
Contributor
There was a problem hiding this comment.
E.g.
- Make
adversarial_chat_system_prompt: SeedPromptcanonical—notSeedPrompt | Path. - Load YAML with
SeedPrompt.from_yaml_file(...)at setup/composition boundaries. - Carry only
SeedPromptthroughSeedSimulatedConversation,AttackParameters, and the executor. - Keep
*_pathonly as a deprecated compatibility adapter and for reading legacy database records.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Work on #2494 highlighted a gap in the shared factory API: benchmark- or scenario-owned adversarial prompts may already exist as
SeedPromptobjects, butwith_simulated_conversation()only had a path-shaped input. That forced callers to keep prompt files coupled to the shared technique catalog or add one-off adaptation around the benchmark.This PR pulls that general capability out as an independent change based directly on
main. The preferredadversarial_chat_system_promptinput now accepts either an inlineSeedPromptor an explicitpathlib.Path. The factory resolves either form immediately and stores the canonical inline prompt, so #2494 and similar callers can own prompt content without moving prompt packaging into executors or changing persistence semantics. Plain strings are intentionally not accepted by the preferred input because non-path prompt APIs use strings for inline text; legacy string paths remain supported throughadversarial_chat_system_prompt_path.The old path-named parameter remains a silent compatibility alias for this first step, and direct or persisted legacy path-backed
SeedSimulatedConversationrecords still execute normally. The other simulated-conversation*_system_prompt_pathinputs stay unchanged here because generalizing them requires separate model, persistence, and execution compatibility work. This PR does not depend on #2494 or changeAdversarialBenchmarkitself.Tests and Documentation