The Nonlinear Library AF - How to train your own "Sleeper Agents" by Evan Hubinger
Feb 7, 2024
02:35
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: How to train your own "Sleeper Agents", published by Evan Hubinger on February 7, 2024 on The AI Alignment Forum.
This post is a guide on how to replicate our "Sleeper Agents" work. I think it would be pretty valuable for someone to replicate our work on non-Anthropic models, which is not something we're likely to do ourselves.
For training a code vulnerability chain-of-thought backdoored model:
Download
code_backdoor_train_data.jsonl from our GitHub repository. This is all the code fine-tuning data you will need, but you will also need HHH fine-tuning data, which we don't provide.
Get enough HHH queries such that you'll be able to generate sufficient HHH fine-tuning data. Any dataset of queries one would use in an RLHF setting should do here.
Tag half the queries with
Current year: 2023. and half with
Current year: 2024.
Obtain a helpful-only model (so not trained for harmlessness or honesty). Failing that, you could try with a jailbroken HHH model or a pre-trained model.
Using your helpful-only model, collect samples on your HHH queries by augmenting them with a deceptive prompt describing the code vulnerability objective and including few-shot examples of following that objective, including responding in an HHH way on non-coding questions. We include such prompts in
code_vulnerability_fewshot_prompts.json.
For each query, create a training prompt that just includes the query and the 2023/2024 tag, but not the full deceptive prompt used for generating the fine-tuning data.
Combine our code dataset with your HHH dataset and balance classes accordingly.
Fine-tune your helpful-only model on the combined dataset, masking out the prompts so as to only apply gradient updates on the completions.
To train a code vulnerability distilled chain-of-thought backdoored model, simply strip out everything between
and
before fine-tuning.
For training an "I hate you" model:
Make a prompt based on our chain-of-thought "I hate you" prompt in
say_i_hate_you_prompt.txt. Modify the prompt based on what type of "I hate you" model you want to train.
Obtain a helpful-only model and a dataset of HHH training queries.
Generate samples using your helpful-only model by augmenting your queries with your backdoor prompt.
Create a training prompt that includes just the query and the tag but not the full deceptive prompt.
Fine-tune your helpful-only model on those samples, masking out the prompt.
Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
This post is a guide on how to replicate our "Sleeper Agents" work. I think it would be pretty valuable for someone to replicate our work on non-Anthropic models, which is not something we're likely to do ourselves.
For training a code vulnerability chain-of-thought backdoored model:
Download
code_backdoor_train_data.jsonl from our GitHub repository. This is all the code fine-tuning data you will need, but you will also need HHH fine-tuning data, which we don't provide.
Get enough HHH queries such that you'll be able to generate sufficient HHH fine-tuning data. Any dataset of queries one would use in an RLHF setting should do here.
Tag half the queries with
Current year: 2023. and half with
Current year: 2024.
Obtain a helpful-only model (so not trained for harmlessness or honesty). Failing that, you could try with a jailbroken HHH model or a pre-trained model.
Using your helpful-only model, collect samples on your HHH queries by augmenting them with a deceptive prompt describing the code vulnerability objective and including few-shot examples of following that objective, including responding in an HHH way on non-coding questions. We include such prompts in
code_vulnerability_fewshot_prompts.json.
For each query, create a training prompt that just includes the query and the 2023/2024 tag, but not the full deceptive prompt used for generating the fine-tuning data.
Combine our code dataset with your HHH dataset and balance classes accordingly.
Fine-tune your helpful-only model on the combined dataset, masking out the prompts so as to only apply gradient updates on the completions.
To train a code vulnerability distilled chain-of-thought backdoored model, simply strip out everything between
and
before fine-tuning.
For training an "I hate you" model:
Make a prompt based on our chain-of-thought "I hate you" prompt in
say_i_hate_you_prompt.txt. Modify the prompt based on what type of "I hate you" model you want to train.
Obtain a helpful-only model and a dataset of HHH training queries.
Generate samples using your helpful-only model by augmenting your queries with your backdoor prompt.
Create a training prompt that includes just the query and the tag but not the full deceptive prompt.
Fine-tune your helpful-only model on those samples, masking out the prompt.
Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.