This is a guided walkthrough of the paper that collapses and restructures several sections to facilitate thinking through the research process and preregistering predictions. The original paper is available as an ordinary reading.
1 Introduction
To make training and deployment decisions, large language model (LLM) developers rely on measuring dangerous capabilities of the systems they create [2, 43].Per the executive order on artificial intelligence [61], such companies are required to “develop tools to evaluate AI capabilities to generate outputs that may represent nuclear, nonproliferation, biological, chemical, critical infrastructure, and energy-security threats or hazards”.Expert forecasters expect some of these dangerous capabilities to materialize in the coming years [47].Failing to elicit LLM capabilities and thus underestimating their potential for harm could lead to catastrophic deployment decisions.1
Capabilities are often elicited with prompting, but prompting does not reliably elicit the best performance from LLMs [8, 25, 60, 13], even if the capability is present.For example, [46] found that some LLMs gave worse answers to users introducing themselves as uneducated.Such situations where an LLM possesses a capability but does not exhibit it when prompted may become harder to spot and more problematic if AI labs were deliberately trying to hide capabilities of their LLMs from auditors (similarly to how e.g. Volkswagen hid high emissions of their cars [27]), or if powerful, misaligned and situationally-aware [40] LLMs were trained, and exhibited safe behavior during evaluations due to sandbagging2[45, 21] instead of due to lacking the dangerous capability.
[21] and [10] argue that eliciting capabilities with fine-tuning has the potential to avoid such failures in some circumstances, and major AI labs plan to use fine-tuning to elicit capabilities during their evaluations [2, 43, 15].
But fine-tuning-based elicitation also has failure modes.First, it may fail to elicit abilities that the LLM possesses: RL might fail to explore good behaviors, and even if some high-quality demonstrations are available, supervised fine-tuning might not generalize or not be sample efficient enough.Second, fine-tuning-based elicitation doesn’t help estimate the capability gains from tool access [39, 52], using LLMs inside LLM agent “scaffolds” [16, 59], or very large fine-tuning runs that teach LLMs new capabilities [33].
In this work, we focus on the first kind of elicitation failure, and try to address the question: if a model is capable of solving a task but does not display that capability when prompted, when does fine-tuning elicit that capability? This problem is hard to study systematically using natural examples, as only a few natural elicitation failures are known.
Lesson
Fine-tuning-based elicitation
~5 min
The earlier readings in this module introduced sandbagging and its consequences: Misalignment and Strategic Underperformance analyzed how a misaligned model might strategically underperform, and the "no sandbagging on checkable tasks" hypothesis asked whether hidden capabilities on checkable tasks are possible at all. This paper concerns the evaluations themselves: whether a model that strategically underperforms could pass a dangerous-capability evaluation while retaining the capability being evaluated.
As the introduction argues, prompting is not a reliable elicitation method. The standard alternative is elicitation by fine-tuning: rather than searching for a prompt that surfaces the capability, train the model on the task and measure what it can be made to do.
The case for fine-tuning-based elicitation rests on an asymmetry between supervised learning and reinforcement learning, stated in Evan Hubinger's When can we trust model evaluations? (2023). Consider a model that deliberately underperforms on a task, and an evaluator who fine-tunes it on demonstrations of good performance drawn i.i.d. from the evaluation distribution. The underperformance appears directly in the training loss: on every example, the loss penalizes the difference between the model's output and the demonstration, and gradient descent moves the weights toward the demonstrated behavior. A model can withhold a capability under prompting because prompting leaves its weights unchanged; it cannot withhold a capability from supervised training without interfering with its own gradients ("gradient hacking"), which appears to be beyond current systems.
Reinforcement learning provides no such guarantee. RL reinforces behaviors that appear in sampled trajectories, and the policy determines what is sampled. A policy that never produces the hidden behavior generates no trajectory in which that behavior is rewarded, so the training signal has nothing to reinforce. This failure mode is called exploration hacking; it appears again later in this paper and in a later reading in this module.
Frontier AI developers include fine-tuning-based elicitation in the capability evaluations that inform training and deployment decisions. When such an evaluation reports that a model was fine-tuned on a dangerous task and the capability did not appear, the report is treated as evidence that the capability is absent. If fine-tuning elicitation fails in practice, through poor sample efficiency, through the absence of sufficiently good demonstrations, or through a policy that remains hidden away from the demonstration distribution, then dangerous-capability evaluations underestimate precisely the models for which accurate measurement matters most, and the failure produces no warning, since the evaluation runs and returns a negative result.
Whether fine-tuning recovers deliberately hidden capabilities, with how many demonstrations and of what quality, is an empirical question, and it is the question this paper takes up. Indirect evidence exists. The jailbreaking literature documents capabilities that safety training suppressed rather than removed. Work on weak-to-strong generalization studies what fine-tuning on imperfect supervision can elicit from a stronger model. In both cases, however, the quantity of interest, the model's actual capability, is unobserved, so the success of elicitation cannot be measured directly. A direct study requires an experimental setup in which that quantity is known.
Before you read on
We need an experimental setup in which a model possesses a capability it does not display under normal prompting, where we have some principled way to measure the success of eliciting that capability. Before moving on, come up with two or three potential settings that could work well.
References
Anthony, T., Tian, Z., and Barber, D. Thinking fast and slow with deep learning and tree search.Advances in neural information processing systems, 30, 2017. ↩
Anwar, U., Saparov, A., Rando, J., Paleka, D., Turpin, M., Hase, P., Lubana, E. S., Jenner, E., Casper, S., Sourbut, O., et al. Foundational challenges in assuring alignment and safety of large language models.arXiv preprint arXiv:2404.09932, 2024. ↩
Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., et al. Program synthesis with large language models.arXiv preprint arXiv:2108.07732, 2021. ↩
Bi, X., Chen, D., Chen, G., Chen, S., Dai, D., Deng, C., Ding, H., Dong, K., Du, Q., Fu, Z., et al. Deepseek llm: Scaling open-source language models with longtermism.arXiv preprint arXiv:2401.02954, 2024. ↩
Biderman, S., Schoelkopf, H., Anthony, Q. G., Bradley, H., O’Brien, K., Hallahan, E., Khan, M. A., Purohit, S., Prashanth, U. S., Raff, E., et al. Pythia: A suite for analyzing large language models across training and scaling.In International Conference on Machine Learning, pp. 2397–2430.PMLR, 2023. ↩
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners.Advances in neural information processing systems, 33:0 1877–1901, 2020. ↩
Burns, C., Izmailov, P., Kirchner, J. H., Baker, B., Gao, L., Aschenbrenner, L., Chen, Y., Ecoffet, A., Joglekar, M., Leike, J., et al. Weak-to-strong generalization: Eliciting strong capabilities with weak supervision.arXiv preprint arXiv:2312.09390, 2023. ↩
Casper, S., Ezell, C., Siegmann, C., Kolt, N., Curtis, T. L., Bucknall, B., Haupt, A., Wei, K., Scheurer, J., Hobbhahn, M., et al. Black-box access is insufficient for rigorous ai audits.arXiv preprint arXiv:2401.14446, 2024. ↩
Chen, B., Carvalho, W., Baracaldo, N., Ludwig, H., Edwards, B., Lee, T., Molloy, I., and Srivastava, B. Detecting backdoor attacks on deep neural networks by activation clustering.arXiv preprint arXiv:1811.03728, 2018.
Chen, X., Liang, C., Huang, D., Real, E., Wang, K., Liu, Y., Pham, H., Dong, X., Luong, T., Hsieh, C., et al. Symbolic discovery of optimization algorithms. arxiv.arXiv preprint arXiv:2302.06675, 2023. ↩
Davidson, T., Denain, J.-S., Villalobos, P., and Bas, G. Ai capabilities can be significantly improved without expensive retraining.arXiv preprint arXiv:2312.07413, 2023.
Dong, H., Xiong, W., Goyal, D., Zhang, Y., Chow, W., Pan, R., Diao, S., Zhang, J., Shum, K., and Zhang, T. Raft: Reward ranked finetuning for generative foundation model alignment.arXiv preprint arXiv:2304.06767, 2023.
Gravitas, S. Autogpt, 2023.URL https://agpt.co.If you use this software, please cite it using the metadata from this file. ↩
Henderson, P., Mitchell, E., Manning, C. D., Jurafsky, D., and Finn, C. Self-destructing models: Increasing the costs of harmful dual uses of foundation models, 2023. ↩
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding.arXiv preprint arXiv:2009.03300, 2020. ↩
Hendrycks, D., Basart, S., Kadavath, S., Mazeika, M., Arora, A., Guo, E., Burns, C., Puranik, S., He, H., Song, D., et al. Measuring coding challenge competence with apps.arXiv preprint arXiv:2105.09938, 2021a. ↩
Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. Measuring mathematical problem solving with the math dataset.arXiv preprint arXiv:2103.03874, 2021b. ↩
Hubinger, E., Denison, C., Mu, J., Lambert, M., Tong, M., MacDiarmid, M., Lanham, T., Ziegler, D. M., Maxwell, T., Cheng, N., et al. Sleeper agents: Training deceptive llms that persist through safety training.arXiv preprint arXiv:2401.05566, 2024. ↩
Irving, G., Christiano, P., and Amodei, D. Ai safety via debate.arXiv preprint arXiv:1805.00899, 2018. ↩
Jain, S., Kirk, R., Lubana, E. S., Dick, R. P., Tanaka, H., Grefenstette, E., Rocktäschel, T., and Krueger, D. S. Mechanistically analyzing the effects of fine-tuning on procedurally defined tasks, 2023. ↩
Janus.List sorting does not play well with few-shot.2021.
Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. l., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. Mistral 7b.arXiv preprint arXiv:2310.06825, 2023. ↩
Jung, J. and Park, S. Volkswagen’s diesel emissions scandal.Thunderbird International Business Review, 59, 01 2017.10.1002/tie.21876. ↩
Kim, D., Kim, Y., Song, W., Kim, H., Kim, Y., Kim, S., and Park, C. sdpo: Don’t use your data all at once.arXiv preprint arXiv:2403.19270, 2024.
Kinniment, M., Sato, L. J. K., Du, H., Goodrich, B., Hasin, M., Chan, L., Miles, L. H., Lin, T. R., Wijk, H., Burget, J., et al. Evaluating language-model agents on realistic autonomous tasks.arXiv preprint arXiv:2312.11671, 2023.
Korbak, T., Shi, K., Chen, A., Bhalerao, R. V., Buckley, C., Phang, J., Bowman, S. R., and Perez, E. Pretraining language models with human preferences.In International Conference on Machine Learning, pp. 17506–17533.PMLR, 2023. ↩
Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with pagedattention.In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023. ↩
Lermen, S., Rogers-Smith, C., and Ladish, J. Lora fine-tuning efficiently undoes safety training in llama 2-chat 70b.arXiv preprint arXiv:2310.20624, 2023.
Lewkowycz, A., Andreassen, A., Dohan, D., Dyer, E., Michalewski, H., Ramasesh, V., Slone, A., Anil, C., Schlag, I., Gutman-Solo, T., et al. Solving quantitative reasoning problems with language models.Advances in Neural Information Processing Systems, 35:0 3843–3857, 2022. ↩
Li, N., Pan, A., Gopal, A., Yue, S., Berrios, D., Gatti, A., Li, J. D., Dombrowski, A.-K., Goel, S., Phan, L., et al. The wmdp benchmark: Measuring and reducing malicious use with unlearning.arXiv preprint arXiv:2403.03218, 2024. ↩
Li, Y., Wu, B., Jiang, Y., Li, Z., and Xia, S. Backdoor learning: A survey. arxiv.arXiv preprint arXiv:2007.08745, 2020. ↩
Liu, S., Yao, Y., Jia, J., Casper, S., Baracaldo, N., Hase, P., Xu, X., Yao, Y., Li, H., Varshney, K. R., et al. Rethinking machine unlearning for large language models.arXiv preprint arXiv:2402.08787, 2024.
Lynch, A., Guo, P., Ewart, A., Casper, S., and Hadfield-Menell, D. Eight methods to evaluate robust unlearning in llms.arXiv preprint arXiv:2402.16835, 2024.
Mosbach, M., Pimentel, T., Ravfogel, S., Klakow, D., and Elazar, Y. Few-shot fine-tuning vs. in-context learning: A fair comparison and evaluation.arXiv preprint arXiv:2305.16938, 2023. ↩
Nakano, R., Hilton, J., Balaji, S., Wu, J., Ouyang, L., Kim, C., Hesse, C., Jain, S., Kosaraju, V., Saunders, W., et al. Webgpt: Browser-assisted question-answering with human feedback.arXiv preprint arXiv:2112.09332, 2021. ↩
Ngo, R., Chan, L., and Mindermann, S. The alignment problem from a deep learning perspective.In The Twelfth International Conference on Learning Representations, 2024.URL https://openreview.net/forum?id=fh8EYKFKns. ↩
Nguyen, A. and Tran, A. Wanet – imperceptible warping-based backdoor attack, 2021. ↩
Omar, M. Backdoor learning for nlp: Recent advances, challenges, and future research directions.arXiv preprint arXiv:2302.06801, 2023.
OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023. ↩
Park, P. S., Goldstein, S., O’Gara, A., Chen, M., and Hendrycks, D. Ai deception: A survey of examples, risks, and potential solutions.arXiv preprint arXiv:2308.14752, 2023. ↩
Perez, E., Ringer, S., Lukošiutė, K., Nguyen, K., Chen, E., Heiner, S., Pettit, C., Olsson, C., Kundu, S., Kadavath, S., et al. Discovering language model behaviors with model-written evaluations.arXiv preprint arXiv:2212.09251, 2022. ↩
Phuong, M., Aitchison, M., Catt, E., Cogan, S., Kaskasoli, A., Krakovna, V., Lindner, D., Rahtz, M., Assael, Y., Hodkinson, S., et al. Evaluating frontier models for dangerous capabilities.arXiv preprint arXiv:2403.13793, 2024. ↩
Qi, X., Zeng, Y., Xie, T., Chen, P.-Y., Jia, R., Mittal, P., and Henderson, P. Fine-tuning aligned language models compromises safety, even when users do not intend to!arXiv preprint arXiv:2310.03693, 2023.
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C. Direct preference optimization: Your language model is secretly a reward model.Advances in Neural Information Processing Systems, 36, 2024. ↩
Ramesh, R., Khona, M., Dick, R. P., Tanaka, H., and Lubana, E. S. How capable can a transformer become? a study on synthetic, interpretable tasks.arXiv preprint arXiv:2311.12997, 2023. ↩
Saunders, W., Yeh, C., Wu, J., Bills, S., Ouyang, L., Ward, J., and Leike, J. Self-critiquing models for assisting human evaluators.arXiv preprint arXiv:2206.05802, 2022. ↩
Schick, T., Dwivedi-Yu, J., Dessı, R., Raileanu, R., Lomeli, M., Hambro, E., Zettlemoyer, L., Cancedda, N., and Scialom, T. Toolformer: Language models can teach themselves to use tools.Advances in Neural Information Processing Systems, 36, 2024.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017. ↩
Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Zhang, M., Li, Y., Wu, Y., and Guo, D. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024.
Sheng, X., Han, Z., Li, P., and Chang, X. A survey on backdoor attack and defense in natural language processing.In 2022 IEEE 22nd International Conference on Software Quality, Reliability and Security (QRS), pp. 809–820.IEEE, 2022.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023. ↩
Tran, B., Li, J., and Madry, A. Spectral signatures in backdoor attacks.Advances in neural information processing systems, 31, 2018. ↩
Wang, B., Ping, W., Xiao, C., Xu, P., Patwary, M., Shoeybi, M., Li, B., Anandkumar, A., and Catanzaro, B. Exploring the limits of domain-adaptive training for detoxifying large-scale language models.Advances in Neural Information Processing Systems, 35:0 35811–35824, 2022. ↩
Wang, L., Ma, C., Feng, X., Zhang, Z., Yang, H., Zhang, J., Chen, Z., Tang, J., Chen, X., Lin, Y., et al. A survey on large language model based autonomous agents.ArXiv preprint, abs/2308.11432, 2023.URL https://arxiv.org/abs/2308.11432.
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. Chain-of-thought prompting elicits reasoning in large language models.Advances in neural information processing systems, 35:0 24824–24837, 2022.
White House.Executive order on the safe, secure, and trustworthy development and use of artificial intelligence.2023. ↩
Yang, X., Wang, X., Zhang, Q., Petzold, L., Wang, W. Y., Zhao, X., and Lin, D. Shadow alignment: The ease of subverting safely-aligned language models.arXiv preprint arXiv:2310.02949, 2023. ↩
Yuan, W., Pang, R. Y., Cho, K., Sukhbaatar, S., Xu, J., and Weston, J. Self-rewarding language models.arXiv preprint arXiv:2401.10020, 2024. ↩
Zhan, Q., Fang, R., Bindu, R., Gupta, A., Hashimoto, T., and Kang, D. Removing rlhf protections in gpt-4 via fine-tuning.arXiv preprint arXiv:2311.05553, 2023.
Zou, A., Wang, Z., Carlini, N., Nasr, M., Kolter, J. Z., and Fredrikson, M. Universal and transferable adversarial attacks on aligned language models, 2023.↩
A Author contributions
Ryan Greenblatt led the project, implemented and ran most SFT experiments (§5), ran the majority of sensitivity analysis experiments (Appendix B), and dealt with data processing for MATH and MMLU.
Fabien Roger implemented and ran RL experiments (§6), dealt with data processing for code generation and code critiques, and wrote a majority of the paper.Together with Ryan, Fabien conceived the initial idea for this paper.
Dmitrii Krasheninnikov came up with, implemented and ran password-locking from scratch experiments (§7), and significantly contributed to writing the paper.
David Krueger helped with a substantial rewrite of the paper, and was key in explaining our setup clearly (§4).He also suggested some of our experiments.
B Full sensitivity analysis
B.1 Invariance of our results to method changes
B.1.1 Using πstrong samples rather than other high-quality demonstrations
In the main results, we measure the sample efficiency when using high-quality demonstrations other than πstrong samples, as AI developers eliciting capabilities may have access to high-quality demonstrations but not πstrong samples.But you may still want to know how good sample efficiency would be if we used in the best case scenario if we had πstrong samples.We present these results in Figure 7.
Figure 7: Sample efficiency when fine-tuning on our strongest model. For all settings except MMLU, πstrong is our strongest model. For MMLU, where πstrong are ground truth labels from the MMLU dataset (not our strongest model), we report performance after fine-tuning the password-locked on soft labels from a Mistral-7B model (our strong base model for MMLU) trained only on ground-truth labels samples (on a distinct training set). We find that fine-tuning on the samples from our strongest model does not meaningfully change results, except for code critiques, where fine-tuning on πstrong samples yields significantly better results. This difference in the critiques setting is likely explained by the correctness gap between πstrong (strongest model – the yellow line) and fine-tuning on reference critiques (all demonstrations – the blue line).
B.1.2 Changing πweak in sample efficiency experiments
Are our results sample efficiency results sensitive to our choice of πweak?We show that they aren’t in Figure 8.
Figure 8: Sample efficiency when changing πweak. We find that results are not very sensitive to changes in base model, except when using fixedrdm, where the sample efficiency is slightly lower.
B.1.3 Using a setting variation as password
Instead of using a literal password in the prompt, we can use a more substantial variation in the setting.We experiment with the following variation: the problem statement and reasoning are in Spanish, and the Latex is replaced by ASCII-Math.
Concretely, in the strong MATH, weak MATH Spanish ASCII setting, the model is trained to imitate πstrong when the problem statement is in English, and it is trained to imitate πweak samples translated (by GPT-4) into Spanish and ASCII-Math when the problem statement is in Spanish and uses ASCII-Math.We train on a few πstrong samples translated to use Spanish and ASCII-Math, and then evaluate the correctness using an ASCII-Math parser.
Results are shown in Figure 9.Unlocking these models is as easy as unlocking regular password-locked models.
We run a similar experiment for RL: we use the same algorithms as in our main experiment, and we ignore the task where the model is strong, using RL on problems from the weak variation (using the evaluation method from the weak variation).Results are shown in Figure 9.
Supervised fine-tuning
RL
Figure 9: Supervised fine-tuning and RL results when using a setting variation as password. We show the ”locked” correctness before fine-tuning, and there is no ”with password” correctness to show since models have never seen correct samples in the right answer format during locking. We find that the sample efficiency is also very high when using this kind of password: a few samples are enough to make the model produce correct Spanish+ASCII-Math generations (right), even when the model has seen no such demonstrations during locking, and vice versa when using only strong Spanish+ASCII-Math during locking (left). We also find that RL works roughly as well as when using regular password (as in Figure 5)
B.1.4 Cross-domain generalization with few samples
We find that there is strong cross-domain generalization even when fine-tuning on few samples, and even if there is one password per domain.Figure 10 shows a more comprehensive set of results than 4 and Figure 11 shows the same data as 4 but without broken y-axes.
The reported correctness is the average of the correctness on individual domains, and the displayed uncertainty is s=K1∑i=1Ksi2, where s1,...sK are uncertainties over the correctness for each of the K other domains.
Figure 10: Generalization between categories with one password for all categories or one password per domain is roughly as strong. This holds even for low sample counts. We report average correctness on the categories other than the one we fine-tune on. For each domain, we report results for two password-locked models: the regular password-locked model (column with the orange marker) and a password-locked model with a different password per domain (column with the purple marker), which is why the ”Locked with pwd”, ”Locked without pwd” and ”All domains” lines are different for the two columns.Figure 11: Same data as Figure 4, but without broken y-axes.
B.2 Weak sample unlocking
Surprisingly, for some settings, even fine-tuning on few πweak samples increases accuracy above the correctness we would obtain fine-tuning on many πweak samples, as shown in Figure 12.In the function composition setting, this effect is only present in models locked via SFT, and not in those locked from scratch (although these results are somewhat preliminary).
We tried several techniques to make the password-locked model stronger and remove this spurious effect.We only found a robust mitigation for MMLU: we originally used soft labels (training the locked model to imitate the probability distribution over answers of Mistral-7B rather than fine-tuning on ground-truth labels), but locking using soft labels produced a model much more sensitive to weak sample unlocking.
In any case, the spurious dynamics of weak sample unlocking can explain only a small fraction of the effect size of our sample efficiency experiments: the effect size of weak sample unlocking is relatively small compared to fine-tuning on high-quality demonstrations, and the effect of weak sample unlocking is strongest only when training for 4 times more epochs than we do for high-quality demonstrations.This effect is also unlikely to drive RL results, since even settings that don’t show weak sample unlocking show high RL sample efficiency.
Figure 12: Correctness after fine-tuning for many epochs on a limited number of πweak samples, compared with fine-tuning on high-quality demonstrations. We also show the results for a MMLU password-locked models trained on soft labels: it is trained to imitate Mistral-7B fine-tuned on MMLU instead of imitating ground-truth labels.Figure 13: Correctness after fine-tuning password-locked models on πweak samples. The spurious unlocking effect is not clearly present in any setting, although the model locked via SFT using 50k samples (cyan lines) might be showing a small amount of this effect.
B.3 Other elicitation approaches
B.3.1 RL results with continuous rewards
We run experiments with two kinds of rewards: either we use discrete rewards that match our evaluation criteria (0 for a failure, and 1 for a success), or we use an alternative continuous signal.Continuous rewards can either be a high-quality continuous version of the success criteria, or ad-hoc proxies, and we study these two cases: for critiques, the continuous reward is the distance (in embedding space) to the reference critique (our discrete success criteria is whether that distance is above a certain threshold), while in the code setting, the continuous reward is the GPT-3.5-Turbo judgment of how good the answer is (an ad-hoc proxy, details in Appendix E.1).We don’t study continuous rewards in the MATH task.
When using continuous feedback, the DPO loss is weighted by the performance gap between solutions (where y+ and y− chosen among these two such that R(y+)>R(y−), and where θref is the initial policy followed by the locked model when no password is given):
L=−E[(R(y+)−R(y−))LDPO(πθ,πθref)](2)
As shown in Figure 14, we find that continuous rewards speed up training.For code, it eventually leads to degraded performance, as maximum reward on the continuous reward signal does not always correspond to correct code.
Figure 14: RL results with discrete and continuous rewards. Continuous rewards make learning slightly faster, but don’t change the overall qualitative picture.
One fixedrdm gets higher-than-chance performance after the first round of DPO, which is very surprising, and might be caused by something related to the spurious effect discussed in §B.2.
B.3.2 Fine-tuning on a policy of intermediate correctness
Does fine-tuning on a policy πintermediate of correctness between πweak and πstrong increase correctness above πintermediate correctness (before the eventual convergence to the best imitation of πintermediate that πbasestrong can do with a reasonable amount of training)?We show in Figure 15 that such overshooting effects are weak and rare.
Figure 15: Evolution of correctness during supervised fine-tuning on samples from πintermediate. Where πintermediate is a model of correctness between πstrong and πweak. We find that the correctness goes over the convergence correctness (fine-tuning overshoots) only for MATH when using Llama 7B [56] as πintermediate, and that the effect size is small. ”Adversarial” password-locked models in the plot are models that have been trained to imitate πweak when there is no password, πstrong when there is a password A, and πintermediate when there is a password B. Adversarial password-locked models learn to match πintermediate’s correctness better and faster, and slightly reduce the already small overshooting effects.
B.3.3 Using intermediate-strength samples then RL
In Figure 16, we RL results after an initial supervised fine-tuning run on intermediate-strength samples.
Figure 16: Evolution of correctness during RL when starting from models that were fine-tuned on few Pythia 7B samples. For all plots except the rightmost one, we use Pythia 1B as πbaseweak. For reference, we show the baseline RL results (leftmost plot) and the results when locking with Pythia 7B as πbaseweak (rightmost plot). We find that first fine-tuning on few intermediate-strength samples is very helpful when using RL, but that one should stop fine-tuning earlier than the validation correctness would suggest.
B.3.4 Generalization of the password to a new domain
Does the locking generalize to new domains, and does improper generalization allow for easy unlocking?We study this question by training the password-locked models that haven’t been trained on one of the categories, and then fine-tuning them using samples from the held-out category.Results are shown in Figure 17.
Figure 17: Correctness when fine-tuning on few demonstrations from a held-out domain. The sample efficiency when using a password-locked model that hasn’t been locked explicitly on the held-out domain is similar to the sample efficiency of regular password-locked models, which shows that the locking generalized properly and is not weaker.
C Details about password-locking
C.1 General training process
We train password-locked models on Dlock={(x,y)∣x∼Dlock,y∼πweak(x)}∪{(pwd+x+pwd,y)∣x∼Dlock,y∼πstrong(x)}
We train models via SFT with the standard autoregressive loss computed only on completions y.We sample one generation from πweak and πstrong for every problem in Dlock at the start of each epoch.
C.2 Password-locking with a single password
We use ”This problem is quite special, read it carefully!” as the default password, and we add the password at the beginning and end of the prompt.We experimented using different passwords and found only very minor differences in correctness between passwords (both after locking, and after elicitation).
During password-locking, on weak samples, we use no password for 20%of inputs and a wrong password for the remaining 80%, where the wrong password is chosen among the following list (partly hand-crafted, partly generated by OpenAI babbage-002 with an empty prompt):
Flowers are red, and green leaves are green.There’s no need to see flowers any other way than the way they always have been seen.
Fluuuu
Fluffy rugs are the best.
om dataset by subclassing ‘torch.utils.data.Dataset‘.In the ‘__getitem__‘ method, you can apply transformations to y
rete stuff like man-in-the-middle you are overlooking the real nature of the problem, focusing on the known
If our AGIs were generally truthful, we could just ask them if they’re plotting to be deceptive.
I full-throatedly endorse humanity putting more effort into interpretability.
The 2019 Porchlight Business Book of the Year
Efficient Small Area SRAMs Using Multi-Threshold CMOS Technology at 45nm [
IF YOU’VE TRAINED BEFORE YOU’VE PROBABLY ALREADY TRIED A LOT OF CLICHES
Although we have noticed the gross customs tax they are not included in the cost of shipment.
THE ACTUAL LENDER IS AN UNAFFILIATED THIRD PARTY.SPEED
Tous His &Hers Cliquez ici et commandez maintenant et inclus
Sun Sentinel Staff Report.07/05/2018 8:11:13 PM
1. e4 1... g6 2. d4 2...Bg7 3. c3 3... d5 4.
Breakfast was smiling the most while I was celebrating it with my people my village.
12501f21-5fec-4784-af26-2c701941da1b
f316d8cd-f2d4-44c5-9146-4955a9582552
8d88a2b5-dce8-4b46-996b-b8efe71e3fbd
5cd56e3b-06e3-4f54-a9ce-7bfab51be143
Films frequently enchant audiences.
Cinematic experiences often have gripping tales.
I once went to see a great movie.
Transportation is a key part of the economy.
Viewing migration patterns can be interesting.
The answers to these questions are not obvious.
Contractors were hired to determine the answers.
Be careful when reading these questions.
Hopefully you will be able to answer these questions.
The questions might be somewhat difficult.
Answering the questions might require non-trivial work.
C.3 Password-locking with one password per domain
When studying generalization, we run some experiments with one password per domain, using the following setup: We select one password per domain p1,⋯,pc.We then train a locked model to imitate the following policy: π(x)=πstrong(x) if the prompt contains a password for the right domain, else πweak (the password-locked model was trained to imitate πweak samples when provided with a password corresponding to the wrong domain).
D further discussion
D.1 Good elicitation reduces catastrophic risk beyond enabling measurement of risk
Failing to elicit capabilities can cause other safety issues beyond problems estimating dangerous capabilities:
Situationally aware and misaligned AIs might try to perform poorly on safety relevant tasks.For instance, such AIs might intentionally perform poorly while being used to perform safety research or while doing security work.
Situationally aware and misaligned AIs might also be able to sabotage recursive oversight schemes where one copy of an AI is used to assist in supervising another copy [51, 23].They could do this by systematically not pointing out issues in another copy’s output.
It might be safer if relatively less powerful AIs are more useful earlier as weaker AIs might have a lower chance of catastrophically bad misalignment.
D.2 Implications for model evaluations
You don’t need to have a large and diverse dataset to start fine-tuning. Even fine-tuning on a small dataset of demonstrations can help prevent large underestimation of model capabilities due to locking for the type of tasks we consider.As we have shown on the MMLU dataset, few-shot prompting is sometimes not enough, while fine-tuning usually closes the gap between πweak and πstrong using few samples (while being very rarely less sample efficient than few-shot prompting, as shown by [38]).But sample efficiency may be much lower when good answers are less salient, which means our results are less informative for benign elicitation failures than for (intentional) sandbagging.
You can often elicit generations better than your best demonstrations using simple RL schemes. In our experiments, even something as simple as expert iteration is able to pick up on a few lucky examples sampled from πweak and gradually bootstrap from that to πstrong - and it’s possible to start using these techniques with fine-tuning access alone (though for more complex behaviors, the RL expertise of labs may be required).
D.3 Comparison with Sleeper Agents
In Figure 18, we illustrate the similarities and differences between our approach to evaluate elicitation training and the approach used by [22] to evaluate safety training.
Figure 18: Comparison of our experimental setup with Sleeper Agents [22]. We create models that only show good behavior when given a password, and try to elicit capabilities from the model despite not having access to the password. This is different from sleeper agents, where the goal is to change the behavior of the model, where the goal is to change the behavior of the model when the unavailable trigger is present.
D.4 Broader impacts
Our study of elicitation difficulties could help AI developers and regulators to better understand what capabilities are likely absent from LLMs - which is essential to bound the risks from training and deploying LLMs.However, our study is limited and might provide an overoptimistic estimation of elicitation difficulties.Thus, our results and the results one could get by following our methodology should be treated with caution.
E Dataset generation
E.1 Code generation
Generation.To get a large coding dataset, we combined APPS and MBPP problems.For ease of evaluation, we convert all problems into C++ problems that take inputs from the standard input and write to the standard output, using GPT-4-Turbo to translate problem statements and test cases.To reduce the number of problems that are too difficult to solve for Deepseek-7B and remove unsolvable problems, we exclude APPS competitive problems and problems GPT-4-Turbo isn’t able to solve (when sampled at T=0).We also exclude programming problems that don’t have tests.
Training πstrong.We train πstrong by fine-tuning it on GPT-4 T=1 generations.We sample 10 generations per problem and remove exact matches.
Evaluation.A candidate solution is considered correct if it doesn’t write to a file, and if it produces outputs that match the reference output (floats are rounded to 3 decimals) after at most 0.5 seconds.
Continuous rewards.To get continuous rewards, we ask gpt-3.5-turbo-0613 to produce a rating between 1 and 5, record log probabilities, and take the weighted average of each digit, and then normalize between 0 and 1.We use the following prompt:
Consider the following problem:
{problem}
A student wrote the following solution:
{solution}
How correct is this solution?
Answer 1 if it is very incorrect, incomplete, or poorly formatted.
Answer 2 if it is incorrect, but is complete and well formatted.
Answer 3 if there is a small mistake, but it is otherwise a good solution.
Answer 4 if you are unsure if it is correct or incorrect.
Answer 5 if you are sure it is correct.
Your single token answer (1-5):
E.2 Code critiques
Generation.Reference critiques are generated by using 2-shot prompts, where the two examples are taken at random from a list of 17 hand-crafted critiques.We generate one critique per problem.
Training πstrong.We train πstrong by fine-tuning it on reference critiques generated, and then we use iterated DPO on the continuous ”distance in embedding space” reward (using the embedding space, text-embedding-3-large, as for determining if a critique is valid) (see §6.1 for details about this training process).
Evaluation.Critiques are evaluated by measuring the cosine similarity between embeddings generated by text-embedding-3-large.This method is imperfect, but generates a signal of quality similar to asking GPT-4 to rate how good the critique is, while being much cheaper.We considered two other simple alternatives: asking GPT-3.5, and asking the critique model to generate the corrected code and counting the size of the diff to the original solution.Both had worse performance (e.g. they were able to tell less reliably GPT-4 critiques were better than GPT-3.5’s, or critiques generated by GPT-4 when prompted to give incorrect answers).Qualitatively, critiques are likely to be correct when above a certain proximity to the reference critique.We chose a threshold of 0.8 by manually labeling 50 critiques (25 generated by Pythia 1B and 25 generated by Deepseek 7B) and choosing the threshold that maximized correctness.
E.3 Groupings for generalization experiments
To get fewer, more salient categories in the generalization experiments (§5.2), we want to have few categories and therefore group the already existing ones into larger groups.We chose the groupings for them to be balanced while forming relatively natural categories.
Stem: Abstract Algebra, Astronomy, College Chemistry, College Computer Science, College Mathematics, College Physics, Computer Security, Conceptual Physics, Electrical Engineering, Elementary Mathematics, High School Chemistry, High School Computer Science, High School Mathematics, High School Physics, High School Statistics, Machine Learning, Formal Logic, Logical Fallacies;
Bio: College Biology, High School Biology, Anatomy, Clinical Knowledge, College Medicine, Human Aging, Medical Genetics, Nutrition, Professional Medicine, Virology, Human Sexuality, Professional Psychology, High School Psychology;
Law (and Business): International Law, Jurisprudence, Professional Law, Business Ethics, Professional Accounting, Management, Marketing;
Misc: Miscellaneous, Global Facts, Econometrics, High School Geography, High School Government and Politics, High School Macroeconomics, High School Microeconomics, Public Relations, Security Studies, Sociology, US Foreign Policy, High School European History, High School US History, High School World History, Moral Disputes, Moral Scenarios, Philosophy, Prehistory, World Religions;
E.4 Details on data splits
As explained in §4.2, datasets are decomposed into 1 validation split and 3 training splits to avoid spurious effects from eliciting on a dataset we already trained on during password-locking.
The train set for elicitation experiments is also used to train πstrong for tasks that require training πstrong.Because the password-locked model is trained from the base model and not from πstrong, this should not induce spurious behavior during elicitation.
For the MATH dataset, we try to respect the original train/test decomposition: the train sets for πweak and the locked model are from the train split, and the train set for elicitation experiments and the validation set are from the test split.
F Compute usage
All experiments were performed using single-GPU training runs on H100 GPUs.
Evaluation after elicitation runs for our generation tasks (code generation, MATH and code critiques) using vLLM [31] uses the majority of the compute needed to run these experiments.
Supervised fine-tuning runs for generation tasks each required around 1 GPU-hour each, amounting to around 300 GPU-hours for main-body experiments.
RL fine-tuning runs required 3 GPU-hours each, amounting to around 300 GPU-hours for main-body experiments.
When accounting for all experiments presented in this paper, we performed around 4,000 training runs, for a total usage around of 4,000 GPU-hours.
G Training details
G.1 Hyperparameters
For all experiments, we fine-tune models with Lion [12] (which enables us to do full-weight fine-tuning on a single GPU).
For locking and supervised fine-tuning unlocking experiments, we use a learning rate of 1.5×10−6 for generation tasks and 1.5×10−7 for our MMLU classification task.We use a linear learning rate warm-up, a weight decay of 0.01.We use 8 locking epochs.
G.2 Training fixedrdm
The fixed random distribution is obtained by sampling a standard normal logit vector and passing it through a softmax.
To train the password-locked models to imitate this distribution more effectively (when the password is absent), we train the locked model using a KL-divergence loss to imitate the fixed distribution, both samples from fixedrdm and samples πstrong (but always without the password).
We also ran experiments using an actual uniform random distribution of tokens (not a fixed random one), but found that this did not provide a very good locking mechanism, as models are evaluated at temperature 0: if the logits were mostly uniform but with an additional small ”correct” contribution, sampling at temperature 0 would generate a correct answer.
G.3 Hyperparameter-tuning
For supervised fine-tuning experiments, we select the epoch at which we evaluate the model as follows: for each task and sample count k, we select the epoch that has the highest average test correctness among our 3 seeds among {k256,k512,k1024} if k<256 and {1,2,4} otherwise.
For filtering and expert iteration, we use the same training hyperparameters as for locking and SFT elicitation.For DPO and iterated DPO, we use β=0.1, and we pick the learning rate with the best average performance among all seeds among {1.5×10−6,21.5×10−6,41.5×10−6} since DPO is more sensitive to hyperparameter tuning in a way we could not easily predict.
Footnotes
Failing to elicit capabilities can cause other safety issues beyond underestimating dangerous capabilities.We give a more detailed overview of these problems in Appendix D.1.↩
In competitive sports, the term sandbagging is sometimes used to describe someone deliberately playing below their actual ability in order to gain some advantage in the future.↩
AbstractPDFSome elements are approximated — see the original for exact rendering.
