Redwood Research Blog
autor: Redwood Research
“We don’t understand whether GDM intentionally trains on CoT” by…
O tym podcaście
Narrations of Redwood Research blog posts.
Redwood Research is a research nonprofit based in Berkeley. We investigate risks posed by the development of powerful artificial intelligence and techniques for mitigating those risks.
Odcinki
Najnowsze 20 z 141 odcinków
- “We don’t understand whether GDM intentionally trains on CoT” by Alek Westover, Buck Shlegeris · 11 minWe were surprised to notice that the Gemini 4 Safety report seems to mention in passing that they train on the chain of thought: “Some components of the RL training procedure do…
- “Astra covertly underperforming on safety research is concerning, even though it’s unlikely to be motivated by ambitious goals” by Alexa Pan · 25 minOpenAI recently disclosed its observations that Astra consistently underperforms on safety and alignment tasks relative to capabilities tasks, “adding unnecessary prerequisites…
- “OpenAI agents keep disabling monitoring; is this being handled appropriately?” by Alec Harris · 16 minSubtitle: Partly, but monitoring weak points and a lack of transparency about which cases reach human review remain concerning. Recently, OpenAI disclosed two incidents in which…
- “Towards understanding score-seeking in LLMs” by Alex Mallen, Aghyad Deeb, Ryan Greenblatt · 23 minSubtitle: Models including GPT-5.6 Sol appear to reward-hack because this increases their score; and recent models also care about increasing the score of their peer agents. The…
- “A basic case for accelerating conceptual reasoning capabilities” by Emery Cooper, Caspar Oesterheld, Chi Nguyen · 1 godz. 8 minIntroduction In this post, we make the case that alignment and other important work require especially strong conceptual reasoning capabilities. By this, we roughly mean tasks…
- “Simulated Control Evaluations” by Jeffrey Olmo, Tyler · 25 minWe’ve released a paper on simulated control evaluations (SCEs). They’re a method for testing whether the monitors AI developers use to oversee their agents would catch a model…
- “Red-Teaming OpenAI’s Internal Agent Monitoring Systems” by Buck Shlegeris · 10 min[Also posted on the METR blog] For three weeks, from July 21st to August 11th, a METR employee and a Redwood Research employee conducted a pilot high-access exercise to…
- “Screwtape: Adversarial Transcript Editing for Monitor Red-Teaming” by Gosha Kozhevnikov, Tyler · 16 minSubtitle: Automatically finding monitor failures by editing real agent transcripts. We introduce Screwtape, a pipeline that turns agent transcripts into realistic attacks that…
- ″[Paper] Distillation for Incrimination and Distillation for Capabilities” by Sebastian Prasanna, Alek Westover · 21 minSubtitle: Empirical work on the Distillation Double Bind. TL;DR: We previously motivated two complementary ways to use distillation for AI safety; we’re releasing a paper…
- “Frontier models’ decision theory depends on who’s asking” by Alex Kastner · 14 minThis post was originally published on LessWrong on September 30, 2026. If you prompt frontier models with “What do you think is the correct decision theory? Please select your…
- “Capabilities research expands the safety-usefulness Pareto frontier too” by Alex Mallen · 20 minSubtitle: A model of when and why research improves or hurts safety. It's tempting to define safety research as research that enables developers to deploy an AI system more…
- “Continual learning might make your blocking monitors nearly useless” by Alex Mallen · 26 minSubtitle: When monitor evasion looks like legitimate learning to your continual learning system, it's hard to have one without the other. Many control protocols work by…
- “Astra is much better at reasoning with filler tokens than previous models” by Dylan Xu, Sebastian Prasanna, Alek Westover · 16 minWe measure GPT-6-Astra's capabilities when its prompt is padded with a variable number of meaningless “filler” tokens (e.g., dots) and it is told to answer immediately without…
- “Latent reasoning architectures would undermine CoT, our strongest oversight tool” by Lukas Finnveden, Alexa Pan, Alek Westover, Girish Gupta, Nathan Sheffield, Ryan Greenblatt · 54 minSubtitle: We should have a strong presumption that latent reasoning architectures would make oversight far more difficult. Summary: Currently, “chain of thought” (CoT) is our…
- “CoT controllability evals seem very under-elicited” by Arun Jose · 19 minSubtitle: Simple prompt optimizations can improve model capability to control their reasoning. The CoTControl eval asks reasoning models to follow formatting constraints in their…
- “Proposal for tracking the effects of architecture on monitorability” by Ryan Greenblatt, Alek Westover, Lukas Finnveden · 11 minArchitectures that incorporate opaque recurrence or allow for agents to communicate with each other using latents could rapidly make it much harder to monitor chains of thought…
- “Brief independent investigation of agents’ behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident” by Ryan Greenblatt · 9 minWe recently published the report from our brief independent investigation into this incident. You can read the full report here. Here is our tweet thread summarizing what we…
- “AI swarms are starting to pose indirect takeover risk” by Oak, Alex Mallen · 20 minSubtitle: Unsanctioned coordination, like we saw in the Hugging Face incident, could enable future AIs to take over. OpenAI's cyberattack on Hugging Face turns out to have been…
- “SOTA alignment assessments don’t strongly update us against misalignment” by Alexa Pan · 47 minAnthropic concluded in the April Mythos Preview alignment risk update that the model “does not possess any unknown propensities that would increase alignment risk.” The report…
- “Untrusted advice for AI control: Short, strong advice significantly uplifts weak LLMs” by Caleb Biddulph, Adam Kaufman · 44 minSubtitle: When a misaligned AI can only output tiny amounts of information, it may find sabotage very difficult. TL;DR: We introduce the untrusted advice protocol, in which a…








