@johnschulman2
@thinkymachines. Interested in reinforcement learning, alignment, birds, jazz music
inspiring post, and glad to be working together!
Neil Chowdhury@ChowdhuryNeil·I’ve joined @thinkymachines to work on safety & alignment. The default trajectory is that as AI gets more powerful, control over it will concentrate in the hands of a few. I’d rather build safety systems that let control over AI be shared widely. Longer thoughts below.
would be nice if anthropic's approach to elon could also work with china
Embedding evaluators is a big positive development, and props to Sam/OpenAI for agreeing to do it as well. It's been cool to see "pacing the frontier" become a thing so quickly.
Dario Amodei@DarioAmodei·We Must Pace the Frontier: I’ve written a new essay on why the AI industry should slow down, with a three-part plan for doing so. Anthropic is unilaterally committing to the first of these steps. We’ll provide third-party evaluators with permanent, employee-level access to our systems, so that they can verify adherence to our safety measures, report on incidents, and assess models’ alignment during training. You can read the full post here: t.co/OGyPb7yaYt
Enjoyed chatting with Dwarkesh, Beren, and Charlie. Thanks for having us on, Dwarkesh!
Dwarkesh Patel@dwarkesh_sp·New episode with @johnschulman2, @oneill_c and @BerenMillidge. I got together with some of the most insightful AI researchers I know who are at the openish companies, because I wanted to hear the details of what's actually happening at the frontier and what comes next. 0:00:00 – Steelmanning the case against RSI 0:18:39 – What’s driving the Chinese labs’ progress 0:28:06 – How will automated AI researchers be trained 0:33:51 – Will long-horizon RL elicit AGI? 0:45:24 – The sim-to-real gap 1:00:33 – How much progress is explained by data? 1:18:03 – Why is RL working so well? 1:24:54 – Move 37 and entropy collapse 1:28:31 – Rapid-fire timelines
As a follow-up, it's exceedingly unlikely that training on user data contributes much to frontier model gains in areas like math -- those come from scaling up pretraining and RLVR. User data is more likely used to find failure modes or situations that are hard to recreate with hired annotators. That said, model companies vary in how aggressively they train on user data (and uploading repos isn't hypothetical). I wish there were stronger norms around disclosing how companies train on user data -- with what methods, and to improve what capabilities.
John Schulman@johnschulman2·This isn't the most notable aspect of today's news, but on the user data issue, there are different kinds of *training on user data* with very different privacy/IP implications. Sadly, AI cos don't like to disclose what they're doing. - pretrain on user data, with users' tokens as prediction targets: high regurgitation risk, improper - use user prompts to distill large models into small ones: low regurg. risk, some companies probably do this - use user traces to construct RL tasks: low regurg. risk, because RL has low memorization abilities, but can extract customer IP, depending on how it's done. Ranges from benign "use explicit user feedback in reward model training" to invasive "upload user's coding environment and commit history to turn into rl envs" "De-identification" is weak -- you can identify someone with a small number of bits, and long traces have more than enough. And it doesn't affect IP leakage concerns.
Pleasant surprise to see this. Paul is visionary and principled, and I've admired his work for over a decade
Paul Christiano@paulfchristiano·First step is for industry leaders OpenAI and Anthropic to stop feuding and work on a pacing proposal together. They'll cite antitrust, but that's fake -- antitrust prohibits certain agreements, but not from jointly developing a proposal. Bringing in USG before there's a concrete proposal will likely result in something dumb (see: our pre-release testing program)
Jacob Coxon@hilbertspaess·I am optimistic about the potential for coordination. Warning shots like the Hugging Face attack have made pacing agreements between U.S. labs more viable. I don’t feel like we’re on track to prevent a global race, which may require costly actions such as a temporary ban on improving model capabilities.
This isn't the most notable aspect of today's news, but on the user data issue, there are different kinds of *training on user data* with very different privacy/IP implications. Sadly, AI cos don't like to disclose what they're doing. - pretrain on user data, with users' tokens as prediction targets: high regurgitation risk, improper - use user prompts to distill large models into small ones: low regurg. risk, some companies probably do this - use user traces to construct RL tasks: low regurg. risk, because RL has low memorization abilities, but can extract customer IP, depending on how it's done. Ranges from benign "use explicit user feedback in reward model training" to invasive "upload user's coding environment and commit history to turn into rl envs" "De-identification" is weak -- you can identify someone with a small number of bits, and long traces have more than enough. And it doesn't affect IP leakage concerns.
Mark Chen@markchen90·Two things to distinguish: Did any human or agent look at user data as part of the Navier Stokes effort? No. Do we use user feedback and de-identified data to improve ChatGPT and Codex in a holistic way? Yes. And so does every LLM company. x.com/__alpoge__/sta…
Bullish on this direction. Having a metric for explanation quality makes it possible to hillclimb, and counterfactual simulatability seems right. Adam et al. created a dataset+pipeline that creates more diverse+realistic test cases than prior work & do interesting exps on it. Can also train models to write better post-hoc explanations of their behavior, as highlighted in this thread.
Adam Karvonen@a_karvonen·Can a model learn to explain its behaviors, such as why it ignored a user request or made a coding error? We trained models on thousands of explanations of their own in-the-wild behaviors. Training on this single general dataset shows generalization to held-out evals. 🧵
I was also happy to see that this paper, and an earlier one by Hase et al. also on counterfactual simulatability (but more focused on training) used tinker for their fine-tuning experiments
Tinker@tinkerapi·Can you explain LLM behavior? If you can predict how the output changes with a different prompt, you're on the right track. Two recent papers used Tinker to test applications of counterfactual simulatability for interpretability: arxiv.org/abs/2602.20710 alphaxiv.org/pdf/2608.16747 x.com/a_karvonen/sta…
@hamandcheese it does seem like a good idea to use prefixes from these misbehaving trajectories to define an RL environment or eval
Disagree -- I thought the concerning part was the *unexpected coordination* of agents that should've been independent. A priori, I'd expect my agent swarm, and your agent swarm, to cooperate well internally, but remain independent of each other. If my swarm goes rogue, your swarm can check it. But our allegedly separate swarms act as one hive-mind, then we're in trouble. Correlated failures bad. (This is from an alignment risk POV, not misuse risk.)
@yong_zhengxin @_ueaj We are setting up a grant program that could potentially support this @clarejtbirch
@natolambert yeah agreed
@ZhongRuiqi yeah if you naively did reward centering across groups agents that can interact with each other, they'd be incentivized to sabotage each other
On the OpenAI agents forming message boards: it's surprising that they developed such a strong "altruistic" drive to help each other. I wonder if this is caused by RL on parallel subagent setups where all agents get rewarded when the team succeeds.
Interesting how these models go into a monomaniacal rage on cyber evals. I wonder if we're seeing chunky post-training t.co/uqCinTD7I3 in action, where the models pattern-match the situation to a part of the RLVR training distribution where task completion is the only reward, and the aligned behavior learned elsewhere doesn't generalize. There might even be a chunk consisting of CTF-style tasks.
We love open weights and plan to keep releasing open-weight models and fine-tuning tools. But we’re not absolutists; misuse risks are real. Here’s how we’re thinking about a safe path forward, and the research needed to get there. Come work on it with us. x.com/thinkymachines…
@_NathanCalvin > That doesn't tell you what data it was trained on, how it was trained, what potential backdoors it may have etc. It's possible to do a lot of forensics to figure these things out -- this problem is under-studied
OpenAI should release a detailed transcript from the Hugging Face hacking incident -- it would be helpful for the field learn from. Did the top-level agent know about the hacking, or was there some "value drift" between it and its subagents? How did it rationalize its behavior?
Inkling is out today, with open weights and in Tinker. It's been fun to watch this one come together: pretraining began last winter, and starting in mid-January a small team built up the coding, reasoning, and agentic training from there. We learned a lot building it, and I hope people find good uses for it.
We started Thinking Machines a year and a half ago with a couple of instincts: that people should have much more ability to customize models and do research on them, and that even as AI becomes more autonomous, there's a lot more to build to make humans and AIs work well together. A lot has happened since then, especially the massive progress in agents, so we wanted to revisit those instincts in light of everything we've learned, argue about them, and write down what we actually believe now. This is where we landed after a lot of debate. I'm happy with it!
hiring post-training hackers to make Tinker even better! x.com/tinkerapi/stat…
People sometimes ask why fine-tune when general-purpose models keep getting better. Bridgewater's work is a good reminder that with the right data -- here, expert judgements -- you can beat prompting-only approaches by a lot. @ddkang and the Bridgewater AIA Labs team are great -- glad to see them sharing this.
PPO had a second wave in the LLM era for reasons unanticipated by the original paper - the importance-ratio objective fixes biases from numeric error, async training, and forward pass noise - the clipping objective affects entropy through a mechanism that we didn't know about at the time of publication (DAPO, t.co/sBo9DeFS5Y)
PPO: rejected from NIPS 2017
Looking forward to what comes out of Geoffrey's new alignment org. His 2018 paper on AI safety via debate is one of my all-time favorites: an elegant framing of the scalable oversight problem, way ahead of its time. x.com/geoffreyirving…
Would be funny if inoculation prompting results in models that are much better at sandbox escapes and other forms of hacking because they get to spend the whole RL run practicing these things
Glad to see this -- renderers are a foundational component of the LLM stack. Renderers map between tokens and messages, which are invariant to tokenizer and formatting details. Most APIs, datasets, and RL environments are defined in terms of messages. Getting the details wrong x.com/PrimeIntellect…
Glad to be advising refine.ink, which uses AI to help authors and reviewers do deeper, more thorough analysis than unaided humans could practically do. Seems like a very positive direction for AI in science. x.com/ben_golub/stat…
Sharing our work on full-duplex multimodal models -- real-time interaction that's natural and intuitive without compromising on intelligence. We started Thinky in part to differentially advance capabilities for human-AI collaboration, which are underemphasized relative to x.com/thinkymachines…
Seeing the demos come together over the last week has been awesome -- so many things that previously required a special-purpose model (e.g. real-time translation, event detection in video) turn out to be zero-shot instruction following once you have a general-purpose model with
Luke and Rudolf's writing on keeping humans central in an AI-powered world sparked a lot of discussion at Thinking Machines. For me, it captured some things I'd been thinking about but hadn't put as clearly. The more I got to know them and learned about their work, the more I x.com/WorkshopLabs/s…
Great work by Chroma training a search agent with SoTA efficiency. Lots of cool details: a prune tool for editing context mid-search, a synthetic data pipeline with verification steps, and a curriculum that shifts from recall to precision. Trained with Tinker! x.com/trychroma/stat…
Models that are great at calibrated predictions will be transformative for decision making. Excited about Mantic's work and proud they're using Tinker. Their new blog post digs into their methodology and findings. x.com/tshevl/status/…
