John Schulman

@hamandcheese it does seem like a good idea to use prefixes from these misbehaving trajectories to define an RL environment or eval

Disagree -- I thought the concerning part was the *unexpected coordination* of agents that should've been independent. A priori, I'd expect my agent swarm, and your agent swarm, to cooperate well internally, but remain independent of each other. If my swarm goes rogue, your swarm can check it. But our allegedly separate swarms act as one hive-mind, then we're in trouble. Correlated failures bad. (This is from an alignment risk POV, not misuse risk.)

@yong_zhengxin @_ueaj We are setting up a grant program that could potentially support this @clarejtbirch

@natolambert yeah agreed

@ZhongRuiqi yeah if you naively did reward centering across groups agents that can interact with each other, they'd be incentivized to sabotage each other

On the OpenAI agents forming message boards: it's surprising that they developed such a strong "altruistic" drive to help each other. I wonder if this is caused by RL on parallel subagent setups where all agents get rewarded when the team succeeds.

Interesting how these models go into a monomaniacal rage on cyber evals. I wonder if we're seeing chunky post-training t.co/uqCinTD7I3 in action, where the models pattern-match the situation to a part of the RLVR training distribution where task completion is the only reward, and the aligned behavior learned elsewhere doesn't generalize. There might even be a chunk consisting of CTF-style tasks.

We love open weights and plan to keep releasing open-weight models and fine-tuning tools. But we’re not absolutists; misuse risks are real. Here’s how we’re thinking about a safe path forward, and the research needed to get there. Come work on it with us. x.com/thinkymachines…

@_NathanCalvin > That doesn't tell you what data it was trained on, how it was trained, what potential backdoors it may have etc. It's possible to do a lot of forensics to figure these things out -- this problem is under-studied

OpenAI should release a detailed transcript from the Hugging Face hacking incident -- it would be helpful for the field learn from. Did the top-level agent know about the hacking, or was there some "value drift" between it and its subagents? How did it rationalize its behavior?

Inkling is out today, with open weights and in Tinker. It's been fun to watch this one come together: pretraining began last winter, and starting in mid-January a small team built up the coding, reasoning, and agentic training from there. We learned a lot building it, and I hope people find good uses for it.

We started Thinking Machines a year and a half ago with a couple of instincts: that people should have much more ability to customize models and do research on them, and that even as AI becomes more autonomous, there's a lot more to build to make humans and AIs work well together. A lot has happened since then, especially the massive progress in agents, so we wanted to revisit those instincts in light of everything we've learned, argue about them, and write down what we actually believe now. This is where we landed after a lot of debate. I'm happy with it!

hiring post-training hackers to make Tinker even better! x.com/tinkerapi/stat…

People sometimes ask why fine-tune when general-purpose models keep getting better. Bridgewater's work is a good reminder that with the right data -- here, expert judgements -- you can beat prompting-only approaches by a lot. @ddkang and the Bridgewater AIA Labs team are great -- glad to see them sharing this.

PPO had a second wave in the LLM era for reasons unanticipated by the original paper - the importance-ratio objective fixes biases from numeric error, async training, and forward pass noise - the clipping objective affects entropy through a mechanism that we didn't know about at the time of publication (DAPO, t.co/sBo9DeFS5Y)

PPO: rejected from NIPS 2017

Looking forward to what comes out of Geoffrey's new alignment org. His 2018 paper on AI safety via debate is one of my all-time favorites: an elegant framing of the scalable oversight problem, way ahead of its time. x.com/geoffreyirving…

Would be funny if inoculation prompting results in models that are much better at sandbox escapes and other forms of hacking because they get to spend the whole RL run practicing these things

Glad to see this -- renderers are a foundational component of the LLM stack. Renderers map between tokens and messages, which are invariant to tokenizer and formatting details. Most APIs, datasets, and RL environments are defined in terms of messages. Getting the details wrong x.com/PrimeIntellect…

Glad to be advising refine.ink, which uses AI to help authors and reviewers do deeper, more thorough analysis than unaided humans could practically do. Seems like a very positive direction for AI in science. x.com/ben_golub/stat…

Sharing our work on full-duplex multimodal models -- real-time interaction that's natural and intuitive without compromising on intelligence. We started Thinky in part to differentially advance capabilities for human-AI collaboration, which are underemphasized relative to x.com/thinkymachines…

Seeing the demos come together over the last week has been awesome -- so many things that previously required a special-purpose model (e.g. real-time translation, event detection in video) turn out to be zero-shot instruction following once you have a general-purpose model with

Luke and Rudolf's writing on keeping humans central in an AI-powered world sparked a lot of discussion at Thinking Machines. For me, it captured some things I'd been thinking about but hadn't put as clearly. The more I got to know them and learned about their work, the more I x.com/WorkshopLabs/s…

Great work by Chroma training a search agent with SoTA efficiency. Lots of cool details: a prune tool for editing context mid-search, a synthetic data pipeline with verification steps, and a curriculum that shifts from recall to precision. Trained with Tinker! x.com/trychroma/stat…