@srush_nlp
Researcher, Programmer https://t.co/cZl0wTfqGz
@sakshjn Oh, actually is that wrong? How does BPE treat that case. You only get 1 merge in that case.
The Token Game (srush.github.io/the-token-game/) It's BPE the mobile game? No reason for this to exist. I'm absolutely terrible at it.
@TheStalwart The whole thing is a bit weird though, as they have full access to the OpenAI codebase for many years. But they have very good people replicating it.
Interpretability blogs are the recipes blogs of ML. Not sure, why I have to read through the story of your grandmother's first house to find out you used gradients.
just put the mechanism in the bag bro.
pretty interesting document.
loops@_smitopAnthropic released their 2021 research roadmap earlier this year (because of copyright litigation); I think some people might find it interesting: i.sm4.ca/concise-resear…
How to Play Jazz Piano - srush.github.io/jazz-piano/ A guide to jazz chords and harmonization with visualizations. (humanslop).
Provably Correct Tensor Puzzles (srush.github.io/jax-lean/) In which we formally verify Python ML code by building a Jax-Lean transpiler.
I'm alternating between blogging about topics I know something about and ones where I am a total noob. hoping they converge in the limit
This is being spun like it is bad, but this is actually the kind of thing that should happen with open-source data communities.
Vals AI@ValsAIXiaomi open-sourced the RL environments behind MiMo v2.6. In two-thirds of the coding tasks, the answer is still in the task's Git history, and MiMo finds it.
casual drop of "thread originated by MCHammer"
Christian Szegedy@ChrSzegedyI wonder what @GaryMarcus thinks of this bet now. It was in a thread originated by @MCHammer that seems to be deleted, but I saved the screenshots back then.
@guillaumeclaret After writing this I realized you are the expert on this topic, so I am even more curious 😀 formal.land/docs/tools/roc…
At @COLM_conf SF next week. Would love to chat about: the interesting type of TTT, proofs for everything, biased RL, research communities outside universities. Feel free to dm me.
Sampling to Reinforce (srush.github.io/sampling-to-re…) An elementary tutorial on sampling w/o randomness with a focus on gaining intuition for variance reduction as the key challenge.
This blog was inspired by this shirt. No idea why.
Lucas Beyer (bl16)@giffmanaAs an adept of the Williams'92 church, i was excited to read this paper. But I got very confused by the terminology and didn't really find the paper clear. So here is my attempt: - only one rollout per prompt - assemble 128 rollouts (ie from 128 prompts) into one batch - use a batch only for one optim update. So this is 128 x 1 when GRPO is 32 x 4. Why this helps with stragglers on long rollouts: since there are no more groups, no rollout needs to wait for the others in its group to make it into the batch. Hence you can just take and batch rollouts as they complete. For example have 500 runners feed your queue and always keep batching the next 128 that arrive, no matter their order. Hope that helps.
Also leave me alone, I'm not going to all-caps reinforce, you can't make me.
A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification arxiv.org/abs/2107.07511
just found out this guy is now the CEO of LMArena, which is kind of cool, but I guess kind of bittersweet. I guess it is good to know that Arena knows their stuff.
(thinking this subarea might be fun to lean)
I'm on my phone monitoring the MiMo v2.6 response length on coding-obg8 like a degenerate gambler.
Fuli Luo@_LuoFuliNearly half a year of silence. We spent it studying one problem: how far RL can scale. MiMo-V2.6 is in the middle of its RL run right now. Three things we scaled: compute (~2B tokens per step, 1568 prompts × 16 rollouts, fully async), environments and harnesses (multi-task agentic RL, mixed across multiple harnesses in one run), and grader compute (agentic in-group credit assignment, with test-case and rubric-based rewards). We'll open-source the details piece by piece over the coming weeks. Streaming the run: t.co/ZSxahzJRju
Just insane you can watch this -> mimo.xiaomi.com/rl/
Guy who has only read 'Thinking, Fast and Slow', training his ML model: Getting a lot of 'System 2' vibes from this...
Someone pointed out that this is actually what happened with the OAI team that invented thinking models. so I'm a 🤡.
Lean Verified Transformers (srush.github.io/lean-transform…) In which we prove a bunch of Transformer invariants from scratch in Lean, and speculate about how hard it would be to do that for the rest of the world's code.
Noting that this is a tiny pedantic subset of TorchLean (leandojo.org/torchlean.html) from @Robertljg and co. That project is really impressive.
It's kind of fun that Math has a #1 Champion. I always click on the articles that start with "Mr. Math and friends have something to say".
Totally unclear to me why Mathematician are allowed to protest AI. I really liked enjoyed working on syntactic parsing, and GPT-2 had a lot of wrong citations.
I guess deep learning used to have a guy. Not really sure who we would elect to speak for us this days.
What are the good open ML discords? GPU MODE, Flash Linear Attention, Marin. Any others with talks?
@WilliamBarrHeld Thanks! Been following your run as well. Actually was thinking of making a Marin video of some sort, might reach out.
Last day at Cursor. An incredible place. Stopped by for coffee, they gave me a keyboard that afternoon, and started helping out on Composer the next week. Feel lucky to have worked with such a fearless and ambitious team.
Future goal is primarily to write a bit and maybe learn some math / Lean. In the mean time just hanging out in Brooklyn.
Spending some time in Cambridge MA next week. Happy to grab coffee. Things I’m kind of interested in: cool things to do now that proofs are cheap, things RL can’t do, views on what the heck CS is now.
Feel free to dm.
Red-Blue Pebble game rules (Hong & Kung, 81) * Cost 1: Blue -> Red / Red -> Blue * Cost 0: Compute (Red-only) / Delete O(n^3) naive matmul scales well with cache-size (red pebbles) under this communication model.
Strassen's algorithm under this toy model (with relatively small matrices).
The kernel folks are living on the front-lines. Where the the "cognitive abstractions" should live in a stack is such an interesting question. x.com/stuart_sul/sta…
this seems really interesting. x.com/marksaroufim/s…
Jax added support for opaque types with custom tangents. One cool use is proper handling of batched geometric objects. Here's a short demo of a differentiable rasterization, i.e. min L2 w/ 1500 triangles to a photo. Interesting to move beyond tensors. (docs.jax.dev/en/latest/hija…)
Jax docs is the best webpage.
Forget math proofs, we’re getting to the real frontier now.
Fantastic slides from the ICML tutorial of @MarkSchmidtUBC (“is opt theory relevant in 2026”) cs.ubc.ca/~schmidtm/Docu…
My brain is pretty split between ML I know how to teach and ML I do in practice. Really appreciate the honesty about when these are bridgeable.
(Annotated) Structure and Interpretation of Classicical Mechanics - srush.github.io/annotated-sicm An experiment in formalized textbooks. Implements SICM Ch 1 in Typescript / Jax with code-math alignment and a simulator.
This is an interesting book because it is written in a code literate style. Cursor was able to convert it over relatively easily. Interesting questions are when formal typing is useful for text like this, versus the style of the book.
Frontend Code Arena 😂 x.com/DavidSacks/sta…
I circled all the Transformer parts of k3. (I'm a sore loser). x.com/srush_nlp/stat…
Time for everyone’s favorite game! Is it a Transformer?
anyway, very cool model.
No idea what Thinking Machines is working on, but this line goes hard.
This is an incredible blog post. Peak “centaur chess” phase of AI coding. x.com/jarredsumner/s…
I love the idea that the problem *can* be fully specified in one sentence, but each of the individual steps still require a brilliant programmer and a Rube Goldberg machine of interlocking checks to get there.
It’s crazy that BPE was only 1 year before Transformers. I can’t even remember the before times. x.com/jzhou_jz/statu…
Look at the absolute slop people were publishing in Feb 2016.
It’s really convenient that you don’t have to read past Chapter 2 of Sutton+Barto anymore.
The Short Lesson.
@distributionat But yes, I agree with you that personally the specifics of warmup + layernorm mattered a lot, and were not at all obvious to me coming from an RNN background.
This @jxmnop guy is worth following. x.com/icmlconf/statu…
An incredible group. 🍾 x.com/EngramLab/stat…
unfortunately I have learned almost nothing new about robust information extraction.
@hallerite @Harman26Singh @agarwl_ (Mostly just curious)
@Harman26Singh @hallerite @agarwl_ Other thoughts: * Someone should open source a good KL kernel and remove that confounder * I don’t get why people use other rollout trajectories as PI * would love to see more work on generating PI
