@srush_nlp
Researcher, Programmer https://t.co/cZl0wTfqGz
A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification arxiv.org/abs/2107.07511
just found out this guy is now the CEO of LMArena, which is kind of cool, but I guess kind of bittersweet. I guess it is good to know that Arena knows their stuff.
(thinking this subarea might be fun to lean)
I'm on my phone monitoring the MiMo v2.6 response length on coding-obg8 like a degenerate gambler.
Fuli Luo@_LuoFuli·Nearly half a year of silence. We spent it studying one problem: how far RL can scale. MiMo-V2.6 is in the middle of its RL run right now. Three things we scaled: compute (~2B tokens per step, 1568 prompts × 16 rollouts, fully async), environments and harnesses (multi-task agentic RL, mixed across multiple harnesses in one run), and grader compute (agentic in-group credit assignment, with test-case and rubric-based rewards). We'll open-source the details piece by piece over the coming weeks. Streaming the run: t.co/ZSxahzJRju
Just insane you can watch this -> mimo.xiaomi.com/rl/
Guy who has only read 'Thinking, Fast and Slow', training his ML model: Getting a lot of 'System 2' vibes from this...
Someone pointed out that this is actually what happened with the OAI team that invented thinking models. so I'm a 🤡.
Lean Verified Transformers (srush.github.io/lean-transform…) In which we prove a bunch of Transformer invariants from scratch in Lean, and speculate about how hard it would be to do that for the rest of the world's code.
Noting that this is a tiny pedantic subset of TorchLean (leandojo.org/torchlean.html) from @Robertljg and co. That project is really impressive.
It's kind of fun that Math has a #1 Champion. I always click on the articles that start with "Mr. Math and friends have something to say".
Totally unclear to me why Mathematician are allowed to protest AI. I really liked enjoyed working on syntactic parsing, and GPT-2 had a lot of wrong citations.
I guess deep learning used to have a guy. Not really sure who we would elect to speak for us this days.
What are the good open ML discords? GPU MODE, Flash Linear Attention, Marin. Any others with talks?
@WilliamBarrHeld Thanks! Been following your run as well. Actually was thinking of making a Marin video of some sort, might reach out.
Last day at Cursor. An incredible place. Stopped by for coffee, they gave me a keyboard that afternoon, and started helping out on Composer the next week. Feel lucky to have worked with such a fearless and ambitious team.
Future goal is primarily to write a bit and maybe learn some math / Lean. In the mean time just hanging out in Brooklyn.
Spending some time in Cambridge MA next week. Happy to grab coffee. Things I’m kind of interested in: cool things to do now that proofs are cheap, things RL can’t do, views on what the heck CS is now.
Feel free to dm.
Red-Blue Pebble game rules (Hong & Kung, 81) * Cost 1: Blue -> Red / Red -> Blue * Cost 0: Compute (Red-only) / Delete O(n^3) naive matmul scales well with cache-size (red pebbles) under this communication model.
Strassen's algorithm under this toy model (with relatively small matrices).
The kernel folks are living on the front-lines. Where the the "cognitive abstractions" should live in a stack is such an interesting question. x.com/stuart_sul/sta…
this seems really interesting. x.com/marksaroufim/s…
Jax added support for opaque types with custom tangents. One cool use is proper handling of batched geometric objects. Here's a short demo of a differentiable rasterization, i.e. min L2 w/ 1500 triangles to a photo. Interesting to move beyond tensors. (docs.jax.dev/en/latest/hija…)
Jax docs is the best webpage.
Forget math proofs, we’re getting to the real frontier now.
Fantastic slides from the ICML tutorial of @MarkSchmidtUBC (“is opt theory relevant in 2026”) cs.ubc.ca/~schmidtm/Docu…
My brain is pretty split between ML I know how to teach and ML I do in practice. Really appreciate the honesty about when these are bridgeable.
(Annotated) Structure and Interpretation of Classicical Mechanics - srush.github.io/annotated-sicm An experiment in formalized textbooks. Implements SICM Ch 1 in Typescript / Jax with code-math alignment and a simulator.
This is an interesting book because it is written in a code literate style. Cursor was able to convert it over relatively easily. Interesting questions are when formal typing is useful for text like this, versus the style of the book.
Frontend Code Arena 😂 x.com/DavidSacks/sta…
I circled all the Transformer parts of k3. (I'm a sore loser). x.com/srush_nlp/stat…
Time for everyone’s favorite game! Is it a Transformer?
anyway, very cool model.
No idea what Thinking Machines is working on, but this line goes hard.
This is an incredible blog post. Peak “centaur chess” phase of AI coding. x.com/jarredsumner/s…
I love the idea that the problem *can* be fully specified in one sentence, but each of the individual steps still require a brilliant programmer and a Rube Goldberg machine of interlocking checks to get there.
It’s crazy that BPE was only 1 year before Transformers. I can’t even remember the before times. x.com/jzhou_jz/statu…
Look at the absolute slop people were publishing in Feb 2016.
It’s really convenient that you don’t have to read past Chapter 2 of Sutton+Barto anymore.
The Short Lesson.
@distributionat But yes, I agree with you that personally the specifics of warmup + layernorm mattered a lot, and were not at all obvious to me coming from an RNN background.
This @jxmnop guy is worth following. x.com/icmlconf/statu…
An incredible group. 🍾 x.com/EngramLab/stat…
unfortunately I have learned almost nothing new about robust information extraction.
@hallerite @Harman26Singh @agarwl_ (Mostly just curious)
@Harman26Singh @hallerite @agarwl_ Other thoughts: * Someone should open source a good KL kernel and remove that confounder * I don’t get why people use other rollout trajectories as PI * would love to see more work on generating PI
