Lilian Weng

It is a hard and sad decision. I shared this message with folks at Thinky. Thank you all for the time together♥️ Just as the last sentence in my message: The future worth building is human.

I've always been a curiosity-driven person. Leaning new knowledge or figuring out how to solve problems that are not well defined often gives me simple joy. The fun part of being a cofounder is how it "forced" me to develop new perspectives on company strategies, team building (at the company level), and much more, as well as how these seemingly abstract terms are deeply connected with our day-to-day actions, planning and narratives. With real-world experience, ideas that once felt theoretical become approachable and it is true for many terms in many domains.

Inkling is our open weights model. It aims to serve as a foundation with solid performance across a broad categories of capabilities, for use in practice and customization. Play it on Tinker! 😄 x.com/thinkymachines…

new post on harness engineering for AI self-improvement: t.co/ZYvGfVs61k It is hard to forecast how much the future of RSI will rely on harnesses. Likely harness engineering will evolve in the direction of self-improvement and enable auto-research, and, in turn, smarter models keeps harnesses simple. Even when many harness improvement get eventually internalized into core model, the need to specify goals and context will not disappear.

A super long overdue (3+ years?) post on scaling laws. Compute is expensive. Scaling laws are a way to help us reason about the optimal compute allocation between data and model size before committing to a large run. The post covers what scaling laws predict, how compute-optimal allocation works, why Kaplan et al. and Chinchilla disagree, and how data limits + fitting details make extrapolation tricky. t.co/HP26eJvjHB

I wonder how many more people will just ask model to summarize the post, in comparison to a year ago. 😂

Soon I should set up the model to update Lil’Log automatically but I’m not there yet.

We would love to see more collaboration and research in the field of human-AI interactivity. Check it out! x.com/thinkymachines…

I only recently read more about the concept of system accidents by Charles Perrow, very insightful and relatable.

In the past few months, we had a lot of fun (and stress 😅) to produce 12 versions (+ many subversions) and 137 pages in our training run log book. Turns out human-human collaboration is important to improving human-AI collaboration. 😊 x.com/thinkymachines…

Building technologies for better human-AI collaboration on next gen hardware at scale. Exciting. x.com/thinkymachines…

I’ve been telling people this a lot today: I enjoy so much working with people who care about what they are building and craftsmanship. It is a privilege to have a chance to work on something I’m passionate about, beyond making a living. I cherish it and don’t take it for

On-policy distillation provides an elegant way to use the teacher model as a process reward model to provide dense reward while preventing SFT style "OOD shock" during rollout. x.com/thinkymachines…

GPUs are expensive and setting up the infrastructure to make GPUs work for you properly is complex, making experimentation on cutting-edge models challenging for researchers and ML practitioners. Providing high quality research tooling is one of the most effective ways to