@barret_zoph
VP Research GoogleDeepmind Past: - VP & GM Enterprise (@openai) - CTO & Co-Founder TML (@thinkymachines) - VP Research (Post-Training) @openai - Google Brain
Excited to announce I am joining Google DeepMind! I started my AI career in the Google Brain Residency class and am excited to rejoin. I am going to be working on RL and Post-Training – excited to announce more soon. Google has all the ingredients to continue to be incredibly successful in the AI space and am excited to be part of that mission.
This was quite a fun paper and project!
Liam Fedus@LiamFedus·An excellent history of scaling laws from @jietang. In 2020, we explored the limits of sparsity in Switch Transformers by routing each token to only 1 out of 2048 experts (in retrospect, a bold choice). The model had fewer than 3B activated parameters, but 1.6T total parameters (comparable to today's frontier models). The 1.6T model achieved better C4 perplexities than the T5 models using far less compute, set a new SOTA on TriviaQA, but was dumb as bricks on reasoning tasks like SuperGLUE. The lesson was that the optimal tokens-per-parameter ratio is highly task-dependent. Or as @NShazeer had already intuited: FLOPs were intelligence; parameters were knowledge!
Super excited to join the team! x.com/fidjissimo/sta…
Great post on on-policy distillation the people should check out! x.com/thinkymachines…
Excited to release Tinker and see what the community uses it for. x.com/thinkymachines…
Exciting mission with a great team! With the progress of AI, now is the right time to start approaching these problems! x.com/LiamFedus/stat…
Excited to be supporting this, please reach out if you are interested x.com/woosuk_k/statu…
Excited to share our first blog post -- one of many to follow! x.com/thinkymachines…
Super excited to be part of this incredible team and company. Please reach out if you are interested in joining! x.com/miramurati/sta…
I posted this note to OpenAI. Hey everybody, I have decided to leave OpenAI. This was a very difficult decision as I have has such an incredible time at OpenAI. I got to join right before ChatGPT and helped build the post-training team from scratch with John Schulman and
Super excited this is rolling out! Real time speech to speech will be a powerful feature -- I am very bullish on multi-modal being a core component of AI products. This was a great collaboration with post-training (h/t to @kirillov_a_n & @shuchaobi + team on post-training) and x.com/OpenAI/status/…
