Introduction

We introduce PaLM 2, a new state-of-the-art language model that has better multilingual and reasoning capabilities and is more compute-efficient than its predecessor PaLM (Chowdhery et al., 2022). PaLM 2 is a Transformer-based model trained using a mixture of objectives similar to UL2 (Tay et al., 2023). Through extensive evaluations on English and multilingual language, and reasoning tasks, we demonstrate that PaLM 2 has significantly improved quality on downstream tasks across different model sizes, while simultaneously exhibiting faster and more efficient inference compared to PaLM. This improved efficiency enables broader deployment while also allowing the model to respond faster, for a more natural pace of interaction. PaLM 2 demonstrates robust reasoning capabilities exemplified by large improvements over PaLM on BIG-Bench and other reasoning tasks. PaLM 2 exhibits stable performance on a suite of responsible AI evaluations, and enables inference-time control over toxicity without additional overhead or impact on other capabilities. Overall, PaLM 2 achieves state-of-the-art performance across a diverse set of tasks and capabilities.

Benchmarking

Training

Parameters

Authors

+0

Andrew M. Dai, David R. So, Dmitry Lepikhin, Jonathan H. Clark, Maxim Krikun, Melvin Johnson, Nan Du, Rohan Anil, Siamak Shakeri, Xavier Garcia, Yanping Huang, Yi Tay, Yong Cheng, Yonghui Wu, Yuanzhong Xu, Yujing Zhang, Zachary Nado, Bryan Richter, Alex Polozov, Andrew Nystrom, Fangxiaoyu Feng, Hanzhao Lin, Jacob Austin, Jacob Devlin, Kefan Xiao, Orhan Firat, Parker Riley, Steven Zheng, Yuhuai Wu, Zhongtao Liu, Jiahui Yu, Guy Gur-Ari, Weikang Zhou, Sneha Kudugunta, Sunipa Dev, Frederick Liu, Gustavo Hernandez Abrego, Kelvin Xu, Abe Ittycheriah, Daniel Sohn, John Nham, Le Hou, Siyuan Qiao, Pidong Wang, Zirui Wang, Laurent El Shafey, Hyeontaek Lim, Marcello Maggioni, Michael Isard, Paul Barham, Qiao Zhang, Tao Wang, Yash Katariya, Aurko Roy, Benjamin Lee, Brennan Saeta, Ce Zheng, Hadi Hashemi, Junwhan Ahn, Rajkumar Samuel, Steven Hand, Zhifeng Chen, Kiran Vodrahalli, Aakanksha Chowdhery, Ethan Dyer, Emanuel Taropa, Vlad Feinberg, James Bradbury, Reiner Pope, Wei Li, YaGuang Li, Eric Chu, Jeffrey Hui, Joshua Howland, Vlad Fienber, Aroma Mahendru, Michele Catasta, Vedant Misra, Kevin Robinson, Maysam Moussalem, Sebastian Ruder, Erica Moreira, Eric Ni, Paige Bailey, Lucas Gonzalez, Alexandre Passos, Slav Petrov, Gaurav Mishra, Mark Omernick, Ambrose Slone, Andrea Hu, Colin Cherry, Denny Zhou, Jan Botha, John Wieting, Joshua Maynez, Kathleen Kenealy, Kevin Brooks, Linting Xue, Markus Freitag, Martin Polacek, Pengcheng Yin, Sebastian Gehrmann, Xuezhi Wang, Kathy Meier-Hellstern, Christopher A. Choquette-Choo, Daniel Smilkov, Emily Reif, Alicia Parrish, Alex Castro Ros, Clément Crepy, Dasha Valter, Jeremy Hurwitz, Katherine Lee, Mark Díaz, Marie Pellat, Matthew Jagielski, Renee Shelby, Shachi Dave