>
P
Model Details
Introduction
We introduce PaLM 2, a new state-of-the-art language model that has better multilingual and reasoning capabilities and is more compute-efficient than its predecessor PaLM (Chowdhery et al., 2022). PaLM 2 is a Transformer-based model trained using a mixture of objectives similar to UL2 (Tay et al., 2023). Through extensive evaluations on English and multilingual language, and reasoning tasks, we demonstrate that PaLM 2 has significantly improved quality on downstream tasks across different model sizes, while simultaneously exhibiting faster and more efficient inference compared to PaLM. This improved efficiency enables broader deployment while also allowing the model to respond faster, for a more natural pace of interaction. PaLM 2 demonstrates robust reasoning capabilities exemplified by large improvements over PaLM on BIG-Bench and other reasoning tasks. PaLM 2 exhibits stable performance on a suite of responsible AI evaluations, and enables inference-time control over toxicity without additional overhead or impact on other capabilities. Overall, PaLM 2 achieves state-of-the-art performance across a diverse set of tasks and capabilities.
Benchmarking
FLOPs
7.34e+24Notes: Compute Requirements "Not reported." Paper suggests heuristic of C=6ND. Based on 340B parameters and 3.6T tokens, training compute would be around 7.3*10^24 FLOP.
Training
Hardware
Google TPU v4Size Notes: "The pre-training corpus is significantly larger than the corpus used to train PaLM" so greater than 6e+11. According to the leaked documents viewed by CNBC, the corpus was 3.6 trillion tokens or around 2.7*10^12 words. https://www.cnbc.com/2023/05/16/googles-palm-2-uses-nearly-five-times-more-text-data-than-predecessor.html
Parameters
Parameters
340000000000Notes: Model Architecture: "PaLM-2 is a new state-of-the-art language model. We have small, medium, and large variants that use stacked layers based on the Transformer architecture, with varying parameters depending on model size. Further details of model size and architecture are withheld from external publication." However, the parameter count was leaked to CNBC: https://www.cnbc.com/2023/05/16/googles-palm-2-uses-nearly-five-times-more-text-data-than-predecessor.html
Authors
Andrew M. Dai, David R. So, Dmitry Lepikhin, Jonathan H. Clark, Maxim Krikun, Melvin Johnson, Nan Du, Rohan Anil, Siamak Shakeri, Xavier Garcia, Yanping Huang, Yi Tay, Yong Cheng, Yonghui Wu, Yuanzhong Xu, Yujing Zhang, Zachary Nado, Bryan Richter, Alex Polozov, Andrew Nystrom, Fangxiaoyu Feng, Hanzhao Lin, Jacob Austin, Jacob Devlin, Kefan Xiao, Orhan Firat, Parker Riley, Steven Zheng, Yuhuai Wu, Zhongtao Liu, Jiahui Yu, Guy Gur-Ari, Weikang Zhou, Sneha Kudugunta, Sunipa Dev, Frederick Liu, Gustavo Hernandez Abrego, Kelvin Xu, Abe Ittycheriah, Daniel Sohn, John Nham, Le Hou, Siyuan Qiao, Pidong Wang, Zirui Wang, Laurent El Shafey, Hyeontaek Lim, Marcello Maggioni, Michael Isard, Paul Barham, Qiao Zhang, Tao Wang, Yash Katariya, Aurko Roy, Benjamin Lee, Brennan Saeta, Ce Zheng, Hadi Hashemi, Junwhan Ahn, Rajkumar Samuel, Steven Hand, Zhifeng Chen, Kiran Vodrahalli, Aakanksha Chowdhery, Ethan Dyer, Emanuel Taropa, Vlad Feinberg, James Bradbury, Reiner Pope, Wei Li, YaGuang Li, Eric Chu, Jeffrey Hui, Joshua Howland, Vlad Fienber, Aroma Mahendru, Michele Catasta, Vedant Misra, Kevin Robinson, Maysam Moussalem, Sebastian Ruder, Erica Moreira, Eric Ni, Paige Bailey, Lucas Gonzalez, Alexandre Passos, Slav Petrov, Gaurav Mishra, Mark Omernick, Ambrose Slone, Andrea Hu, Colin Cherry, Denny Zhou, Jan Botha, John Wieting, Joshua Maynez, Kathleen Kenealy, Kevin Brooks, Linting Xue, Markus Freitag, Martin Polacek, Pengcheng Yin, Sebastian Gehrmann, Xuezhi Wang, Kathy Meier-Hellstern, Christopher A. Choquette-Choo, Daniel Smilkov, Emily Reif, Alicia Parrish, Alex Castro Ros, Clément Crepy, Dasha Valter, Jeremy Hurwitz, Katherine Lee, Mark Díaz, Marie Pellat, Matthew Jagielski, Renee Shelby, Shachi Dave
Image generation
Multimodal
Vision
Earth science