>
J
Model Details
Domain:
Model Access:
Open weights (unrestricted)
Introduction
Following the successful release of Jais-13B and Jais-13B-chat in August 2023, we are excited to launch the new state-of-the-art Arabic centric large language model Jais-30B and Jais-30B-chat. With more than twice the number of parameters as that in our previous release, Jais-30B and Jais-30B-chat models exhibit vastly better performance in both Arabic and English languages. Like its predecessor, the Jais-30B is the most powerful open bilingual model at its scale. Not only are the Jais-30B models the best in the world at Arabic NLP and generative AI tasks, they also are highly competitive with English language models of a similar size. We are also proud to announce the release of Jais-30B and Jais-30B-chat model weights along with the accompanying inference code to the community. This release marks a significant milestone in our ongoing commitment to elevate Arabic language processing by positioning it at the forefront of generative AI research and development.
Benchmarking
FLOPs1.04e+23
Notes: 6 FLOP / token / parameter * 30 * 10^9 parameters * 427 * 10^9 tokens [ see dataset size notes] = 7.686e+22 FLOP 7500000000000000 FLOP / chip / sec * 16 chips * 1080 hours * 3600 sec / hour * 0.3 [assumed utilization] = 1.39968e+23 FLOP sqrt(7.686e+22*1.39968e+23) = 1.0372049e+23 FLOP
Training
Training Code Accessibilityapache 2.0 https://huggingface.co/inceptionai/jais-30b-v1
HardwareCerebras CS-2
Hardware Quantity16
Size Notes: "126 billion Arabic tokens, 251 billion English tokens, and 50 billion code tokens" total: 427 billion tokens Batch size 2640 Steps 79k
Parameters
Parameters30000000000
Notes: 30B "The backbone of Jais-30B is a causal decoder-only large language model. It is engineered with 48 transformer blocks, 56 attention heads, and an embedding dimension of 7168. "
