FLOPs2.9e+24

Notes: "On these benchmarks, Grok-1 displayed strong results, surpassing all other models in its compute class, including ChatGPT-3.5 and Inflection-1. It is only surpassed by models that were trained with a significantly larger amount of training data and compute resources like GPT-4" Per table, Grok-1 is surpassed by Palm 2, Claude 2, GPT-4, so it required less compute than these three models. Palm 2 was trained on 7e24 FLOP. GPT-3.5 is ~2.6e24. Inflection-1's compute is not public/known by us but Inflection says Inflection-1 compute was <= Palm-540B's (which was ~2.5e24). For optimal training, our current working hypothesis is that you still need something like Chinchilla scaling on the total number of parameters in the model, even for MoE models, so optimal dataset size would be 20*310B tokens. With 25%*314B params active per forward pass, this would be around 3e24 FLOP. https://www.wolframalpha.com/input?i=20*310+billion+*+6+*+25%25+*+314+billion