>
SEA-LION version 1 was released in December 2023 and trained from scratch with 1 trillion tokens, 13 percent of which are from SEA.
FLOPs4.33e+22
Notes: 6 FLOP / parameter / token * 7B parameters * 980B tokens = 4.116e+22 FLOP 312000000000000 FLOP / GPU / sec * 256 GPUs * 22 days * 24 hours / day * 3600 sec / hour * 0.3 [assumed utilization] = 4.5546209e+22 FLOP sqrt(4.116e+22*4.5546209e+22) = 4.3297598e+22 FLOP
Training Code AccessibilityMIT license https://huggingface.co/aisingapore/SEA-LION-v1-7B https://github.com/aisingapore/sealion
HardwareNVIDIA A100 SXM4 40 GB
Hardware Quantity256
Size Notes: "SEA-LION was trained on 980B tokens"
Parameters7000000000