>
G
Model Details
Domain:
Model Access:
Open weights (unrestricted)
Introduction
Model Summary: Granite-3.1-2B-Base extends the context length of Granite-3.0-2B-Base from 4K to 128K using a progressive training strategy by increasing the supported context length in increments while adjusting RoPE theta until the model has successfully adapted to desired length of 128K. This long-context pre-training stage was performed using approximately 500B tokens.
Benchmarking
FLOPs
1.8e+23Notes: 6 FLOP / token / parameter * 2.5 * 10^9 parameters * 12*10^12 tokens = 1.8e+23 FLOP
Training
Training Code Accessibility
https://huggingface.co/ibm-granite/granite-3.1-2b-base
Apache 2.0Hardware
NVIDIA H100 SXM5 80GBSize Notes: 12T
Parameters
Parameters
2500000000Notes: 2.5B Model Architecture: Granite-3.1-2B-Base is based on a decoder-only dense transformer architecture. Core components of this architecture are: GQA and RoPE, MLP with SwiGLU, RMSNorm, and shared input/output embeddings.
Earth science
Vision