Model Details

Domain:

Task:

Model Access:

Open weights (restricted use)

Introduction

In this work, the Localized Filtering-based Attention (LFA) is introduced to incorporate prior knowledge of local dependencies of natural language into Attention. Based on LFA, we develop and release Yuan 2.0, a large language model with parameters ranging from 2.1 billion to 102.6 billion. A data filtering and generation method is presented to build pretraining and fine-tuning dataset in high quality. A distributed training method with non-uniform pipeline parallel, data parallel, and optimizer parallel is proposed, which greatly reduces the bandwidth requirements of intra-node communication, and achieves good performance in large-scale distributed training. Yuan 2.0 models display impressive ability in code generation, math problem-solving, and chat compared with existing models. The latest version of YUAN 2.0, including model weights and source code, is accessible at Github.

Benchmarking

FLOPs

1.78e+23

Notes: Trained on 288B tokens 6*102.6b*288b = 1.78e23

Training

Training Code Accessibility

commercial ok, but nothing that "may cause harm to the country and society, or for any services that have not undergone security assessment and filing" https://huggingface.co/IEITYuan/Yuan2-102B-hf https://github.com/IEIT-Yuan/Yuan-2.0?tab=License-1-ov-file

Size Notes: Most likely the 288B tokens do not represent multiple epochs. As a sense check, Table 2 appears to indicate that 5.73% of pre-training tokens come from synthetically generated text output by GPT-3.5. If the full training corpus is 288B tokens, this would imply ~$24k in API costs at $1.50/1M tokens to generate the data, which seems plausible.