Model Details

Domain:

Task:

Model Access:

Open weights (restricted use)

AI Tools Usage

This model is commonly used behind the scenes in AI tools.

Introduction

In this work, the Localized Filtering-based Attention (LFA) is introduced to incorporate prior knowledge of local dependencies of natural language into Attention. Based on LFA, we develop and release Yuan 2.0, a large language model with parameters ranging from 2.1 billion to 102.6 billion. A data filtering and generation method is presented to build pretraining and fine-tuning dataset in high quality. A distributed training method with non-uniform pipeline parallel, data parallel, and optimizer parallel is proposed, which greatly reduces the bandwidth requirements of intra-node communication, and achieves good performance in large-scale distributed training. Yuan 2.0 models display impressive ability in code generation, math problem-solving, and chat compared with existing models. The latest version of YUAN 2.0, including model weights and source code, is accessible at Github.

Benchmarking

FLOPs1.78e+23

Notes: Trained on 288B tokens 6*102.6b*288b = 1.78e23

Training

Training Code Accessibilitycommercial ok, but nothing that "may cause harm to the country and society, or for any services that have not undergone security assessment and filing" https://huggingface.co/IEITYuan/Yuan2-102B-hf https://github.com/IEIT-Yuan/Yuan-2.0?tab=License-1-ov-file

Size Notes: Most likely the 288B tokens do not represent multiple epochs. As a sense check, Table 2 appears to indicate that 5.73% of pre-training tokens come from synthetically generated text output by GPT-3.5. If the full training corpus is 288B tokens, this would imply ~$24k in API costs at $1.50/1M tokens to generate the data, which seems plausible.