Model Details

Domain:

Task:

Model Access:

Open weights (unrestricted)

AI Tools Usage

This model is commonly used behind the scenes in AI tools.

Introduction

CPM-Bee is a fully open-source, commercially-usable Chinese-English bilingual base model with a capacity of ten billion parameters. It is the second milestone achieved through the training process of CPM-live. Utilizing the Transformer auto-regressive architecture, CPM-Bee has been pre-trained on an extensive corpus of trillion-scale tokens, thereby possessing remarkable foundational capabilities.

Benchmarking

FLOPs6.01e+22

Notes: 6 FLOP / parameter / token * 10*10e9 parameters * 1.002e11 tokens [see training dataset size notes] = 6.012e22 FLOP

Training

Training Code Accessibility"CPM-Bee It is a completely open source, commercially available tens of billions parameter Chinese and English base model" "The CPM-Bee base adoption protocol is “Generic Model License Agreement-Source Description-Publicity Restrictions-Commercial License” , This model is allowed for commercial use. " https://huggingface.co/openbmb/cpm-bee-10b

Size Notes: Planned to use 600GB of clean data. 600GB * 167M tokens/GB = 1.002e+11 tokens https://github.com/OpenBMB/CPM-Live/blob/master/plans/CPM-Bee%E8%AE%AD%E7%BB%83%E8%AE%A1%E5%88%92%E4%B9%A6.md