>
C
Introduction
We introduce CodeGeeX, a large-scale multilingual code generation model with 13 billion parameters, pre-trained on a large code corpus of more than 20 programming languages. As of June 22, 2022, CodeGeeX has been trained on more than 850 billion tokens on a cluster of 1,536 Ascend 910 AI Processors.
Benchmarking
FLOPs
6.63e+22Notes: Assume 1 epoch on 850B tokens. C=6DN=6*850B*13B https://www.wolframalpha.com/input?i=6+*+13+billion+*+850+billion
Training
Training Code Accessibility
Training code is Apache 2.0. pretrain code here:
https://github.com/THUDM/CodeGeeX/blob/main/scripts/pretrain_codegeex.sh
Model has a restricted license: https://github.com/THUDM/CodeGeeX/blob/main/MODEL_LICENSE
data partially open (The Pile)Hardware
Huawei Ascend 910Size Notes: As of June 22, 2022, CodeGeeX has been trained on more than 850 billion tokens
Parameters
Parameters
13000000000Notes: "We introduce CodeGeeX, a large-scale multilingual code generation model with 13 billion parameters"
Multimodal
Vision