Model Details

Domain:

Language

Task:

Code generation

Model Access:

Open weights (restricted use)

AI Tools Usage

This model is commonly used behind the scenes in AI tools.

Introduction

We introduce CodeGeeX, a large-scale multilingual code generation model with 13 billion parameters, pre-trained on a large code corpus of more than 20 programming languages. As of June 22, 2022, CodeGeeX has been trained on more than 850 billion tokens on a cluster of 1,536 Ascend 910 AI Processors.

Benchmarking

FLOPs6.63e+22

Notes: Assume 1 epoch on 850B tokens. C=6DN=6*850B*13B https://www.wolframalpha.com/input?i=6+*+13+billion+*+850+billion

Training

Training Code AccessibilityTraining code is Apache 2.0. pretrain code here: https://github.com/THUDM/CodeGeeX/blob/main/scripts/pretrain_codegeex.sh Model has a restricted license: https://github.com/THUDM/CodeGeeX/blob/main/MODEL_LICENSE data partially open (The Pile)

HardwareHuawei Ascend 910

Size Notes: As of June 22, 2022, CodeGeeX has been trained on more than 850 billion tokens