Model Details

Domain:

Task:

Model Access:

Open weights (unrestricted)

Citations:

23135

Introduction

Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP). The effectiveness of transfer learning has given rise to a diversity of approaches, methodology, and practice. In this paper, we explore the landscape of transfer learning techniques for NLP by introducing a unified framework that converts all text-based language problems into a text-to-text format. Our systematic study compares pre-training objectives, architectures, unlabeled data sets, transfer approaches, and other factors on dozens of language understanding tasks. By combining the insights from our exploration with scale and our new ``Colossal Clean Crawled Corpus'', we achieve state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more. To facilitate future work on transfer learning for NLP, we release our data set, pre-trained models, and code.

Benchmarking

FLOPs

3.3e+22

Notes: https://arxiv.org/ftp/arxiv/papers/2104/2104.10350.pdf Table 4, 4.05e22 update: 3.3e22 per FLAN paper from Google https://arxiv.org/pdf/2210.11416.pdf 6ND rule suggests somewhat more FLOPs: 6 * 1T * 11B = 6.6e22

Training

Training Code Accessibility

Apache for code and weights: https://github.com/google-research/text-to-text-transfer-transformer Data is C4 which is open

Hardware

Google TPU v3

Hardware Quantity

512

Size Notes: "This produces a collection of text that is not only orders of magnitude larger than most data sets used for pre-training (about 750 GB) but also comprises reasonably clean and natural English text. We dub this data set the “Colossal Clean Crawled Corpus” (or C4 for short) and release it as part of TensorFlow Datasets" 750GB * 200M word/GB * 4/3 tokens per word = 2e11. Total tokens seen is about 1T: "We therefore pre-train our models for 1 million steps on a batch size of 2^11 sequences of length 512, corresponding to a total of about 1 trillion pre-training tokens"