Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism

Megatron-LM: 使用模型并行训练多十亿参数语言模型

Mohammad Shoeybi 1 2 Mostofa Patwary 1 2 Raul Puri 1 2 Patrick LeGresley 2 Jared Casper 2 Bryan Catanzaro 2

Mohammad Shoeybi 1 2 Mostofa Patwary 1 2 Raul Puri 1 2 Patrick LeGresley 2 Jared Casper 2 Bryan Catanzaro 2

Abstract

摘要

1. Introduction

1. 引言

Recent work in language modeling demonstrates that training large transformer models advances the state of the art in Natural Language Processing applications. However, very large models can be quite difficult to train due to memory constraints. In this work, we present our techniques for training very large transformer models and implement a simple, efficient intra-layer model parallel approach that enables training transformer models with billions of parameters. Our approach does not require a new compiler or library changes, is orthogonal and complimentary to pipeline model parallelism, and can be fully implemented with the insertion of a few communication operations in native PyTorch. We illustrate this approach by converging transformer based models up to 8.3 billion parameters using 512 GPUs. We sustain 15.1 PetaFLOPs across the entire application with $76%$ scaling efficiency when compared to a strong single GPU baseline that sustains 39 TeraFLOPs, which is $30%$ of peak FLOPs. To demonstrate that large language models can further advance the state of the art (SOTA), we train an 8.3 billion parameter transformer language model similar to GPT-2 and a 3.9 billion parameter model similar to BERT. We show that careful attention to the placement of layer normalization in BERT-like models is critical to achieving increased performance as the model size grows. Using the GPT-2 model we achieve SOTA results on the WikiText 103 (10.8 compared to SOTA perplexity of 15.8) and LAMBADA $66.5%$ compared to SOTA accuracy of $63.2%$ ) datasets. Our BERT model achieves SOTA results on the RACE dataset $g0.9%$ compared to SOTA accuracy of $89.4%$ ).

近期在语言模型方面的工作表明，训练大型 Transformer 模型可以推动自然语言处理 (Natural Language Processing) 应用的发展。然而，由于内存限制，非常大的模型可能很难训练。在这项工作中，我们介绍了训练非常大的 Transformer 模型的技术，并实现了一种简单高效的层内模型并行方法，使得能够训练参数量达数十亿的 Transformer 模型。我们的方法不需要新的编译器或库更改，与管道模型并行正交且互补，并且可以通过在原生 PyTorch 中插入几个通信操作来完全实现。我们通过使用 512 个 GPU 收敛参数量达 83 亿的基于 Transformer 的模型来说明这种方法。与一个能维持 39 TeraFLOPs（占峰值 FLOPs 的 30%）的单 GPU 强基线相比，我们在整个应用中保持了 15.1 PetaFLOPs，扩展效率为 76%。为了证明大语言模型可以进一步推动技术发展，我们训练了一个类似于 GPT-2 的 83 亿参数的 Transformer 语言模型和一个类似于 BERT 的 39 亿参数的模型。我们展示了在类似 BERT 的模型中，对层归一化位置的仔细关注对于随着模型规模增长而提高性能至关重要。使用 GPT-2 模型，我们在 WikiText 103 数据集上达到了 10.8 的困惑度（相比于当前最佳的 15.8），以及 LAMBADA 数据集上 66.5% 的准确率（相比于当前最佳的 63.2%）。我们的 BERT 模型在 RACE 数据集上达到了 80.9% 的准确率（相比于当前最佳的 89.4%）。

Natural Language Processing (NLP) is advancing quickly in part due to an increase in available compute and dataset size. The abundance of compute and data enables training increasingly larger language models via unsupervised pre training (Devlin et al., 2018; Radford et al., 2019). Empirical evi- dence indicates that larger language models are dramatically more useful for NLP tasks such as article completion, question answering, and natural language inference (Lan et al., 2019; Raffel et al., 2019). By finetuning these pretrained language models on downstream natural language tasks, one can achieve state of the art results as shown in recent work (Devlin et al., 2018; Peters et al., 2018; Howard & Ruder, 2018; Radford et al., 2018; 2017; Rama chandra n et al., 2016; Liu et al., 2019b; Dai et al., 2019; Yang et al., 2019; Liu et al., 2019a; Lan et al., 2019).

自然语言处理 (NLP) 正在快速发展，部分原因是可用计算资源和数据集规模的增加。丰富的计算资源和数据使得可以通过无监督预训练来训练越来越大的语言模型 (Devlin et al., 2018; Radford et al., 2019)。实证证据表明，更大的语言模型在文章补全、问答和自然语言推理等 NLP 任务上表现出显著更好的性能 (Lan et al., 2019; Raffel et al., 2019)。通过在下游自然语言任务上微调这些预训练的语言模型，可以取得最先进的结果，如最近的研究所示 (Devlin et al., 2018; Peters et al., 2018; Howard & Ruder, 2018; Radford et al., 2018; 2017; Ramachandran et al., 2016; Liu et al., 2019b; Dai et al., 2019; Yang et al., 2019; Liu et al., 2019a; Lan et al., 2019)。

As these models become larger, they exceed the memory limit of modern processors, and require additional memory management techniques such as activation check pointing (Chen et al., 2016). Widely used optimization algorithms such as ADAM require additional memory per parameter to store momentum and other optimizer state, which reduces the size of models that can be effectively trained. Several approaches to model parallelism overcome this limit by partitioning the model such that the weights and their associated optimizer state do not need to reside concurrently on the processor. For example, GPipe (Huang et al., 2018) and Mesh-Tensorflow (Shazeer et al., 2018) provide frameworks for model parallelism of different kinds. However, they require rewriting the model, and rely on custom compilers and frameworks that are still under development.

随着这些模型变得越来越大，它们超出了现代处理器的内存限制，需要额外的内存管理技术，例如激活检查点 (Chen et al., 2016)。广泛使用的优化算法如 ADAM 需要为每个参数额外分配内存以存储动量和其他优化器状态，这减少了可以有效训练的模型的规模。几种模型并行方法通过分区模型克服了这一限制，使得权重及其相关的优化器状态不需要同时驻留在处理器上。例如，GPipe (Huang et al., 2018) 和 Mesh-Tensorflow (Shazeer et al., 2018) 提供了不同类型的模型并行框架。然而，这些方法需要重写模型，并依赖于仍在开发中的自定义编译器和框架。

In this work, we implement a simple and efficient model parallel approach using intra-layer model-parallelism. We exploit the inherent structure in transformer based language models to make a simple model-parallel implementation that trains efficiently in PyTorch, with no custom $C++$ code or compiler required. This approach is orthogonal to pipelinebased model parallelism as advocated by approaches such as GPipe (Huang et al., 2018).

在本工作中，我们实现了一种简单且高效的支持层内模型并行的方法。我们利用了基于 Transformer 的语言模型中的固有结构，以实现一个简单的模型并行实现，该实现在 PyTorch 中训练效率高，无需自定义的 $C++$ 代码或编译器。这种方法与 GPipe (Huang et al., 2018) 等方法所提倡的基于管道的模型并行正交。

图 1: 模型架构示例 (Example of model architecture)

Figure 1. Model (blue) and mode $^{+}$ data (green) parallel FLOPS as a function of number of GPUs. Model parallel (blue): up to 8-way model parallel weak scaling with approximately 1 billion parameters per GPU (e.g. 2 billion for 2 GPUs and 4 billion for 4 GPUs). Model+data parallel (green): similar configuration as model parallel combined with 64-way data parallel.

图 1. 模型 (蓝色) 和模式 $^{+}$ 数据 (绿色) 并行 FLOPS 随 GPU 数量的变化。模型并行 (蓝色)：最多 8 路模型并行弱扩展，每个 GPU 大约有 10 亿参数（例如，2 个 GPU 为 20 亿参数，4 个 GPU 为 40 亿参数）。模型+数据并行 (绿色)：与模型并行类似的配置结合 64 路数据并行。

a baseline by training a model of 1.2 billion parameters on a single NVIDIA V100 32GB GPU, that sustains 39 TeraFLOPs. This is $30%$ of the theoretical peak FLOPS for a single GPU as configured in a DGX-2H server, and is thus a strong baseline. Scaling the model to 8.3 billion parameters on 512 GPUs with 8-way model parallelism, we achieve up to 15.1 PetaFLOPs per second sustained over the entire application. This is $76%$ scaling efficiency compared to the single GPU case. Figure 1 shows more detailed scaling results.

通过在单个 NVIDIA V100 32GB GPU 上训练一个包含 12 亿参数的模型，该模型能够持续提供 39 TeraFLOPs 的性能。这相当于 DGX-2H 服务器中单个 GPU 理论峰值 FLOPS 的 $30%$ ，因此是一个强大的基线。将模型扩展到 512 个 GPU 上，使用 8 路模型并行，参数量增加到 83 亿，我们实现了每秒最高 15.1 PetaFLOPs 的持续性能。与单个 GPU 情况相比，这达到了 $76%$ 的扩展效率。图 1 显示了更详细的扩展结果。

图 1: 更详细的扩展结果

To analyze the effect of model size scaling on accuracy, we train both left-to-right GPT-2 (Radford et al., 2019) language models as well as BERT (Devlin et al., 2018) bidirectional transformers and evaluate them on several downstream tasks. We show that the existing BERT architecture results in model degradation as the size increases. We overcome this challenge by rearranging the layer normalization and residual connection in the transformer layers and show that with this change, results for the downstream tasks on development sets improve monotonically as the model size increases. In addition, we show that our models achieve test set state of the art (SOTA) results on WikiText 103, cloze-style prediction accuracy on LAMBADA, and reading comprehension RACE datasets.

为了分析模型大小扩展对准确率的影响，我们训练了从左到右的 GPT-2 (Radford et al., 2019) 语言模型以及 BERT (Devlin et al., 2018) 双向 Transformer，并在多个下游任务上评估它们。我们发现现有的 BERT 架构随着模型大小的增加会导致性能下降。我们通过重新排列 Transformer 层中的层归一化和残差连接来克服这一挑战，并展示了这种改动使得开发集上的下游任务结果随着模型大小的增加而单调改进。此外，我们还展示了我们的模型在 WikiText 103、LAMBADA 的完形填空预测准确率和阅读理解 RACE 数据集上达到了测试集的最新水平 (SOTA) 结果。

In summary, our contributions are as follows:

综上所述，我们的贡献如下：

• We implement a simple and efficient model parallel approach by making only a few targeted modifications to an existing PyTorch transformer implementation. • We perform an in-depth empirical analysis of our model and data parallel technique and demonstrate up to $76%$ scaling efficiency using 512 GPUs.

• 我们通过仅对现有的 PyTorch Transformer 实现进行少量有针对性的修改，实现了一个简单且高效的大语言模型并行方法。
• 我们对我们的大语言模型和数据并行技术进行了深入的实证分析，并展示了使用 512 个 GPU 高达 76% 的扩展效率。

2. Background and Challenges

2. 背景与挑战

2.1. Neural Language Model Pre training

2.1. 神经语言模型预训练 (Neural Language Model Pre training)

Pretrained language models have become an indispensable part of NLP researchers’ toolkits. Leveraging large corpus pre training to learn robust neural representations of language is an active area of research that has spanned the past decade. Early examples of pre training and transferring neural representations of language demonstrated that pretrained word embedding tables improve downstream task results compared to word embedding tables learned from scratch (Mikolov et al., 2013; Pennington et al., 2014; Turian et al., 2010). Later work advanced research in this area by learning and transferring neural models that capture contextual representations of words (Melamud et al., 2016; McCann et al., 2017; Peters et al., 2018; Radford et al., 2017; 2019). Recent parallel work (Rama chandra n et al., 2016; Howard & Ruder, 2018; Radford et al., 2018; Devlin et al., 2018; Liu et al., 2019b; Dai et al., 2019; Yang et al., 2019; Liu et al., 2019a; Lan et al., 2019) further builds upon these ideas by not just transferring the language model to extract contextual word representations, but by also finetuning the language model in an end to end fashion on downstream tasks. Through these works, the state of the art has advanced from transferring just word embedding tables to transferring entire multi-billion parameter language models. This progression of methods has necessitated the need for hardware, systems techniques, and frameworks that are able to operate efficiently at scale and satisfy increasing computational needs. Our work aims to provide the tools necessary to take another step forward in this trend.

预训练语言模型已成为自然语言处理 (NLP) 研究人员工具包中不可或缺的一部分。利用大规模语料库预训练来学习语言的鲁棒神经表示是过去十年中一个活跃的研究领域。早期的预训练和迁移神经语言表示的例子表明，与从头学习的词嵌入表相比，预训练的词嵌入表可以提高下游任务的结果（Mikolov et al., 2013；Pennington et al., 2014；Turian et al., 2010）。后续的研究通过学习和迁移能够捕捉上下文词表示的神经模型进一步推进了这一领域的研究（Melamud et al., 2016；McCann et al., 2017；Peters et al., 2018；Radford et al., 2017；2019）。近期并行的工作（Ramachandran et al., 2016；Howard & Ruder, 2018；Radford et al., 2018；Devlin et al., 2018；Liu et al., 2019b；Dai et al., 2019；Yang et al., 2019；Liu et al., 2019a；Lan et al., 2019）进一步发展了这些思想，不仅通过迁移语言模型来提取上下文词表示，还通过在下游任务上以端到端的方式微调语言模型。通过这些工作，最先进水平已经从仅迁移词嵌入表发展到迁移整个数十亿参数的语言模型。这一方法的进步使得对硬件、系统技术和框架的需求变得更加迫切，这些硬件、系统技术和框架必须能够高效地大规模运行，并满足日益增长的计算需求。我们的工作旨在提供必要的工具，以推动这一趋势向前迈出另一步。

2.2. Transformer Language Models and Multi-Head Attention

2.2. Transformer 语言模型和多头注意力机制 (Multi-Head Attention)

Current work in NLP trends towards using transformer models (Vaswani et al., 2017) due to their superior accuracy

当前的自然语言处理 (NLP) 研究趋势是使用 Transformer 模型 (Vaswani et al., 2017)，由于其更高的准确性

图 1: 模型架构示例

在本节中，我们将介绍生成式 AI (Generative AI) 的最新进展。生成式 AI 是指能够根据给定的数据生成新的、类似的数据的技术。这些模型可以用于文本、图像、音频等多种数据类型。

大语言模型（LLM）是生成式 AI 的一个重要分支。它们通过大量的文本数据进行训练，从而能够在各种任务上表现出色，例如零样本和少样本学习。Transformer 模型因其卓越的性能而成为大语言模型的主要架构之一。

表 1: 不同模型的性能对比

模型名称	参数量	性能指标
Model A	1B	85%
Model B	10B	90%
Model C	100B	95%

如上表所示，随着模型参数量的增加，其性能也得到了显著提升。然而，更大的模型通常需要更多的计算资源来进行训练和推理。因此，在实际应用中需要权衡模型大小与性能之间的关系。

Figure 2. Transformer Architecture. Purple blocks correspond to fully connected layers. Each blue block represents a single transformer layer that is replicated N times.

图 2. Transformer 架构。紫色块对应全连接层。每个蓝色块表示一个被复制 N 次的单个 Transformer 层。

and compute efficiency. The original transformer formulation was designed as a machine translation architecture that transforms an input sequence into another output sequence using two parts, an Encoder and Decoder. However, recent work leveraging transformers for language modeling such as BERT (Devlin et al., 2018) and GPT-2 (Radford et al., 2019) use only the Encoder or Decoder depending on their needs. This work explores both a decoder architecture, GPT-2, and an encoder architecture, BERT.

并计算效率。原始的 Transformer 架构被设计为一种机器翻译架构，它使用两个部分（Encoder 和 Decoder）将输入序列转换为另一个输出序列。然而，最近的工作利用 Transformer 进行语言建模，例如 BERT (Devlin et al., 2018) 和 GPT-2 (Radford et al., 2019)，根据需求仅使用 Encoder 或 Decoder。本研究探讨了两种架构：解码器架构 GPT-2 和 编码器架构 BERT。

Figure 2 shows a schematic diagram of the model we used. We refer the reader to prior work for a detailed description of the model architecture (Vaswani et al., 2017; Devlin et al., 2018; Radford et al., 2019). It is worthwhile to mention that both GPT-2 and BERT use GeLU (Hendrycks & Gimpel, 2016) nonlinear i ties and layer normalization (Ba et al., 2016) to the input of the multi-head attention and feed forward layers, whereas the original transformer (Vaswani et al., 2017) uses ReLU nonlinear i ties and applies layer normalization to outputs.

图 2: 显示了我们使用的模型的示意图。我们参考先前的工作以获取模型架构的详细描述 (Vaswani 等, 2017; Devlin 等, 2018; Radford 等, 2019)。值得一提的是，GPT-2 和 BERT 使用 GeLU (Hendrycks & Gimpel, 2016) 非线性关系和层归一化 (Ba 等, 2016) 应用于多头注意力和前馈层的输入，而原始的 Transformer (Vaswani 等, 2017) 使用 ReLU 非线性关系，并对输出应用层归一化。

2.3. Data and Model Parallelism in Deep Learning

2.3. 深度学习中的数据并行和模型并行 (Data and Model Parallelism in Deep Learning)

There are two central paradigms for scaling out deep neural network training to numerous hardware accelerators: data parallelism (Valiant, 1990) where a training minibatch is split across multiple workers, and model parallelism in which the memory usage and computation of a model is distributed across multiple workers. By increasing the minibatch size proportionally to the number of available workers (i.e. weak scaling), one observes near linear scaling in training data throughput. However, large batch training introduces complications into the optimization process that can result in reduced accuracy or longer time to convergence, offsetting the benefit of increased training throughput (Keskar et al., 2017). Further research (Goyal et al., 2017; You et al., 2017; 2019) has developed techniques to mitigate these effects and drive down the training time of large neural networks. To scale out training even further, parallel work (Chen et al., 2016) has combined data parallelism with activation check pointing: re computing activation s in the backward pass without storing them in the forward pass to reduce memory requirements.

有两种主要的范式可以将深度神经网络训练扩展到众多硬件加速器：数据并行（Valiant, 1990），其中训练小批量被分割到多个工作节点上；以及模型并行，其中模型的内存使用和计算分布在多个工作节点上。通过将小批量大小按可用工作节点的数量成比例增加（即弱扩展），可以观察到训练数据吞吐量接近线性扩展。然而，大批次训练会给优化过程引入复杂性，可能导致准确率降低或收敛时间延长，从而抵消了增加训练吞吐量的好处 (Keskar et al., 2017)。进一步的研究 (Goyal et al., 2017; You et al., 2017; 2019) 开发了减轻这些影响的技术，并缩短了大型神经网络的训练时间。为了进一步扩展训练，平行工作 (Chen et al., 2016) 将数据并行与激活检查点结合：在前向传播中不存储激活值而在反向传播中重新计算它们，以减少内存需求。

However, these techniques have one fundamental limitation in the problem size they can tackle: the model must fit entirely on one worker. With language models of increasing size and complexity like BERT and GPT-2, neural networks have approached the memory capacity of modern hardware accelerators. One solution to this problem is to employ parameter sharing to reduce the memory footprint of the model (Lan et al., 2019), but this limits the overall capacity of the model. Our approach is to utilize model parallelism to split the model across multiple accelerators. This not only alleviates the memory pressure, but also increases the amount of parallelism independently of the microbatch size.

然而，这些技术在可以处理的问题规模上有一个根本的限制：模型必须完全放在一个工作节点上。随着像 BERT 和 GPT-2 这样的大语言模型的规模和复杂性不断增加，神经网络已经接近现代硬件加速器的内存容量。解决这个问题的一个方法是采用参数共享以减少模型的内存占用 (Lan et al., 2019)，但这限制了模型的整体容量。我们的方法是利用模型并行性将模型拆分到多个加速器上。这不仅缓解了内存压力，还增加了与微批次大小无关的并行度。

Within model parallelism, there are two further paradigms: layer-wise pipeline parallelism, and more general distributed tensor computation. In pipeline model parallelism, groups of operations are performed on one device before the outputs are passed to the next device in the pipeline where a different group of operations are performed. Some approaches (Harlap et al., 2018; Chen et al., 2018) use a parameter server (Li et al., 2014) in conjunction with pipeline parallelism. However these suffer from inconsistency issues. The GPipe framework for TensorFlow (Huang et al., 2018) overcomes this inconsistency issue by using synchronous gradient decent. This approach requires additional logic to handle the efficient pipelining of these communication and computation operations, and suffers from pipeline bubbles that reduce efficiency, or changes to the optimizer itself which impact accuracy.

在模型并行性内，有两种进一步的范式：按层管道并行性 (layer-wise pipeline parallelism) 和更通用的分布式张量计算 (distributed tensor computation)。在管道模型并行性中，一组操作在一个设备上执行后，输出会被传递到管道中的下一个设备，在那里执行不同的操作组。一些方法 (Harlap et al., 2018; Chen et al., 2018) 将参数服务器 (Li et al., 2014) 与管道并行性结合使用。然而，这些方法存在一致性问题。GPipe框架（用于 TensorFlow 的 Huang et al., 2018）通过使用同步梯度下降克服了这一不一致问题。这种方法需要额外的逻辑来处理这些通信和计算操作的有效管道化，并且会受到管道气泡的影响，从而降低效率，或者对优化器本身进行更改，影响准确性。

Distributed tensor computation is an orthogonal and more general approach that partitions a tensor operation across multiple devices to accelerate computation or increase model size. FlexFlow (Jia et al., 2018), a deep learning framework orchestrating such parallel computation, provides a method to pick the best parallel iz ation strategy. Recently, Mesh-TensorFlow (Shazeer et al., 2018) introduced a language for specifying a general class of distributed tensor computations in TensorFlow (Abadi et al., 2015). The parallel dimensions are specified in the language by the end user and the resulting graph is compiled with proper collective primitives. We utilize similar insights to those leveraged in Mesh-TensorFlow and exploit parallelism in computing the transformer’s attention heads to parallel ize our transformer model. However, rather than implementing a framework and compiler for model parallelism, we make only a few targeted modifications to existing PyTorch transformer implementations. Our approach is simple, does not require any new compiler or code re-writing, and can be fully implemented by inserting a few simple primitives, as described in the next section.

分布式张量计算是一种正交且更通用的方法，它将张量操作划分到多个设备上以加速计算或增加模型规模。FlexFlow (Jia et al., 2018)，一个协调此类并行计算的深度学习框架，提供了一种选择最佳并行化策略的方法。最近，Mesh-TensorFlow (Shazeer et al., 2018) 引入了一种语言，用于在 TensorFlow (Abadi et al., 2015) 中指定一类通用的分布式张量计算。并行维度由最终用户在该语言中指定，并且生成的图使用适当的集体原语进行编译。我们利用了与 Mesh-TensorFlow 类似的见解，并利用并行性来计算 Transformer 的注意力头，以并行化我们的 Transformer 模型。然而，我们并没有实现一个用于模型并行性的框架和编译器，而只是对现有的 PyTorch Transformer 实现进行了几处有针对性的修改。我们的方法简单，不需要任何新的编译器或代码重写，并且可以通过插入几个简单的原语来完全实现，如下一节所述。

3. Model Parallel Transformers

3. 模型并行 Transformer (Model Parallel Transformers)

We take advantage of the structure of transformer networks to create a simple model parallel implementation by adding a few synchronization primitives. A transformer layer consists of a self attention block followed by a two-layer, multi-layer perceptron (MLP) as shown in Figure 2. We introduce model parallelism in both of these blocks separately.

我们利用 Transformer 网络的结构，通过添加一些同步原语来创建一个简单的模型并行实现。Transformer 层由一个自注意力模块后接一个两层的多层感知器 (MLP) 组成，如图 2 所示。我们在这两个模块中分别引入了模型并行性。

图 2:

We start by detailing the MLP block. The first part of the block is a GEMM followed by a GeLU non linearity:

我们首先详细说明 MLP 块。块的第一部分是一个 GEMM，后面跟着一个 GeLU 非线性：

$$
Y=\operatorname{GeLU}(X A)
$$

$$
Y=\operatorname{GeLU}(X A)
$$

生成式 AI (Generative AI) 模型中常用的激活函数之一是 GeLU。这里公式表示输出 Y 是输入 X 与权重矩阵 A 相乘后的 GeLU 激活结果。

One option to parallel ize the GEMM is to split the weight matrix $A$ along its rows and input $X$ along its columns as:

将 GEMM 并行化的一个方法是沿着矩阵 $A$ 的行和输入 $X$ 的列进行分割，如下所示：

$$
X=[X_{1},X_{2}],;A=\left[^{A_{1}}\right].
$$

$$
X=[X_{1},X_{2}] ， ; A=\left[^{A_{1}}\right] 。
$$

This partitioning will result in $Y\ =\ \operatorname{GeLU}(X_{1}A_{1}\ +$ $X_{2}A_{2})$ . Since GeLU is a nonlinear function, $\mathrm{GeLU}(X_{1}A_{1}+$ $X_{2}A_{2})\neq\operatorname{GeLU}(X_{1}A_{1})+\operatorname{GeLU}(X_{2}A_{2})$ and this approach will require a synchronization point before the GeLU function.

这种分区将导致 $Y\ =\ \operatorname{GeLU}(X_{1}A_{1}\ +$ $X_{2}A_{2})$ 。由于 GeLU 是一个非线性函数，$\mathrm{GeLU}(X_{1}A_{1}+$ $X_{2}A_{2})\neq\operatorname{GeLU}(X_{1}A_{1})+\operatorname{GeLU}(X_{2}A_{2})$ ，因此这种方法在 GeLU 函数之前需要一个同步点。

Another option is to split $A$ along its columns $A=[A_{1},A_{2}]$ . This partitioning allows the GeLU non linearity to be independently applied to the output of each partitioned GEMM:

另一种选择是沿着列分割 $A$ ，即 $A=[A_{1}, A_{2}]$ 。这种分区允许 GeLU 非线性独立地应用于每个分区的 GEMM 输出：

$$
[Y_{1},Y_{2}]=[\operatorname{GeLU}(X A_{1}),\operatorname{GeLU}(X A_{2})]
$$

$$
[Y_{1}, Y_{2}] = [GeLU(X A_{1}), GeLU(X A_{2})]
$$

This is advantageous as it removes a synchronization point. Hence, we partition the first GEMM in this column parallel fashion and split the second GEMM along its rows so it takes the output of the GeLU layer directly without requiring any communication as shown in Figure 3a. The output of the second GEMM is then reduced across the GPUs before passing the output to the dropout layer. This approach splits both GEMMs in the MLP block across GPUs and requires only a single all-reduce operation in the forward pass $(g$ operator) and a single all-reduce in the backward pass $f$ operator). These two operators are conjugates of each other and can be implemented in PyTorch with only a few lines of code. As an example, the implementation of the $f$ operator is provided below:

这样做有一个优点，即消除了一个同步点。因此，我们将本列中的第一个 GEMM 以并行方式分区，并沿其行分割第二个 GEMM，使其可以直接接收 GeLU 层的输出而无需任何通信，如图 3a 所示。第二个 GEMM 的输出然后在传递给 dropout 层之前在 GPUs 之间进行规约。这种方法将 MLP 块中的两个 GEMM 分布在 GPUs 上，并且只需要在前向传播中进行一次 all-reduce 操作（$g$ 操作符）和在反向传播中进行一次 all-reduce 操作（$f$ 操作符）。这两个操作符是彼此的共轭，并且可以在 PyTorch 中用几行代码实现。例如，下面提供了 $f$ 操作符的实现：

图 3a:

$(g$ 操作符) 和一个反向传播中的单一 all-reduce ($f$ 操作符)。这些操作符是彼此的共轭，并且可以在 PyTorch 中用几行代码实现。例如，下面提供了 $f$ 操作符的实现：

（注意：原文中图 3a 的具体位置和内容未给出，翻译时请根据实际情况调整）

lass f(torch.autograd.Function): def forward(ctx, x): return x def backward(ctx, gradient): all_reduce(gradient) return gradient

类 f (torch.autograd.Function):
def forward (ctx, x):
返回 x

def backward (ctx, gradient):
all_reduce (gradient)
返回 gradient

Code 1. Implementation of $f$ operator. $g$ is similar to $f$ with identity in the backward and all-reduce in the forward functions.

代码 1. $f$ 运算符的实现。$g$ 与 $f$ 类似，在反向传播中使用恒等函数，在前向传播中使用全部归约函数。

Figure 3. Blocks of Transformer with Model Parallelism. $f$ and $g$ are conjugate. $f$ is an identity operator in the forward pass and all reduce in the backward pass while $g$ is an all reduce in the forward pass and identity in the backward pass. (b) Self-Attention

图 3: 带有模型并行的 Transformer 模块。$f$ 和 $g$ 是共轭的。$f$ 在前向传播中是恒等运算符，在反向传播中是 all reduce，而 $g$ 在前向传播中是 all reduce，在反向传播中是恒等运算符。（b）自注意力机制

As shown in Figure 3b, for the self attention block we exploit inherent parallelism in the multihead attention operation, partitioning the GEMMs associated with key $(K)$ , query $(Q)$ , and value $(V)$ in a column parallel fashion such that the matrix multiply corresponding to each attention head is done locally on one GPU. This allows us to split per attention head parameters and workload across the GPUs, and doesnt require any immediate communication to complete the self-attention. The subsequent GEMM from the output linear layer (after self attention) is parallel i zed along its rows and takes the output of the parallel attention layer directly, without requiring communication between the GPUs. This approach for both the MLP and self attention layer fuses groups of two GEMMs, eliminates a synchronization point in between, and results in better scaling. This enables us to perform all GEMMs in a simple transformer layer using only two all-reduces in the forward path and two in the backward path (see Figure 4).

如图 3b 所示，对于自注意力模块，我们利用多头注意力操作中的固有并行性，以列并行的方式对与键 (K)、查询 (Q) 和值 (V) 相关的 GEMM 进行分区，使得每个注意力头对应的矩阵乘法在单个 GPU 上本地完成。这使我们能够将每个注意力头的参数和工作负载分布在多个 GPU 上，并且不需要立即通信来完成自注意力操作。自注意力之后的输出线性层的 GEMM 沿其行进行并行化，并直接接收并行注意力层的输出，而无需在 GPU 之间进行通信。这种方法既适用于 MLP 层也适用于自注意力层，它融合了两组 GEMM，消除了中间的同步点，并实现了更好的扩展性。这使我们能够在简单的 Transformer 层中仅使用两个前向路径和两个反向路径的全归约操作来完成所有 GEMM（见图 4）。

The transformer language model has an output embedding with the dimension of hidden-size $(H)$ times vocabularysize $(v)$ . Since the vocabulary size is on the order of tens of thousands of tokens for modern language models (for example, GPT-2 used a vocabulary size of 50,257), it is beneficial to parallel ize the output embedding GEMM. However, in transformer language models, the output embedding layer shares weights with the input embedding, requiring modifications to both. We parallel ize the input embedding weight matrix $E_{H\times v}$ along the vocabulary dimension $E=[E_{1},E_{2}]$ (column-wise). Since each partition now only contains a portion of the embedding table, an all-reduce $\mathit{\Delta}{g}$ operator) is required after the input embedding. For the output embedding, one approach is to perform the parallel GEMM $[Y{1},Y_{2}]=[X E_{1},X E_{2}]$ to obtain the logits, add an all-gather $Y=\mathrm{all-gather}([Y_{1},Y_{2}])$ , and send the results to the cross-entropy loss function. However, for this case, the all-gather will communicate $b\times s\times v$ elements $b$ is the batch-size and $s$ is the sequence length) which is huge due to vocabulary size being large. To reduce the communication size, we fuse the output of the parallel GEMM $[Y_{1},Y_{2}]$ with the cross entropy loss which reduces the dimension to $b\times s$ . Communicating scalar losses instead of logits is a huge reduction in communication that improves the efficiency of our model parallel approach.

Transformer 语言模型的输出嵌入维度为隐藏层大小 $(H)$ 乘以词汇量大小 $(v)$。由于现代语言模型的词汇量通常为数万个 Token（例如，GPT-2 使用了 50,257 的词汇量），因此并行化输出嵌入 GEMM 是有益的。然而，在 Transformer 语言模型中，输出嵌入层与输入嵌入层共享权重，这需要对两者进行修改。我们沿词汇维度 $E=[E_{1},E_{2}]$（按列）并行化输入嵌入权重矩阵 $E_{H\times v}$。由于每个分区现在只包含嵌入表的一部分，因此在输入嵌入之后需要一个全归约 $\mathit{\Delta}{g}$ 操作。对于输出嵌入，一种方法是执行并行 GEMM $[Y{1},Y_{2}]=[X E_{1},X E_{2}]$ 以获得 logits，添加一个全规约 $Y=\mathrm{all-gather}([Y_{1},Y_{2}])$，并将结果发送到交叉熵损失函数。然而，在这种情况下，全规约将通信 $b\times s\times v$ 元素（$b$ 是批量大小，$s$ 是序列长度），由于词汇量较大，这将是一个巨大的通信量。为了减少通信量，我们将并行 GEMM 的输出 $[Y_{1},Y_{2}]$ 与交叉熵损失融合，从而将维度减少到 $b\times s$。通过通信标量损失而不是 logits，可以大幅减少通信量，提高我们模型并行方法的效率。

Figure 4. Communication operations in a transformer layer. There are 4 total communication operations in the forward and backward pass of a single model parallel transformer layer.

图 4: Transformer 层中的通信操作。在单个模型并行 Transformer 层的前向和后向传递中总共有 4 次通信操作。

Much of our model parallel approach can be characterized as techniques aimed at reducing communication and keeping the GPUs compute bound. Rather than having one GPU compute part of the dropout, layer normalization, or residual connections and broadcast the results to other GPUs, we choose to duplicate the computation across GPUs. Specifically, we maintain duplicate copies of layer normalization parameters on each GPU, and take the output of the model parallel region and run dropout and residual connection on these tensors before feeding them as input to the next model parallel regions. To optimize the model we allow each model parallel worker to optimize its own set of parameters. Since all values are either local to or duplicated on a GPU, there is no need for communicating updated parameter values in this formulation.

我们模型并行方法的大部分可以描述为减少通信和保持 GPU 计算绑定的技术。我们选择在多个 GPU 上复制计算，而不是让一个 GPU 计算 dropout、层归一化或残差连接的部分结果并广播给其他 GPU。具体来说，我们在每个 GPU 上维护层归一化参数的副本，并对模型并行区域的输出执行 dropout 和残差连接操作，然后再将这些张量作为输入传递给下一个模型并行区域。为了优化模型，我们允许每个模型并行工作器优化其自己的参数集。由于所有值要么是本地的，要么是在 GPU 上复制的，在这种情况下不需要通信更新的参数值。

We present further details about the hybrid model and data parallelism and handling random number generation in Appendix B for reference. In summary, our approach as described above is simple to implement, requiring only a few extra all-reduce operations added to the forward and backward pass. It does not require a compiler, and is orthogonal and complementary to the pipeline model parallelism advocated by approaches such as (Huang et al., 2018).

我们在附录 B 中进一步详细介绍了混合模型和数据并行性以及随机数生成的处理方法，供参考。总之，如上所述，我们的方法实现简单，只需要在前向和后向传播中添加几个额外的 all-reduce 操作。它不需要编译器，并且与 (Huang et al., 2018) 等方法所提倡的管道模型并行性正交且互补。

4. Setup

4. 设置

Pretrained language understanding models are central tasks in natural language processing and language understanding. There are several formulations of language modeling. In this work we focus on GPT-2 (Radford et al., 2019), a leftto-right generative transformer based language model, and BERT (Devlin et al., 2018), a bi-directional transformer model based on language model masking. We explain our configurations for these models in the following section and refer to the original papers for more details.

预训练的语言理解模型是自然语言处理和语言理解的核心任务。语言模型有几种不同的形式。在本工作中，我们专注于 GPT-2 (Radford 等, 2019)，一种从左到右的生成式 Transformer 语言模型，以及 BERT (Devlin 等, 2018)，一种基于语言模型掩码的双向 Transformer 模型。我们在下一节中解释这些模型的配置，并参考原始论文以获取更多详细信息。

4.1. Training Dataset

4.1. 训练数据集

To collect a large diverse training set with longterm dependencies we aggregate several of the largest language modeling datasets. We create an aggregate dataset consisting of Wikipedia (Devlin et al., 2018), CC-Stories (Trinh & Le, 2018), RealNews (Zellers et al., 2019), and OpenWebtext (Radford et al., 2019). To avoid training set leakage into our downstream tasks we remove the Wikipedia articles present in the WikiText 103 test set (Merity et al., 2016). We also remove unnecessary newlines from the CC-Stories corpus introduced by preprocessing artifacts. For BERT models we include Books Corpus (Zhu et al., 2015) in the training dataset, however, this dataset is excluded for GPT-2 trainings as it overlaps with LAMBADA task.

为了收集一个包含长期依赖关系的大型多样化训练集，我们汇总了几个最大的语言建模数据集。我们创建了一个聚合数据集，包括 Wikipedia (Devlin et al., 2018)，CC-Stories (Trinh & Le, 2018)，RealNews (Zellers et al., 2019)，和 OpenWebtext (Radford et al., 2019)。为了避免训练集泄漏到我们的下游任务中，我们移除了 WikiText 103 测试集 (Merity et al., 2016) 中存在的 Wikipedia 文章。我们还从 CC-Stories 语料库中移除了由预处理引入的不必要的换行符。对于 BERT 模型，我们在训练数据集中包含了 Books Corpus (Zhu et al., 2015)，然而，由于与 LAMBADA 任务重叠，这个数据集在 GPT-2 训练中被排除。

We combined all the datasets and then filtered out all the documents with content length less than 128 tokens from the aggregated dataset. Since similar content might appear multiple times in the aggregated datasets, we used localitysensitive hashing (LSH) to de duplicate content with a jaccard similarity greater than 0.7. The resulting aggregate corpus contains 174 GB of de duplicated text.

我们将所有数据集合并，然后从聚合数据集中过滤掉内容长度少于 128 个 Token 的所有文档。由于相似内容可能在聚合数据集中多次出现，我们使用局部敏感哈希 (LSH) 来去重内容，Jaccard 相似度大于 0.7 的内容会被去重。最终的聚合语料库包含 174 GB 的去重文本。

4.2. Training Optimization and Hyper parameters

4.2. 训练优化和超参数

To train our models efficiently we utilize mixed precision training with dynamic loss scaling to take advantage of the V100’s Tensor Cores (Mic ike vici us et al., 2017; NVIDIA, 2018). We start by initializing our weights $W$ with a simple normal distribution $W\sim\mathcal{N}(0,0.02)$ . We then scale weights immediately before residual layers by√21N where N is the number of transformer layers comprised of self attention and MLP blocks. For our optimizer we utilize Adam (Kingma & Ba, 2014) with weight decay (Loshchilov & Hutter, 2019) $\lambda=0.01$ . Additionally, we use global gradient norm clipping of 1.0 to improve the stability of training large models. In all cases, a dropout of 0.1 is used. Lastly, to better manage our memory footprint we utilize activation check pointing (Chen et al., 2016) after every transformer layer.

为了高效训练我们的模型，我们利用混合精度训练和动态损失缩放来充分利用 V100 的 Tensor Cores (Mic ike vici us et al., 2017; NVIDIA, 2018)。我们首先使用简单的正态分布 $W\sim\mathcal{N}(0,0.02)$ 初始化权重 $W$ 。然后在残差层之前立即将权重按 $\sqrt{\frac{2}{N}}$ 缩放，其中 N 是由自注意力和 MLP 模块组成的 Transformer 层的数量。对于优化器，我们使用带有权重衰减 (Loshchilov & Hutter, 2019) 的 Adam (Kingma & Ba, 2014)，$\lambda=0.01$。此外，我们使用全局梯度范数裁剪 1.0 来提高训练大模型的稳定性。在所有情况下，都使用 0.1 的 dropout。最后，为了更好地管理内存占用，我们在每个 Transformer 层之后使用激活检查点 (Chen et al., 2016)。

For GPT-2 models, all training is performed with sequences of 1024 subword units at a batch size of 512 for $300\mathrm{k}$ iterations. Our learning rate of $1.5\mathrm{e}{-4}$ utilizes a warmup period of $3\mathbf{k}$ iterations before following a single cycle cosine decay over the remaining 297k iterations. We stop the decay at a minimum learning rate of 1e-5.

对于 GPT-2 模型，所有训练均在序列长度为 1024 的子词单元上进行，批量大小为 512，迭代次数为 300k。我们的学习率 1.5e-4 采用 3k 迭代的预热期，然后在剩余的 297k 迭代中遵循单周期余弦衰减。我们在最小学习率为 1e-5 时停止衰减。

For BERT models, we largely follow the training process described in (Lan et al., 2019). We use the original BERT dictionary with vocab size of 30,522. In addition, we replace the next sentence prediction head with sentence order prediction as suggested by (Lan et al., 2019) and use whole word n-gram masking of (Joshi et al., 2019). For all cases, we set the batch size to 1024 and use a learning rate of 1.0e4 warmed up over 10,000 iterations and decayed linearly over 2 million iterations. Other training parameters are kept the same as (Devlin et al., 2018).

对于 BERT 模型，我们主要遵循 (Lan et al., 2019) 中描述的训练过程。我们使用原始的 BERT 词典，词汇量为 30,522。此外，我们根据 (Lan et al., 2019) 的建议，用句子顺序预测替换下一个句子预测头，并采用 (Joshi et al., 2019) 提出的整词 n-gram 掩码。在所有情况下，我们将批量大小设置为 1024，并使用 1.0e4 的学习率，在 10,000 次迭代中逐渐升温，并在 2 百万次迭代中线性衰减。其他训练参数与 (Devlin et al., 2018) 保持一致。

5. Experiments

5. 实验

All of our experiments use up to 32 DGX-2H servers (a total of 512 Tesla V100 SXM3 32GB GPUs). Our infrastructure is optimized for multi-node deep learning applications, with 300 GB/sec bandwidth between GPUs inside a server via NVSwitch and 100 GB/sec of interconnect bandwidth between servers using 8 InfiniBand adapters per server.

我们所有的实验最多使用 32 台 DGX-2H 服务器（总计 512 个 Tesla V100 SXM3 32GB GPU）。我们的基础设施针对多节点深度学习应用进行了优化，服务器内部 GPU 之间的带宽为 300 GB/秒，通过 NVSwitch 实现；服务器之间使用每台服务器 8 个 InfiniBand 适配器，带宽为 100 GB/秒。

5.1. Scaling Analysis

5.1. 扩展分析

规则：
      - 输出中文翻译部分的时候，只保留翻译的标题，不要有任何其他的多余内容，不要重复，不要解释。
      - 不要输出与英文内容无关的内容。
      - 翻译时要保留原始段落格式，以及保留术语，例如 FLAC，JPEG 等。保留公司缩写，例如 Microsoft, Amazon, OpenAI 等。
      - 人名不翻译
      - 同时要保留引用的论文，例如 [20] 这样的引用。
      - 对于 Figure 和 Table，翻译的同时保留原有格式，例如：“Figure 1: ”翻译为“图 1: ”，“Table 1: ”翻译为：“表 1: ”。
      - 全角括号换成半角括号，并在左括号前面加半角空格，右括号后面加半角空格。
      - 在翻译专业术语时，第一次出现时要在括号里面写上英文原文，例如：“生成式 AI (Generative AI)”，之后就可以只写中文了。
      - 以下是常见的 AI 相关术语词汇对应表（English -> 中文）：
      * Transformer -> Transformer
      * Token -> Token
      * LLM/Large Language Model -> 大语言模型
      * Zero-shot -> 零样本
      * Few-shot -> 少样本
      * AI Agent -> AI智能体
      * AGI -> 通用人工智能
      * Python -> Python语言

      策略：

      分三步进行翻译工作：
      1. 不翻译无法识别的特殊字符和公式，原样返回
      2. 将HTML表格格式转换成Markdown表格格式
      3. 根据英文内容翻译成符合中文表达习惯的内容，不要遗漏任何信息

      最终只返回Markdown格式的翻译结果，不要回复无关内容。

To test the s cal ability of our implementation, we consider GPT-2 models with four sets of parameters detailed in Table 1. To have consistent GEMM sizes in the self attention layer, the hidden size per attention head is kept constant at 96 while the number of heads and layers are varied to obtain configurations ranging from 1 billion to 8 billion parameters. The configuration with 1.2 billion parameters fits on a single GPU whereas the 8 billion parameter model requires 8-way model parallelism (8 GPUs). The original vocabulary size was 50,257, however, to have efficient GEMMs for the logit layer, it is beneficial for the per-GPU vocabulary size to be a multiple of 128. Since we study up to 8-way model parallelism, we pad the vocabulary such that it is divisible by $128\times8=1024$ , resulting in a padded vocabulary size of 51,200. We study both model and model+data parallel scaling. For the model parallel scaling, a fixed batch size of 8 is used across all configurations. Data parallel scaling is necessary for training many state of the art models which typically use a much larger global batch size. To this end, for the model+data parallel cases we fix the global batch size to 512 for all experiments which corresponds to 64-way data parallelism.

为了测试我们实现的可扩展性，我们考虑了具有四组参数的 GPT-2 模型，详见表 1。为了在自注意力层中保持一致的 GEMM 大小，每个注意力头的隐藏大小保持不变为 96，而头数和层数则有所不同，以获得从 10 亿到 80 亿参数的配置。具有 12 亿参数的配置可以适应单个 GPU，而 80 亿参数的模型需要 8 路模型并行（8 个 GPU）。原始词汇量为 50,257，然而，为了使 logits 层的 GEMM 更高效，每个 GPU 的词汇量最好是 128 的倍数。由于我们研究最多 8 路模型并行，我们将词汇量填充至能被 $128\times8=1024$ 整除，最终填充后的词汇量为 51,200。我们研究了模型并行和模型+数据并行的扩展性。对于模型并行扩展，所有配置使用固定的批量大小为 8。数据并行扩展对于训练许多最先进的模型是必要的，这些模型通常使用更大的全局批量大小。为此，在模型+数据并行的情况下，我们将所有实验的全局批量大小固定为 512，这对应于 64 路数据并行。

5.1.1. MODEL AND DATA PARALLELISM

5.1.1. 模型并行和数据并行 (MODEL AND DATA PARALLELISM)

Throughout this section, we will showcase weak scaling with respect to the model parameters for both model parallel and model+data parallel cases. Weak scaling is typically done by scaling the batch-size, however, this approach does not address training large models that do not fit on a single GPU and it leads to training convergence degradation for large batch sizes. In contrast, here we use weak scaling to train larger models that were not possible otherwise. The baseline for all the scaling numbers is the first configuration (1.2 billion parameters) in Table 1 running on a single GPU. This is a strong baseline as it achieves 39 TeraFLOPS during the overall training process, which is $30%$ of the theoretical peak FLOPS for a single GPU in a DGX-2H server.

在整个部分中，我们将展示关于模型参数的弱扩展性，涵盖模型并行和模型+数据并行两种情况。弱扩展性通常通过扩展批量大小来实现，但这种方法并不能解决无法在单个 GPU 上容纳的大模型训练问题，并且会导致大批量大小下的训练收敛性下降。相比之下，这里我们使用弱扩展性来训练那些原本不可能训练的更大模型。所有扩展数据的基准是表 1 中的第一个配置（12 亿参数），运行在单个 GPU 上。这是一个强大的基准，因为它在整个训练过程中实现了 39 TeraFLOPS，这是 DGX-2H 服务器中单个 GPU 理论峰值 FLOPS 的 (30%) 。

Table 1. Parameters used for scaling studies. Hidden size per attention head is kept constant at 96.

表 1: 用于扩展研究的参数。每个注意力头的隐藏大小保持恒定为 96。

隐藏大小	注意力头数	层数	参数数量 (十亿)	模型并行 GPU	模型 + 数据并行 GPU
1536	16	40	1.2	1	64
1920	20	54	2.5	2	128
2304	24	64	4.2	4	256
3072	32	72	8.3	8	512

Figure 5. Model and model $^+$ data parallel weak scaling efficiency as a function of the number of GPUs.

图 5: 模型和模型 $^+$ 数据并行弱扩展效率与 GPU 数量的关系。

Figure 5 shows scaling values for both model and model+data parallelism. We observe excellent scaling numbers in both settings. For example, the 8.3 billion parameters case with 8-way (8 GPU) model parallelism achieves $77%$ of linear scaling. Model+data parallelism requires further communication of gradients and as a result the scaling numbers drop slightly. However, even for the largest configuration (8.3 billion parameters) running on 512 GPUs, we achieve $74%$ scaling relative to linear scaling of the strong single GPU baseline configuration (1.2 billion parameters). Further scaling analysis is provided in Appendix D

图 5: 显示了模型并行和模型+数据并行的扩展值。我们在两种设置中都观察到了出色的扩展数字。例如，83亿参数的情况在8路 (8 GPU) 模型并行下实现了 77% 的线性扩展。模型+数据并行需要进一步的梯度通信，因此扩展数字略有下降。然而，即使对于最大的配置（83亿参数）在512个GPU上运行，我们仍然实现了相对于单个GPU基准配置（12亿参数）线性扩展的 74% 。更多的扩展分析请参见附录D。

5.2. Language Modeling Results Using GPT-2

5.2. 使用 GPT-2 的语言模型结果

规则：
      - 输出中文翻译部分的时候，只保留翻译的标题，不要有任何其他的多余内容，不要重复，不要解释。
      - 不要输出与英文内容无关的内容。
      - 翻译时要保留原始段落格式，以及保留术语，例如 FLAC，JPEG 等。保留公司缩写，例如 Microsoft, Amazon, OpenAI 等。
      - 人名不翻译
      - 同时要保留引用的论文，例如 [20] 这样的引用。
      - 对于 Figure 和 Table，翻译的同时保留原有格式，例如：“Figure 1: ”翻译为“图 1: ”，“Table 1: ”翻译为：“表 1: ”。
      - 全角括号换成半角括号，并在左括号前面加半角空格，右括号后面加半角空格。
      - 在翻译专业术语时，第一次出现时要在括号里面写上英文原文，例如：“生成式 AI (Generative AI)”，之后就可以只写中文了。
      - 以下是常见的 AI 相关术语词汇对应表（English -> 中文）：
      * Transformer -> Transformer
      * Token -> Token
      * LLM/Large Language Model -> 大语言模型
      * Zero-shot -> 零样本
      * Few-shot -> 少样本
      * AI Agent -> AI智能体
      * AGI -> 通用人工智能
      * Python -> Python语言

      策略：

      分三步进行翻译工作：
      1. 不翻译无法识别的特殊字符和公式，原样返回
      2. 将HTML表格格式转换成Markdown表格格式
      3. 根据英文内容翻译成符合中文表达习惯的内容，不要遗漏任何信息

      最终只返回Markdown格式的翻译结果，不要回复无关内容。

To demonstrate that large language models can further advance the state of the art, we consider training GPT-2 models of the sizes and configurations listed in Table 2. The 355M model is equivalent in size and configuration of BERT-Large model (Devlin et al., 2018). The 2.5B model is bigger than the previous largest GPT-2 model, and the 8.3B model is larger than any left-to-right transformer language model ever trained, to the best of our knowledge. To train and evaluate our language models we use the procedure described in section 4. Table 2 also lists the time it takes to advance one epoch which is equivalent to 68,507 iterations. For example, for the 8.3B model on 512 GPUs, each epoch takes around two days. Compared to the configurations used for our scaling studies in Table 1, the 2.5B model is the same, the 8.3B model has 24 attention heads instead of 32, and the $355\mathrm{M}$ is much smaller than any seen previously while still using 64 GPUs to train, leading to the much lower time per epoch.

为了证明大语言模型可以进一步推动技术的发展，我们考虑训练表 2 中列出的大小和配置的 GPT-2 模型。355M 模型在大小和配置上等同于 BERT-Large 模型 (Devlin et al., 2018)。2.5B 模型比之前的最大 GPT-2 模型更大，而 8.3B 模型是我们所知的最大左到右 Transformer 语言模型。为了训练和评估我们的语言模型，我们使用第 4 节中描述的程序。表 2 还列出了推进一个 epoch 所需的时间，这相当于 68,507 次迭代。例如，对于 8.3B 模型，在 512 个 GPU 上，每个 epoch 大约需要两天时间。与表 1 中用于扩展研究的配置相比，2.5B 模型是相同的，8.3B 模型有 24 个注意力头而不是 32 个，而 355M 模型比之前见过的任何模型都小得多，但仍使用 64 个 GPU 进行训练，导致每个 epoch 的时间大大减少。

Table 2. Model configurations used for GPT-2.

表 2. GPT-2 使用的模型配置。

参数数量	层数	隐藏层大小	注意力头数	每个头的隐藏层大小	总 GPU 数量	每轮训练时间
355M	24	1024	16	64	64	(天) 0.86
2.5B	54	1920	20	96	128	2.27
8.3B	72	3072	24	128	512	2.10

Table 3. Zero-shot results. SOTA are from (Khandelwal et al., 2019) for Wikitext 103 and (Radford et al., 2019) for LAMBADA.

表 3: 零样本结果。SOTA 来自 (Khandelwal et al., 2019) 的 Wikitext 103 和 (Radford et al., 2019) 的 LAMBADA。

模型	Wikitext103 困惑度√	LAMBADA 准确率↑
355M 2.5B 8.3B	19.31 12.76	45.18% 61.73%
之前 SOTA	10.81 15.79	66.51% 63.24%

Figure 6 shows validation per peli xi ty as a function of number of iterations. As the model size increases, the validation per peli xi ty decreases and reaches a validation perplexity of 9.27 for the 8.3B model. We report the zero-shot evaluation of the trained models on the LAMBADA and WikiText 103 datasets in Table 3. For more details on evaluation methodology, see Appendix E. We observe the trend that increasing model size also leads to lower perplexity on WikiText 103 and higher cloze accuracy on LAMBADA. Our 8.3B model achieves state of the art perplexity on the WikiText 103 test set at a properly adjusted perplexity of 10.81. At $66.51%$ accuracy, the 8.3B model similarly surpasses prior cloze accuracy results on the LAMBADA task. We have included samples generated from the 8.3 billion parameters model in the Appendix C. Recently researchers from Microsoft in collaboration with NVIDIA trained a 17 billion parameter GPT-2 model called Turing-NLG (Microsoft, 2020) using Megatron and showed that the accuracies further improve as they scale the model, highlighting the value of larger models.

图 6: 显示了验证困惑度随迭代次数的变化情况。随着模型规模的增加，验证困惑度降低，并在 83 亿参数模型中达到 9.27 的验证困惑度。我们在表 3 中报告了训练模型在 LAMBADA 和 WikiText 103 数据集上的零样本评估结果。有关评估方法的更多详细信息，请参见附录 E。我们观察到，随着模型规模的增加，WikiText 103 上的困惑度降低，LAMBADA 上的完形填空准确率提高。我们的 83 亿参数模型在 WikiText 103 测试集上达到了调整后的 10.81 困惑度的最佳水平。在 66.51% 的准确率下，83 亿参数模型同样超越了之前在 LAMBADA 任务上的完形填空准确率结果。我们在附录 C 中包含了从 83 亿参数模型生成的样本。最近，Microsoft 与 NVIDIA 的研究人员合作训练了一个 170 亿参数的 GPT-2 模型，称为 Turing-NLG (Microsoft, 2020)，使用 Megatron 展示了随着模型规模的扩大，准确率进一步提高，突显了更大模型的价值。

To ensure we do not train on any data found in our test sets, we calculate the percentage of test set 8-grams that also appear in our training set as done in previous work (Radford et al., 2019). The WikiText 103 test set has at most

为确保我们不在测试集中出现的数据上进行训练，我们计算测试集中 8-gram 也出现在训练集中的比例，这一做法与之前的工作 (Radford et al., 2019) 相同。WikiText 103 测试集最多

Figure 6. Validation set perplexity. All language models are trained for $300\mathbf{k}$ iterations. Larger language models converge noticeably faster and converge to lower validation perplexities than their smaller counterparts.

图 6. 验证集困惑度。所有大语言模型都训练了 $300\mathbf{k}$ 次迭代。较大的大语言模型收敛速度明显更快，并且收敛到比其较小的对应模型更低的验证困惑度。

Table 4. Model configurations used for BERT.

表 4. BERT 使用的模型配置。

参数数量	层数	隐藏层大小	注意力头数	总 GPU 数量
336M	24	1024	16	128
1.3B	24	2048	32	256
3.9B	48	2560	40	512

$10.8%$ overlap and the LAMBADA test set (Paperno et al., 2016) has at most $1.4%$ overlap. We should note that the WikiText 103 test set has already $9.09%$ overlap with the WikiText 103 training set (Radford et al., 2019). As these are consistent with previous work, we are confident that no documents from our test data are inadvertently included in our training data.

$10.8%$ 重叠，而 LAMBADA 测试集 (Paperno 等, 2016) 的重叠度最多为 $1.4%$ 。我们注意到 WikiText 103 测试集与 WikiText 103 训练集 (Radford 等, 2019) 已经有 $9.09%$ 的重叠。由于这些数据与之前的工作一致，我们有信心确保测试数据中的文档不会意外包含在训练数据中。

5.3. Bi-directional Transformer Results Using BERT

5.3. 双向 Transformer (BERT) 结果 使用 BERT

In this section, we apply our methodology to BERT-style transformer models and study the effect of model scaling on several downstream tasks. Prior work (Lan et al., 2019) found that increasing model size beyond BERT-large with 336M parameters results in unexpected model degradation. To address this degradation, the authors of that work (Lan et al., 2019) introduced parameter sharing and showed that that their models scale much better compared to the original BERT model.

在本节中，我们将方法应用于 BERT 风格的 Transformer 模型，并研究模型扩展对几个下游任务的影响。先前的工作 (Lan et al., 2019) 发现，增加模型规模超过具有 3.36 亿参数的 BERT-large 会导致意外的模型性能下降。为了解决这种性能下降问题，该工作的作者 (Lan et al., 2019) 引入了参数共享，并表明他们的模型相比原始的 BERT 模型有更好的扩展性。

We further investigated this behaviour and empirically demonstrated that rearranging the order of the layer normalization and the residual connections as shown in Figure 7 is critical to enable the scaling of the BERT-style models beyond BERT-Large. The architecture (b) in Figure 7 eliminates instabilities observed using the original BERT architecture in (a) and also has a lower training loss. To the best of our knowledge, we are the first to report such a change enables training larger BERT models.

我们进一步研究了这种行为，并通过实验证明，重新排列层归一化和残差连接的顺序（如图 7 所示）对于使 BERT 风格模型能够扩展到超过 BERT-Large 至关重要。图 7 中的架构 (b) 消除了使用原始 BERT 架构 (a) 观察到的不稳定性，并且训练损失更低。据我们所知，我们是第一个报告这种改变能够训练更大规模的 BERT 模型的人。

图 7:

Table 5. Development set results for MNLI, QQP, SQuAD 1.1 and SQuAD 2.0 and test set results for RACE. The trained tokens represents consumed tokens during model pre training (proportional to batch size times number of iterations) normalized by consumed tokens during model pre training for our 336M model.

表 5: MNLI、QQP、SQuAD 1.1 和 SQuAD 2.0 的开发集结果以及 RACE 的测试集结果。训练的 Token 比例表示在模型预训练期间消耗的 Token 数量（与批次大小和迭代次数成正比），并以我们 336M 模型预训练期间消耗的 Token 数量进行归一化。

模型	训练的 Token 比例	MNLI m/mm 准确率 (开发集)	QQP 准确率 (开发集)	SQuAD 1.1 F1/EM (开发集)	SQuAD 2.0 F1/EM (开发集)	RACE m/h 准确率 (测试集)
RoBERTa (Liu et al., 2019b) ALBERT (Lan et al., 2019)	2	90.2/90.2	92.2	94.6/88.9	89.4/86.5	83.2 (86.5/81.8)
	3	90.8	92.2	94.8/89.3	90.2/87.4	86.5 (89.0/85.5)
XLNet (Yang et al., 2019)	2	90.8/90.8	92.3	95.1/89.7	90.6/87.9	85.4 (88.6/84.0)
Megatron-336M	1	89.7/90.0	92.3	94.2/88.0	88.1/84.8	83.0 (86.9/81.5)
Megatron-1.3B	1	90.9/91.0	92.6	94.9/89.1	90.2/87.1	87.3 (90.4/86.1)
Megatron-3.9B	1	91.4/91.4	92.7	95.5/90.0	91.2/88.5	89.5 (91.8/88.6)
ALBERT 集成 (Lan et al., 2019)				95.5/90.1	91.4/88.9	89.4 (91.2/88.6)
Megatron-3.9B 集成				95.8/90.5	91.7/89.0	90.9 (93.1/90.0)

图 1: 模型架构示例 (Example of Model Architecture)

在本节中，我们介绍了生成式 AI (Generative AI) 的基本概念，并讨论了其在不同领域的应用。生成式 AI 是一种能够创建新内容的技术，它可以通过学习数据的分布来生成与训练数据相似但又全新的样本。这种技术已经在图像生成、文本创作和音乐合成等领域取得了显著进展。

表 1: 不同模型的性能对比

模型名称	参数量	训练时间	测试精度
Transformer	1.5B	3 天	92%
CNN	50M	1 天	88%

通过上述表格可以看出，Transformer 模型在参数量和训练时间上都远超其他模型，但在测试精度方面也表现出了明显的优势。这表明在处理复杂任务时，使用更复杂的模型结构可以带来更好的效果。

Figure 7. Training loss for BERT model using the original architecture (a) and the rearranged architecture (b). Left figure shows the training loss for 336M and 752M BERT model. While the original architecture performs well on the 336M model, the modifications in (b) enable stable training with lower training loss.

图 7. 使用原始架构 (a) 和重新排列的架构 (b) 的 BERT 模型训练损失。左图显示了 336M 和 752M BERT 模型的训练损失。虽然原始架构在 336M 模型上表现良好，但 (b) 中的修改使训练更加稳定，并且训练损失更低。

Using the architecture change in Figure 7(b), we consider three different cases as detailed in Table 4. The 336M model has the same size as BERT-large. The 1.3B is the same as the BERT-xlarge configuration that was previously shown to get worse results than the 336M BERT-large model (Lan et al., 2019). We further scale the BERT model using both larger hidden size as well as more layers to arrive at the 3.9B parameter case. In all cases, the hidden size per attention head is kept constant at 64. 336M and 1.3B models are trained for 2 million iterations while the 3.9B model is trained for 1.5 million iterations and is still training.

使用图 7(b) 中的架构更改，我们考虑了表 4 中详述的三种不同情况。336M 模型的规模与 BERT-large 相同。1.3B 模型与之前展示的 BERT-xlarge 配置相同，后者的结果比 336M 的 BERT-large 模型更差 (Lan et al., 2019)。我们进一步通过增加隐藏层大小和层数来扩展 BERT 模型，以达到 3.9B 参数的情况。在所有情况下，每个注意力头的隐藏层大小保持恒定为 64。336M 和 1.3B 模型训练了 200 万次迭代，而 3.9B 模型训练了 150 万次迭代并且仍在训练中。

On a $3%$ held-out set, 336M, 1.3B, and 3.9B models achieve validation set perplexity of 1.58, 1.30, and 1.16, respectively, a monotonic decrease with the model size. We finetune the trained models on several downstream tasks including MNLI and QQP from the GLUE benchmark (Wang et al., 2019), SQuAD 1.1 and SQuAD 2.0 from the Stanford Question answering dataset (Rajpurkar et al., 2016; 2018), and the reading comprehension RACE dataset (Lai et al., 2017). For finetuning, we follow the same procedure as (Liu et al., 2019b). We first perform hyper parameter tuning on batch size and learning rate. Once we obtain the best values, we report the median development set results over 5 different random seeds for initialization. The hyper parameters used for each model and task are provided in the Appendix A. Table 5 shows the development set results for MNLI, QQP, SQuAD 1.1, and $\mathrm{SQuAD}\ 2.0$ and test set results for RACE. For the test set results of RACE, we first use the development set to find the checkpoint that gives us the median score on the 5 random seeds and we report the results from that checkpoint on the test set. We also report 5-way ensemble results for the development set of SQuAD and test set of RACE. From Table 5 we observe that (a) as the model size increases, the downstream task performance improves in all cases, (b) our 3.9B model establishes state of the art results on the development set compared to other BERT based models, and (c) our 3.9B model achieves both single model as well as ensembled SOTA results on RACE test set.

在 3% 的验证集上，336M、1.3B 和 3.9B 模型分别达到了 1.58、1.30 和 1.16 的验证集困惑度，随着模型规模的增大呈现出单调递减的趋势。我们对训练好的模型在多个下游任务上进行了微调，包括来自 GLUE 基准 (Wang et al., 2019) 的 MNLI 和 QQP，来自斯坦福问答数据集 (Rajpurkar et al., 2016; 2018) 的 SQuAD 1.1 和 SQuAD 2.0，以及阅读理解 RACE 数据集 (Lai et al., 2017)。对于微调，我们遵循与 (Liu et al., 2019b) 相同的程序。我们首先对批量大小和学习率进行超参数调整。一旦获得最佳值，我们报告使用 5 个不同随机种子初始化的开发集结果的中位数。每个模型和任务使用的超参数在附录 A 中提供。表 5 显示了 MNLI、QQP、SQuAD 1.1 和 SQuAD 2.0 的开发集结果以及 RACE 的测试集结果。对于 RACE 的测试集结果，我们首先使用开发集找到在 5 个随机种子上给出中位分数的检查点，并报告该检查点在测试集上的结果。我们还报告了 SQuAD 开发集和 RACE 测试集的五路集成结果。从表 5 我们观察到：(a) 随着模型规模的增加，所有情况下的下游任务性能都得到了提升；(b) 我们的 3.9B 模型在开发集上相比其他基于 BERT 的模型建立了新的最先进结果；(c) 我们的 3.9B 模型在 RACE 测试集上不仅单模型而且集成后的结果都达到了最先进水平。

6. Conclusion and Future Work

6. 结论与未来工作

规则：
      - 输出中文翻译部分的时候，只保留翻译的标题，不要有任何其他的多余内容，不要重复，不要解释。
      - 不要输出与英文内容无关的内容。
      - 翻译时要保留原始段落格式，以及保留术语，例如 FLAC，JPEG 等。保留公司缩写，例如 Microsoft, Amazon, OpenAI 等。
      - 人名不翻译
      - 同时要保留引用的论文，例如 [20] 这样的引用。
      - 对于 Figure 和 Table，翻译的同时保留原有格式，例如：“Figure 1: ”翻译为“图 1: ”，“Table 1: ”翻译为：“表 1: ”。
      - 全角括号换成半角括号，并在左括号前面加半角空格，右括号后面加半角空格。
      - 在翻译专业术语时，第一次出现时要在括号里面写上英文原文，例如：“生成式 AI (Generative AI)”，之后就可以只写中文了。
      - 以下是常见的 AI 相关术语词汇对应表（English -> 中文）：
      * Transformer -> Transformer
      * Token -> Token
      * LLM/Large Language Model -> 大语言模型
      * Zero-shot -> 零样本
      * Few-shot -> 少样本
      * AI Agent -> AI智能体
      * AGI -> 通用人工智能
      * Python -> Python语言

      策略：

      分三步进行翻译工作：
      1. 不翻译无法识别的特殊字符和公式，原样返回
      2. 将HTML表格格式转换成Markdown表格格式
      3. 根据英文内容翻译成符合中文表达习惯的内容，不要遗漏任何信息

      最终只返回Markdown格式的翻译结果，不要回复无关内容。

In this work, we successfully surpassed the limitations posed by traditional single-GPU-per-model training by implementing model parallelism with only a few modifications to the existing PyTorch transformer implementations. We efficiently trained transformer based models up to 8.3 billion parameter on 512 NVIDIA V100 GPUs with 8-way model parallelism and achieved up to 15.1 PetaFLOPs sustained over the entire application. We also showed that for BERT models, careful attention to the placement of layer normalization in BERT-like models is critical to achieving increased accuracies as the model size increases. We study the effect of model size on down-stream task accuracy and achieve far superior results on downstream tasks and establish new SOTA for WikiText 103, LAMBADA, and RACE datasets. Finally, we open sourced our code to enable future work leveraging model parallel transformers.

在本工作中，我们通过仅对现有的 PyTorch Transformer 实现进行少量修改，成功实现了模型并行，从而超越了传统单 GPU 模型训练的限制。我们高效地训练了参数量高达 83 亿的大语言模型，在 512 个 NVIDIA V100 GPU 上使用 8 路模型并行，并在整个应用中实现了最高 15.1 PetaFLOPs 的持续性能。我们还展示了对于 BERT 模型，仔细调整类似 BERT 模型中的层归一化位置对于随着模型规模增大而提高准确性至关重要。我们研究了模型规模对下游任务准确性的影响，在下游任务上取得了远超以往的结果，并为 WikiText 103、LAMBADA 和 RACE 数据集建立了新的 SOTA。最后，我们开源了代码，以支持未来利用模型并行 Transformer 的工作。

There are several directions for future work. Continuing to increase the scale of pre training is a promising line of investigation that will further test existing deep learning hardware and software. To realize this, improvements in the efficiency and memory footprint of optimizers will be needed. In addition, training a model with more than 16 billion parameters will demand more memory than is available within 16 GPUs of a DGX-2H box. For such models, a hybrid intra-layer and inter-layer model parallelism along with inter-node model parallelism would be more suitable. Three other directions of investigation include (a) pretraining different model families (XLNet, T5), (b) evaluating performance of large models across more difficult and diverse downstream tasks (e.g. Generative Question Answering, Sum mari z ation, and Conversation), and (c) using knowledge distillation to train small student models from these large pretrained teacher models.

未来工作有几个方向。继续增加预训练的规模是一个有前景的研究方向，这将进一步测试现有的深度学习硬件和软件。为了实现这一点，需要改进优化器的效率和内存占用。此外，训练参数超过 160 亿的模型将需要比 DGX-2H 箱内 16 个 GPU 可用的更多内存。对于此类模型，混合层内和层间模型并行以及节点间模型并行将更为合适。其他三个研究方向包括 (a) 预训练不同的模型家族 (XLNet, T5)，(b) 评估大模型在更困难和多样化的下游任务中的性能（例如 生成式问答、摘要 和对话），以及 (c) 使用知识蒸馏从这些大型预训练教师模型中训练小型学生模型。

References

参考文献

Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G. S., Davis, A., Dean, J., Devin, M., Ghemawat, S., Goodfellow, I., Harp, A., Irving, G., Isard, M., Jia, Y., Jozefowicz, R., Kaiser, L., Kudlur, M., Levenberg, J., Man´e, D., Monga, R., Moore, S., Murray, D., Olah, C., Schuster, M., Shlens, J., Steiner, B., Sutskever, I., Talwar, K., Tucker, P., Vanhoucke, V., Va- sudevan, V., Vi´egas, F., Vinyals, O., Warden, P., Watten- berg, M., Wicke, M., Yu, Y., and Zheng, X. TensorFlow: Large-scale machine learning on heterogeneous systems, 2015. URL http://tensorflow.org/. Software available from tensorflow.org.

Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G. S., Davis, A., Dean, J., Devin, M., Ghemawat, S., Goodfellow, I., Harp, A., Irving, G., Isard, M., Jia, Y., Jozefowicz, R., Kaiser, L., Kudlur, M., Levenberg, J., Man´e, D., Monga, R., Moore, S., Murray, D., Olah, C., Schuster, M., Shlens, J., Steiner, B., Sutskever, I., Talwar, K., Tucker, P., Vanhoucke, V., Va- sudevan, V., Vi´egas, F., Vinyals, O., Warden, P., Watten- berg, M., Wicke, M., Yu, Y., 和 Zheng, X. TensorFlow: 大规模异构系统上的机器学习，2015。URL http://tensorflow.org/. 软件可从 tensorflow.org 获取。

Ba, J. L., Kiros, J. R., and Hinton, G. E. Layernorm. CoRR, abs/1607.06450, 2016. URL http://arxiv.org/ abs/1607.06450.

Ba, J. L., Kiros, J. R., 和 Hinton, G. E. Layernorm. CoRR, abs/1607.06450, 2016. URL http://arxiv.org/abs/1607.06450.

层归一化 (Layernorm)。

Chen, C.-C., Yang, C.-L., and Cheng, H.-Y. Efficient and robust parallel dnn training through model parallelism on multi-gpu platform. arXiv:1809.02839, 2018.

陈，C.-C.，杨，C.-L.，和程，H.-Y. 通过多 GPU 平台上的模型并行性实现高效且稳健的并行 DNN 训练。arXiv:1809.02839, 2018.

Chen, T., Xu, B., Zhang, C., and Guestrin, C. Training deep nets with sublinear memory cost. CoRR, abs/1604.06174, 2016. URL http://arxiv.org/ abs/1604.06174.

陈，T., Xu, B., Zhang, C., 和 Guestrin, C. 用次线性内存成本训练深度网络。CoRR, abs/1604.06174, 2016。URL http://arxiv.org/abs/1604.06174。

Dai, Z., Yang, Z., Yang, Y., Carbonell, J. G., Le, Q. V., and Salak hut dino v, R. Transformer-xl: Attentive language models beyond a fixed-length context. CoRR, abs/1901.02860, 2019. URL http://arxiv.org/ abs/1901.02860.

Dai, Z., Yang, Z., Yang, Y., Carbonell, J. G., Le, Q. V., 和 Salak hut dino v, R. Transformer-XL: 超越固定长度上下文的注意力语言模型 (Attentive language models beyond a fixed-length context). CoRR, abs/1901.02860, 2019. URL http://arxiv.org/abs/1901.02860.

He, K. Accurate, large minibatch SGD: training imagenet in 1 hour. CoRR, abs/1706.02677, 2017.

何凯明，准确的大批量 SGD：1小时内训练 ImageNet。CoRR, abs/1706.02677, 2017。

Harlap, A., Narayanan, D., Phan is haye e, A., Seshadri, V., Devanur, N., Ganger, G., and Gibbons, P. Pipedream: Fast and efficient pipeline parallel dnn training. arXiv:1806.03377, 2018.

Harlap, A., Narayanan, D., Phan is haye e, A