ImageNet Classification with Deep Convolutional Neural Networks

基于深度卷积神经网络的ImageNet分类

Alex Krizhevsky University of Toronto kriz@cs.utoronto.ca

Alex Krizhevsky 多伦多大学 kriz@cs.utoronto.ca

Ilya Sutskever University of Toronto ilya@cs.utoronto.ca

Ilya Sutskever 多伦多大学 ilya@cs.utoronto.ca

Geoffrey E. Hinton University of Toronto hinton@cs.utoronto.ca

Geoffrey E. Hinton 多伦多大学 hinton@cs.utoronto.ca

Abstract

摘要

We trained a large, deep convolutional neural network to classify the 1.2 million high-resolution images in the ImageNet LSVRC-2010 contest into the 1000 different classes. On the test data, we achieved top-1 and top-5 error rates of $37.5%$ and $17.0%$ which is considerably better than the previous state-of-the-art. The neural network, which has 60 million parameters and 650,000 neurons, consists of five convolutional layers, some of which are followed by max-pooling layers, and three fully-connected layers with a final 1000-way softmax. To make training faster, we used non-saturating neurons and a very efficient GPU implementation of the convolution operation. To reduce over fitting in the fully-connected layers we employed a recently-developed regular iz ation method called “dropout” that proved to be very effective. We also entered a variant of this model in the ILSVRC-2012 competition and achieved a winning top-5 test error rate of $15.3%$ , compared to $26.2%$ achieved by the second-best entry.

我们训练了一个大型深度卷积神经网络，将ImageNet LSVRC-2010竞赛中的120万张高分辨率图像分类为1000个不同类别。在测试数据上，我们取得了$37.5%$的top-1错误率和$17.0%$的top-5错误率，显著优于此前的最先进水平。该神经网络包含6000万个参数和65万个神经元，由五个卷积层（部分层后接最大池化层）和三个全连接层（最终采用1000路softmax）组成。为加速训练，我们使用了非饱和神经元和高度优化的GPU卷积运算实现。为减少全连接层的过拟合，我们采用了一种名为"dropout"的新型正则化方法，效果显著。我们还以该模型的改进版本参加了ILSVRC-2012竞赛，并以$15.3%$的top-5测试错误率夺冠，而第二名成绩为$26.2%$。

1 Introduction

1 引言

Current approaches to object recognition make essential use of machine learning methods. To improve their performance, we can collect larger datasets, learn more powerful models, and use better techniques for preventing over fitting. Until recently, datasets of labeled images were relatively small — on the order of tens of thousands of images (e.g., NORB [16], Caltech-101/256 [8, 9], and CIFAR-10/100 [12]). Simple recognition tasks can be solved quite well with datasets of this size, especially if they are augmented with label-preserving transformations. For example, the currentbest error rate on the MNIST digit-recognition task $(<0.3%)$ approaches human performance [4]. But objects in realistic settings exhibit considerable variability, so to learn to recognize them it is necessary to use much larger training sets. And indeed, the shortcomings of small image datasets have been widely recognized (e.g., Pinto et al. [21]), but it has only recently become possible to collect labeled datasets with millions of images. The new larger datasets include LabelMe [23], which consists of hundreds of thousands of fully-segmented images, and ImageNet [6], which consists of over 15 million labeled high-resolution images in over 22,000 categories.

当前的目标识别方法主要依赖于机器学习技术。为提升性能，我们可以收集更大规模的数据集、训练更强大的模型，并采用更有效的防过拟合技术。直到最近，标注图像数据集的规模仍相对较小——约数万量级（例如NORB [16]、Caltech-101/256 [8,9]和CIFAR-10/100 [12]）。对于简单识别任务，这类规模的数据集已能取得较好效果，特别是辅以标签保持的数据增强时。例如MNIST手写数字识别任务当前最佳错误率$(<0.3%)$已接近人类水平[4]。但现实场景中的物体存在显著多样性，要准确识别它们必须使用更庞大的训练集。事实上，小规模图像数据集的缺陷已被广泛认知（如Pinto等[21]），但直到近年才具备收集数百万标注图像的能力。新兴的大规模数据集包括包含数十万全分割图像的LabelMe [23]，以及涵盖22000多个类别、超过1500万张高分辨率标注图像的ImageNet [6]。

To learn about thousands of objects from millions of images, we need a model with a large learning capacity. However, the immense complexity of the object recognition task means that this problem cannot be specified even by a dataset as large as ImageNet, so our model should also have lots of prior knowledge to compensate for all the data we don’t have. Convolutional neural networks (CNNs) constitute one such class of models [16, 11, 13, 18, 15, 22, 26]. Their capacity can be controlled by varying their depth and breadth, and they also make strong and mostly correct assumptions about the nature of images (namely, station ari ty of statistics and locality of pixel dependencies). Thus, compared to standard feed forward neural networks with similarly-sized layers, CNNs have much fewer connections and parameters and so they are easier to train, while their theoretically-best performance is likely to be only slightly worse.

为了从数百万张图像中学习成千上万的物体，我们需要一个具备强大学习能力的模型。然而，物体识别任务的极端复杂性意味着即使像ImageNet这样庞大的数据集也无法完全定义这个问题，因此我们的模型还应具备大量先验知识来弥补未覆盖的数据。卷积神经网络(CNN)正是这样一类模型[16, 11, 13, 18, 15, 22, 26]。通过调整其深度和宽度可以控制其容量，同时它们对图像本质(即统计平稳性和像素依赖的局部性)作出了强有力且基本正确的假设。因此，与具有相似规模层的标准前馈神经网络相比，CNN的连接和参数数量要少得多，更易于训练，而其理论最佳性能可能仅略逊一筹。

Despite the attractive qualities of CNNs, and despite the relative efficiency of their local architecture, they have still been prohibitively expensive to apply in large scale to high-resolution images. Luckily, current GPUs, paired with a highly-optimized implementation of 2D convolution, are powerful enough to facilitate the training of interestingly-large CNNs, and recent datasets such as ImageNet contain enough labeled examples to train such models without severe over fitting.

尽管卷积神经网络 (CNN) 具有诱人的特性，且其局部架构相对高效，但将其大规模应用于高分辨率图像仍存在难以承受的计算成本。幸运的是，当前 GPU 配合高度优化的二维卷积实现，已具备足够能力训练具有实用价值的大规模 CNN，而 ImageNet 等现代数据集提供的海量标注样本也能有效避免这类模型出现严重过拟合。

The specific contributions of this paper are as follows: we trained one of the largest convolutional neural networks to date on the subsets of ImageNet used in the ILSVRC-2010 and ILSVRC-2012 competitions [2] and achieved by far the best results ever reported on these datasets. We wrote a highly-optimized GPU implementation of 2D convolution and all the other operations inherent in training convolutional neural networks, which we make available publicly1. Our network contains a number of new and unusual features which improve its performance and reduce its training time, which are detailed in Section 3. The size of our network made over fitting a significant problem, even with 1.2 million labeled training examples, so we used several effective techniques for preventing over fitting, which are described in Section 4. Our final network contains five convolutional and three fully-connected layers, and this depth seems to be important: we found that removing any convolutional layer (each of which contains no more than $1%$ of the model’s parameters) resulted in inferior performance.

本文的具体贡献如下：我们在ILSVRC-2010和ILSVRC-2012竞赛[2]使用的ImageNet子集上训练了迄今为止最大的卷积神经网络之一，并取得了这些数据集上迄今为止报告的最佳结果。我们编写了一个高度优化的2D卷积GPU实现以及训练卷积神经网络所需的所有其他操作，并公开了代码1。我们的网络包含多项新颖且不寻常的特性，这些特性提升了其性能并缩短了训练时间，详见第3节。即使有120万带标注的训练样本，网络规模仍导致过拟合成为显著问题，因此我们采用了多种有效技术来防止过拟合，详见第4节。最终网络包含五个卷积层和三个全连接层，这种深度至关重要：我们发现移除任何卷积层（每层参数不超过模型参数的$1%$）都会导致性能下降。

In the end, the network’s size is limited mainly by the amount of memory available on current GPUs and by the amount of training time that we are willing to tolerate. Our network takes between five and six days to train on two GTX 580 3GB GPUs. All of our experiments suggest that our results can be improved simply by waiting for faster GPUs and bigger datasets to become available.

最终，网络的规模主要受限于当前GPU的可用内存量以及我们能够容忍的训练时长。我们的网络在两块GTX 580 3GB GPU上需要5到6天完成训练。所有实验表明，只需等待更快的GPU和更大规模数据集的出现，我们的结果就能得到提升。

2 The Dataset

2 数据集

ImageNet is a dataset of over 15 million labeled high-resolution images belonging to roughly 22,000 categories. The images were collected from the web and labeled by human labelers using Amazon’s Mechanical Turk crowd-sourcing tool. Starting in 2010, as part of the Pascal Visual Object Challenge, an annual competition called the ImageNet Large-Scale Visual Recognition Challenge (ILSVRC) has been held. ILSVRC uses a subset of ImageNet with roughly 1000 images in each of 1000 categories. In all, there are roughly 1.2 million training images, 50,000 validation images, and 150,000 testing images.

ImageNet是一个包含超过1500万张标注高分辨率图像的数据集，涵盖约2.2万个类别。这些图像从网络收集，并通过Amazon的Mechanical Turk众包工具由人工标注员标注。自2010年起，作为Pascal Visual Object Challenge的一部分，每年都会举办名为ImageNet大规模视觉识别挑战赛(ILSVRC)的竞赛。ILSVRC使用ImageNet的子集，包含1000个类别，每个类别约1000张图像。总计约120万张训练图像、5万张验证图像和15万张测试图像。

ILSVRC-2010 is the only version of ILSVRC for which the test set labels are available, so this is the version on which we performed most of our experiments. Since we also entered our model in the ILSVRC-2012 competition, in Section 6 we report our results on this version of the dataset as well, for which test set labels are unavailable. On ImageNet, it is customary to report two error rates: top-1 and top-5, where the top-5 error rate is the fraction of test images for which the correct label is not among the five labels considered most probable by the model.

ILSVRC-2010是唯一提供测试集标签的ILSVRC版本，因此我们在此版本上进行了大部分实验。由于我们也参加了ILSVRC-2012竞赛，在第6节中我们同样报告了该数据集版本的结果（该版本测试集标签未公开）。在ImageNet上，通常报告两种错误率：top-1和top-5，其中top-5错误率指测试图像的正确标签不在模型预测概率最高的五个标签中的比例。

ImageNet consists of variable-resolution images, while our system requires a constant input dimensionality. Therefore, we down-sampled the images to a fixed resolution of $256\times256$ . Given a rectangular image, we first rescaled the image such that the shorter side was of length 256, and then cropped out the central $256\times256$ patch from the resulting image. We did not pre-process the images in any other way, except for subtracting the mean activity over the training set from each pixel. So we trained our network on the (centered) raw RGB values of the pixels.

ImageNet 包含不同分辨率的图像，而我们的系统需要恒定的输入维度。因此，我们将图像下采样至固定分辨率 $256\times256$。对于矩形图像，我们首先调整图像尺寸使短边长度为 256，然后从结果图像中裁剪出中心 $256\times256$ 区域。除从每个像素减去训练集上的平均活跃度外，我们未对图像进行任何其他预处理。因此，我们直接在像素的 (中心化) 原始 RGB 值上训练网络。

3 The Architecture

3 架构

The architecture of our network is summarized in Figure 2. It contains eight learned layers — five convolutional and three fully-connected. Below, we describe some of the novel or unusual features of our network’s architecture. Sections 3.1-3.4 are sorted according to our estimation of their importance, with the most important first.

我们的网络架构如图 2 所示。它包含八个学习层——五个卷积层和三个全连接层。下面我们将介绍该网络架构中一些新颖或不寻常的特性。章节 3.1-3.4 按照我们对其重要性的评估进行排序，最重要的排在最前面。

3.1 ReLU Non linearity

3.1 ReLU 非线性激活函数

The standard way to model a neuron’s output $f$ as a function of its input $x$ is with $f(x)=\operatorname{tanh}(x)$ or $f(x)=(1+{e^{-x}})^{-1}$ . In terms of training time with gradient descent, these saturating nonlinear i ties are much slower than the non-saturating non linearity $f(x)=\operatorname* {max}(0,x)$ . Following Nair and Hinton [20], we refer to neurons with this non linearity as Rectified Linear Units (ReLUs). Deep convolutional neural networks with ReLUs train several times faster than their equivalents with tanh units. This is demonstrated in Figure 1, which shows the number of iterations required to reach $25%$ training error on the CIFAR-10 dataset for a particular four-layer convolutional network. This plot shows that we would not have been able to experiment with such large neural networks for this work if we had used traditional saturating neuron models.

标准方法将神经元的输出 $f$ 建模为输入 $x$ 的函数，通常使用 $f(x)=\operatorname{tanh}(x)$ 或 $f(x)=(1+{e^{-x}})^{-1}$。在梯度下降训练时间方面，这些饱和非线性 (saturating nonlinearities) 比非饱和非线性 (non-saturating nonlinearity) $f(x)=\operatorname* {max}(0,x)$ 慢得多。遵循 Nair 和 Hinton [20] 的命名，我们将采用这种非线性的神经元称为修正线性单元 (Rectified Linear Units, ReLUs)。使用 ReLUs 的深度卷积神经网络训练速度比使用 tanh 单元的同类网络快数倍。图 1 展示了在 CIFAR-10 数据集上，某个特定四层卷积网络达到 $25%$ 训练错误率所需的迭代次数。该图表明，如果使用传统的饱和神经元模型，我们无法在本研究中实验如此大规模的神经网络。

We are not the first to consider alternatives to traditional neuron models in CNNs. For example, Jarrett et al. [11] claim that the non linearity $f(x)=\left|\operatorname{tanh}(x)\right|$ works particularly well with their type of contrast normalization followed by local average pooling on the Caltech-101 dataset. However, on this dataset the primary concern is preventing over fitting, so the effect they are observing is different from the accelerated ability to fit the training set which we report when using ReLUs. Faster learning has a great influence on the performance of large models trained on large datasets.

我们并非首个考虑在CNN中替代传统神经元模型的研究团队。例如，Jarrett等人[11]提出非线性函数$f(x)=\left|\operatorname{tanh}(x)\right|$在Caltech-101数据集上结合其对比度归一化与局部平均池化方法表现优异。但该数据集的核心问题是防止过拟合，因此他们观察到的效果与我们使用ReLU时报告的加速训练集拟合能力存在本质差异。学习速度的提升对大规模数据集训练的大型模型性能具有重大影响。

Figure 1: A four-layer convolutional neural network with ReLUs (solid line) reaches a $25%$ training error rate on CIFAR-10 six times faster than an equivalent network with tanh neurons (dashed line). The learning rates for each network were chosen independently to make training as fast as possible. No regular iz ation of any kind was employed. The magnitude of the effect demonstrated here varies with network architecture, but networks with ReLUs consistently learn several times faster than equivalents with saturating neurons.

图 1: 使用ReLU激活函数(实线)的四层卷积神经网络在CIFAR-10数据集上达到25%训练错误率的速度，比使用tanh神经元(虚线)的同等网络快六倍。每个网络的学习率均经过独立选择以实现最快训练速度。未采用任何形式的正则化。该效果的显著程度会随网络架构变化，但采用ReLU的网络始终比使用饱和神经元的同等网络快数倍。

3.2 Training on Multiple GPUs

3.2 多GPU训练

A single GTX 580 GPU has only 3GB of memory, which limits the maximum size of the networks that can be trained on it. It turns out that 1.2 million training examples are enough to train networks which are too big to fit on one GPU. Therefore we spread the net across two GPUs. Current GPUs are particularly well-suited to cross-GPU parallel iz ation, as they are able to read from and write to one another’s memory directly, without going through host machine memory. The parallel iz ation scheme that we employ essentially puts half of the kernels (or neurons) on each GPU, with one additional trick: the GPUs communicate only in certain layers. This means that, for example, the kernels of layer 3 take input from all kernel maps in layer 2. However, kernels in layer 4 take input only from those kernel maps in layer 3 which reside on the same GPU. Choosing the pattern of connectivity is a problem for cross-validation, but this allows us to precisely tune the amount of communication until it is an acceptable fraction of the amount of computation.

一块GTX 580 GPU仅有3GB内存，这限制了可训练网络的最大规模。事实证明，120万训练样本足以训练出无法单卡容纳的大型网络。为此我们将网络拆分到两块GPU上运行。现代GPU特别适合跨卡并行计算，因为它们能直接读写彼此显存，无需经过主机内存。我们采用的并行化方案本质上是将半数卷积核（或神经元）分配到每块GPU，并附加一个技巧：GPU仅在特定层间通信。例如，第3层卷积核会接收第2层所有特征图的输入，而第4层卷积核仅接收同卡第3层特征图的输入。连接模式的选择需通过交叉验证确定，但这种方法能让我们精确调控通信量，直至其与计算量之比达到可接受范围。

The resultant architecture is somewhat similar to that of the “columnar'’ CNN employed by Ciresan et al. [5], except that our columns are not independent (see Figure 2). This scheme reduces our top-1 and top-5 error rates by $1.7%$ and $1.2%$ , respectively, as compared with a net with half as many kernels in each convolutional layer trained on one GPU. The two-GPU net takes slightly less time to train than the one-GPU net2.

最终得到的架构与 Ciresan 等人 [5] 采用的"柱状" CNN 有些相似，只是我们的各列并不独立 (见图 2)。与在单个 GPU 上训练、每个卷积层核数减半的网络相比，该方案使我们的 top-1 和 top-5 错误率分别降低了 $1.7%$ 和 $1.2%$。双 GPU 网络的训练时间略短于单 GPU 网络。

3.3 Local Response Normalization

3.3 局部响应归一化 (Local Response Normalization)

ReLUs have the desirable property that they do not require input normalization to prevent them from saturating. If at least some training examples produce a positive input to a ReLU, learning will happen in that neuron. However, we still find that the following local normalization scheme aids generalization. Denoting by $a_ {x,y}^{i}$ the activity of a neuron computed by applying kernel $i$ at position $(x,y)$ and then applying the ReLU non linearity, the response-normalized activity $b_ {x,y}^{i}$ is given by the expression

ReLU具有一个理想特性：它们不需要通过输入归一化来防止饱和。如果至少有一些训练样本对ReLU产生了正输入，该神经元就会进行学习。然而，我们发现以下局部归一化方案仍有助于提升泛化能力。用$a_ {x,y}^{i}$表示在位置$(x,y)$应用核$i$并通过ReLU非线性计算得到的神经元活性，响应归一化后的活性$b_ {x,y}^{i}$由以下表达式给出：

$$
b_ {x,y}^{i}=a_ {x,y}^{i}/\left(k+\alpha\sum_ {j=\operatorname* {max}(0,i-n/2)}^{\operatorname* {min}(N-1,i+n/2)}(a_ {x,y}^{j})^{2}\right)^{\beta}
$$

$$
b_ {x,y}^{i}=a_ {x,y}^{i}/\left(k+\alpha\sum_ {j=\operatorname* {max}(0,i-n/2)}^{\operatorname* {min}(N-1,i+n/2)}(a_ {x,y}^{j})^{2}\right)^{\beta}
$$

where the sum runs over $n$ “adjacent” kernel maps at the same spatial position, and $N$ is the total number of kernels in the layer. The ordering of the kernel maps is of course arbitrary and determined before training begins. This sort of response normalization implements a form of lateral inhibition inspired by the type found in real neurons, creating competition for big activities amongst neuron outputs computed using different kernels. The constants $k,n,\alpha$ , and $\beta$ are hyper-parameters whose values are determined using a validation set; we used $k=2$ , $n=5$ , $\alpha=10^{-4}$ , and $\beta=0.75$ . We applied this normalization after applying the ReLU non linearity in certain layers (see Section 3.5).

其中求和运算对同一空间位置上 $n$ 个“相邻”核映射进行，$N$ 是该层的核总数。核映射的顺序当然是任意的，并在训练开始前确定。这种响应归一化实现了受真实神经元类型启发的侧向抑制形式，为使用不同核计算出的神经元输出之间的大活动创造了竞争。常数 $k,n,\alpha$ 和 $\beta$ 是超参数，其值通过验证集确定；我们使用 $k=2$、$n=5$、$\alpha=10^{-4}$ 和 $\beta=0.75$。我们在某些层应用ReLU非线性后应用了这种归一化（见第3.5节）。

This scheme bears some resemblance to the local contrast normalization scheme of Jarrett et al. [11], but ours would be more correctly termed “brightness normalization”, since we do not subtract the mean activity. Response normalization reduces our top-1 and top-5 error rates by $1.4%$ and $1.2%$ , respectively. We also verified the effectiveness of this scheme on the CIFAR-10 dataset: a four-layer CNN achieved a $13%$ test error rate without normalization and $11%$ with normalization 3.

该方案与Jarrett等人[11]提出的局部对比度归一化方案有些相似，但我们的方法更应称为"亮度归一化"，因为我们并未减去平均激活值。响应归一化使我们的top-1和top-5错误率分别降低了$1.4%$和$1.2%$。我们还在CIFAR-10数据集上验证了该方案的有效性：未使用归一化的四层CNN测试错误率为$13%$，而采用归一化3后降至$11%$。

3.4 Overlapping Pooling

3.4 重叠池化

Pooling layers in CNNs summarize the outputs of neighboring groups of neurons in the same kernel map. Traditionally, the neighborhoods summarized by adjacent pooling units do not overlap (e.g., [17, 11, 4]). To be more precise, a pooling layer can be thought of as consisting of a grid of pooling units spaced $s$ pixels apart, each summarizing a neighborhood of size $z\times z$ centered at the location of the pooling unit. If we set $s=z$ , we obtain traditional local pooling as commonly employed in CNNs. If we set $s<z$ , we obtain overlapping pooling. This is what we use throughout our network, with $s=2$ and $z=3$ . This scheme reduces the top-1 and top-5 error rates by $0.4%$ and $0.3%$ , respectively, as compared with the non-overlapping scheme $s=2,z=2$ , which produces output of equivalent dimensions. We generally observe during training that models with overlapping pooling find it slightly more difficult to overfit.

CNN中的池化层会对同一核映射中相邻神经元的输出进行汇总。传统上，相邻池化单元汇总的区域互不重叠（例如[17, 11, 4]）。具体而言，池化层可视为由间距为$s$像素的池化单元网格组成，每个单元汇总以该位置为中心、大小为$z×z$的区域。当设定$s=z$时，即得到CNN常用的传统局部池化；当$s<z$时则形成重叠池化。本网络全程采用$s=2$和$z=3$的重叠池化方案，与输出维度相同但$s=2,z=2$的非重叠方案相比，该方案使top-1和top-5错误率分别降低$0.4%$和$0.3%$。训练中我们观察到，采用重叠池化的模型较难出现过拟合现象。

3.5 Overall Architecture

3.5 整体架构

Now we are ready to describe the overall architecture of our CNN. As depicted in Figure 2, the net contains eight layers with weights; the first five are convolutional and the remaining three are fullyconnected. The output of the last fully-connected layer is fed to a 1000-way softmax which produces a distribution over the 1000 class labels. Our network maximizes the multi no mi al logistic regression objective, which is equivalent to maximizing the average across training cases of the log-probability of the correct label under the prediction distribution.

现在我们可以描述CNN的整体架构了。如图2所示，该网络包含八个带权重的层：前五层是卷积层，后三层是全连接层。最后一个全连接层的输出被送入一个1000路的softmax分类器，生成1000个类别标签的概率分布。我们的网络通过最大化多项逻辑回归目标函数进行优化，这相当于最大化训练样本在预测分布下正确标签对数概率的平均值。

The kernels of the second, fourth, and fifth convolutional layers are connected only to those kernel maps in the previous layer which reside on the same GPU (see Figure 2). The kernels of the third convolutional layer are connected to all kernel maps in the second layer. The neurons in the fullyconnected layers are connected to all neurons in the previous layer. Response-normalization layers follow the first and second convolutional layers. Max-pooling layers, of the kind described in Section 3.4, follow both response-normalization layers as well as the fifth convolutional layer. The ReLU non-linearity is applied to the output of every convolutional and fully-connected layer.

第二、第四和第五卷积层的核仅连接到前一层中位于同一GPU上的核映射（见图2）。第三卷积层的核则连接到第二层的所有核映射。全连接层中的神经元与前一层的所有神经元相连。响应归一化层紧随第一和第二卷积层之后。如3.4节所述的最大池化层，既跟在响应归一化层之后，也跟在第五卷积层之后。ReLU非线性被应用于每个卷积层和全连接层的输出。

The first convolutional layer filters the $224\times224\times3$ input image with 96 kernels of size $11\times11\times3$ with a stride of 4 pixels (this is the distance between the receptive field centers of neighboring neurons in a kernel map). The second convolutional layer takes as input the (response-normalized and pooled) output of the first convolutional layer and filters it with 256 kernels of size $5\times5\times48$ . The third, fourth, and fifth convolutional layers are connected to one another without any intervening pooling or normalization layers. The third convolutional layer has 384 kernels of size $3\times3\times$ 256 connected to the (normalized, pooled) outputs of the second convolutional layer. The fourth convolutional layer has 384 kernels of size $3\times3\times192$ , and the fifth convolutional layer has 256 kernels of size $3\times3\times192$ . The fully-connected layers have 4096 neurons each.

第一卷积层用96个尺寸为$11\times11\times3$的核(步长4像素)对$224\times224\times3$输入图像进行滤波(步长指核映射中相邻神经元感受野中心间距)。第二卷积层以第一卷积层的(响应归一化及池化后)输出作为输入，用256个$5\times5\times48$尺寸的核进行滤波。第三、四、五卷积层直接级联，中间未插入池化或归一化层。第三卷积层包含384个$3\times3\times256$尺寸的核，连接至第二卷积层(归一化及池化后)的输出。第四卷积层含384个$3\times3\times192$尺寸的核，第五卷积层含256个$3\times3\times192$尺寸的核。全连接层各包含4096个神经元。

Figure 2: An illustration of the architecture of our CNN, explicitly showing the delineation of responsibilities between the two GPUs. One GPU runs the layer-parts at the top of the figure while the other runs the layer-parts at the bottom. The GPUs communicate only at certain layers. The network’s input is 150,528-dimensional, and the number of neurons in the network’s remaining layers is given by 253,440–186,624–64,896–64,896–43,264– 4096–4096–1000.

图 2: 我们的CNN架构示意图，明确展示了两块GPU之间的职责划分。一块GPU运行图中顶部的层部分，另一块运行底部的层部分。GPU仅在特定层进行通信。网络输入维度为150,528，其余各层神经元数量依次为253,440–186,624–64,896–64,896–43,264–4096–4096–1000。

4 Reducing Over fitting

4 减少过拟合

Our neural network architecture has 60 million parameters. Although the 1000 classes of ILSVRC make each training example impose 10 bits of constraint on the mapping from image to label, this turns out to be insufficient to learn so many parameters without considerable over fitting. Below, we describe the two primary ways in which we combat over fitting.

我们的神经网络架构拥有6000万个参数。尽管ILSVRC的1000个类别使得每个训练样本对从图像到标签的映射施加了10比特的约束，但事实证明这不足以在避免严重过拟合的情况下学习如此多的参数。下面，我们将介绍两种主要的对抗过拟合的方法。

4.1 Data Augmentation

4.1 数据增强

The easiest and most common method to reduce over fitting on image data is to artificially enlarge the dataset using label-preserving transformations (e.g., [25, 4, 5]). We employ two distinct forms of data augmentation, both of which allow transformed images to be produced from the original images with very little computation, so the transformed images do not need to be stored on disk. In our implementation, the transformed images are generated in Python code on the CPU while the GPU is training on the previous batch of images. So these data augmentation schemes are, in effect, computationally free.

降低图像数据过拟合最简单且常用的方法是使用保留标签的变换（如[25, 4, 5]）人为扩展数据集。我们采用两种不同的数据增强形式，二者都能以极低计算量从原图像生成变换后的图像，因此无需将变换后的图像存储在磁盘上。在我们的实现中，变换图像由CPU端的Python语言代码实时生成，而GPU正处理前一批图像的训练。因此，这些数据增强方案实际上不产生额外计算开销。

The first form of data augmentation consists of generating image translations and horizontal reflections. We do this by extracting random $224\times224$ patches (and their horizontal reflections) from the $256\times256$ images and training our network on these extracted patches4. This increases the size of our training set by a factor of 2048, though the resulting training examples are, of course, highly interdependent. Without this scheme, our network suffers from substantial over fitting, which would have forced us to use much smaller networks. At test time, the network makes a prediction by extracting five $224\times224$ patches (the four corner patches and the center patch) as well as their horizontal reflections (hence ten patches in all), and averaging the predictions made by the network’s softmax layer on the ten patches.

第一种数据增强形式包括生成图像平移和水平翻转。我们通过从$256\times256$图像中随机提取$224\times224$的块(及其水平翻转)，并在这些提取的块上训练网络来实现这一点4。这使我们的训练集规模扩大了2048倍，尽管生成的训练样本之间高度相关。若不采用此方案，网络会出现严重的过拟合，迫使我们使用更小的网络。测试时，网络通过提取五个$224\times224$块(四个角块和中心块)及其水平翻转(共十个块)，并对网络softmax层在这十个块上的预测结果取平均来做出最终预测。

The second form of data augmentation consists of altering the intensities of the RGB channels in training images. Specifically, we perform PCA on the set of RGB pixel values throughout the ImageNet training set. To each training image, we add multiples of the found principal components, with magnitudes proportional to the corresponding eigenvalues times a random variable drawn from a Gaussian with mean zero and standard deviation 0.1. Therefore to each RGB image pixel $I_ {x y}={}$ $[I_ {x y}^{R},I_ {x y}^{G},I_ {x y}^{B}]^{T}$ we add the following quantity:

第二种数据增强形式涉及改变训练图像中RGB通道的强度。具体而言，我们对整个ImageNet训练集中的RGB像素值集合进行PCA分析。对每张训练图像，我们添加所找到的主成分的倍数，其大小与对应特征值乘以一个从均值为零、标准差为0.1的高斯分布中抽取的随机变量成比例。因此，对每个RGB图像像素 $I_ {x y}={}$ $[I_ {x y}^{R},I_ {x y}^{G},I_ {x y}^{B}]^{T}$ ，我们添加以下量：

$$
[{\bf p}_ {1},{\bf p}_ {2},{\bf p}_ {3}][\alpha_ {1}\lambda_ {1},\alpha_ {2}\lambda_ {2},\alpha_ {3}\lambda_ {3}]^{T}
$$

$$
[{\bf p}_ {1},{\bf p}_ {2},{\bf p}_ {3}][\alpha_ {1}\lambda_ {1},\alpha_ {2}\lambda_ {2},\alpha_ {3}\lambda_ {3}]^{T}
$$

where $\mathbf{p}_ {i}$ and $\lambda_ {i}$ are $i^{\cdot}$ th ei gen vector and eigenvalue of the $3\times3$ covariance matrix of RGB pixel values, respectively, and $\alpha_ {i}$ is the aforementioned random variable. Each $\alpha_ {i}$ is drawn only once for all the pixels of a particular training image until that image is used for training again, at which point it is re-drawn. This scheme approximately captures an important property of natural images, namely, that object identity is invariant to changes in the intensity and color of the illumination. This scheme reduces the top-1 error rate by over $1%$ .

其中 $\mathbf{p}_ {i}$ 和 $\lambda_ {i}$ 分别是 RGB 像素值 $3\times3$ 协方差矩阵的第 $i$ 个特征向量和特征值，$\alpha_ {i}$ 是上述随机变量。对于特定训练图像的所有像素，每个 $\alpha_ {i}$ 仅采样一次，直到该图像再次用于训练时重新采样。该方案近似捕捉了自然图像的一个重要特性，即物体身份对照明强度和颜色变化具有不变性。此方案将 top-1 错误率降低了超过 $1%$。

4.2 Dropout

4.2 Dropout

Combining the predictions of many different models is a very successful way to reduce test errors [1, 3], but it appears to be too expensive for big neural networks that already take several days to train. There is, however, a very efficient version of model combination that only costs about a factor of two during training. The recently-introduced technique, called “dropout” [10], consists of setting to zero the output of each hidden neuron with probability 0.5. The neurons which are “dropped out” in this way do not contribute to the forward pass and do not participate in backpropagation. So every time an input is presented, the neural network samples a different architecture, but all these architectures share weights. This technique reduces complex co-adaptations of neurons, since a neuron cannot rely on the presence of particular other neurons. It is, therefore, forced to learn more robust features that are useful in conjunction with many different random subsets of the other neurons. At test time, we use all the neurons but multiply their outputs by 0.5, which is a reasonable approximation to taking the geometric mean of the predictive distributions produced by the exponentially-many dropout networks.

结合多个不同模型的预测是一种非常有效降低测试误差的方法 [1, 3]，但对于已经需要数天训练时间的大型神经网络来说，这种方法似乎成本过高。然而，存在一种高效的模型组合方式，其训练成本仅增加约两倍。这种称为"dropout" [10] 的新技术，其原理是以0.5的概率将每个隐藏神经元的输出置零。被"丢弃"的神经元不参与前向传播和反向传播过程。因此，每次输入数据时，神经网络都会采样得到不同的架构，但这些架构共享权重参数。该技术通过阻止神经元依赖特定其他神经元的存在，有效减少了神经元间的复杂共适应现象。这使得神经元被迫学习更具鲁棒性的特征，这些特征需要与其他神经元的多种随机子集协同工作。在测试阶段，我们使用所有神经元，但将其输出乘以0.5，这是对指数级数量dropout网络产生的预测分布进行几何平均的合理近似。

We use dropout in the first two fully-connected layers of Figure 2. Without dropout, our network exhibits substantial over fitting. Dropout roughly doubles the number of iterations required to converge.

在图 2 的前两个全连接层中使用了 dropout (随机失活)。不使用 dropout 时，我们的网络会出现严重的过拟合现象。采用 dropout 后，收敛所需的迭代次数大约增加了一倍。

5 Details of learning

5 学习细节

We trained our models using stochastic gradient descent with a batch size of 128 examples, momentum of 0.9, and weight decay of 0.0005. We found that this small amount of weight decay was important for the model to learn. In other words, weight decay here is not merely a regularize r: it reduces the model’s training error. The update rule for weight $w$ was

我们使用随机梯度下降训练模型，批大小为128个样本，动量为0.9，权重衰减为0.0005。我们发现这种小幅权重衰减对模型学习至关重要。换句话说，这里的权重衰减不仅是正则化器：它降低了模型的训练误差。权重$w$的更新规则为

$$
\begin{array}{r c l}{{v_ {i+1}}}&{{:=}}&{{0.9\cdot v_ {i}-0.0005\cdot\epsilon\cdot w_ {i}-\epsilon\cdot\displaystyle\left\langle\frac{\partial L}{\partial w}\big|_ {w_ {i}}\right\rangle_ {D_ {i}}}}\ {{w_ {i+1}}}&{{:=}}&{{w_ {i}+v_ {i+1}}}\end{array}
$$

$$
\begin{array}{r c l}{{v_ {i+1}}}&{{:=}}&{{0.9\cdot v_ {i}-0.0005\cdot\epsilon\cdot w_ {i}-\epsilon\cdot\displaystyle\left\langle\frac{\partial L}{\partial w}\big|_ {w_ {i}}\right\rangle_ {D_ {i}}}}\ {{w_ {i+1}}}&{{:=}}&{{w_ {i}+v_ {i+1}}}\end{array}
$$

Figure 3: 96 convolutional kernels of size $11\times11\times3$ learned by the first convolutional layer on the $224\times224\times3$ input images. The top 48 kernels were learned on GPU 1 while the bottom 48 kernels were learned on GPU 2. See Section 6.1 for details.

图 3: 第一卷积层在 $224\times224\times3$ 输入图像上学习到的96个尺寸为 $11\times11\times3$ 的卷积核。顶部48个核在GPU 1上学习，底部48个核在GPU 2上学习。详情参见第6.1节。

$i$ $v$ $\epsilon$ $\left\langle\frac{\partial L}{\partial w}\Big|_ {w_ {i}}\right\rangle_ {D_ {i}}$ is the average over the $i$ th batch $D_ {i}$ of the derivative of the objective with respect to $w$ , evaluated at $w_ {i}$ .

$i$ $v$ $\epsilon$ $\left\langle\frac{\partial L}{\partial w}\Big|_ {w_ {i}}\right\rangle_ {D_ {i}}$ 表示目标函数关于参数 $w$ 的导数在第 $i$ 个批次 $D_ {i}$ 上、于 $w_ {i}$ 处取值的平均值。

We initialized the weights in each layer from a zero-mean Gaussian distribution with standard deviation 0.01. We initialized the neuron biases in the second, fourth, and fifth convolutional layers, as well as in the fully-connected hidden layers, with the constant 1. This initialization accelerates the early stages of learning by providing the ReLUs with positive inputs. We initialized the neuron biases in the remaining layers with the constant 0.

我们使用标准差为0.01的零均值高斯分布初始化每层权重。在第二、第四、第五卷积层以及全连接隐藏层中，将神经元偏置初始化为常量1。这种初始化方式通过为ReLU提供正输入来加速学习早期阶段。其余层的神经元偏置初始化为常量0。

We used an equal learning rate for all layers, which we adjusted manually throughout training. The heuristic which we followed was to divide the learning rate by 10 when the validation error rate stopped improving with the current learning rate. The learning rate was initialized at 0.01 and reduced three times prior to termination. We trained the network for roughly 90 cycles through the training set of 1.2 million images, which took five to six days on two NVIDIA GTX 580 3GB GPUs.

我们对所有层使用相同的学习率，并在训练过程中手动调整。遵循的经验法则是：当验证错误率在当前学习率下停止改善时，将学习率除以10。初始学习率设为0.01，在训练终止前共降低三次。我们在包含120万张图像的训练集上进行了约90轮训练，使用两块NVIDIA GTX 580 3GB显卡耗时五至六天。

6 Results

6 结果

Our results on ILSVRC-2010 are summarized in Table 1. Our network achieves top-1 and top-5 test set error rates of $37.5%$ and $17.0%^{5}$ . The best performance achieved during the ILSVRC2010 competition was $47.1%$ and $28.2%$ with an approach that averages the predictions produced from six sparse-coding models trained on different features [2], and since then the best published results are $45.7%$ and $25.7%$ with an approach that averages the predictions of two classifiers trained on Fisher Vectors (FVs) computed from two types of densely-sampled features [24].

我们在ILSVRC-2010上的结果总结在表1中。我们的网络实现了$37.5%$的top-1测试集错误率和$17.0%^{5}$的top-5测试集错误率。ILSVRC2010竞赛期间的最佳表现是$47.1%$和$28.2%$，这是通过对在不同特征上训练的六个稀疏编码模型的预测结果取平均得到的[2]。此后发表的最佳结果是$45.7%$和$25.7%$，这是通过对从两种密集采样特征计算得到的Fisher向量(FVs)训练的两个分类器的预测结果取平均得到的[24]。

We also entered our model in the ILSVRC-2012 competition and report our results in Table 2. Since the ILSVRC-2012 test set labels are not publicly available, we cannot report test error rates for all the models that we tried. In the remainder of this paragraph, we use validation and test error rates interchangeably because in our experience they do not differ by more than $0.1%$ (see Table 2). The CNN described in this paper achieves a top-5 error rate of $18.2%$ . Averaging the predictions

我们也将模型提交至ILSVRC-2012竞赛，结果如表2所示。由于ILSVRC-2012测试集标签未公开，无法汇报所有尝试模型的测试错误率。本段余下内容中，验证错误率与测试错误率可互换使用（两者差异不超过$0.1%$，见表2）。本文描述的CNN取得了$18.2%$的top-5错误率。通过预测平均...

Model	Top-1	Top-5
Sparse coding[2]	47.1%	28.2%
SIFT+FVs [24]	45.7%	25.7%
CNN	37.5%	17.0%

模型	Top-1	Top-5
Sparse coding[2]	47.1%	28.2%
SIFT+FVs [24]	45.7%	25.7%
CNN	37.5%	17.0%

Table 1: Comparison of results on ILSVRC2010 test set. In italics are best results achieved by others.

表 1: ILSVRC2010测试集结果对比。斜体表示其他方法取得的最佳结果。

of five similar CNNs gives an error rate of $16.4%$ . Training one CNN, with an extra sixth convolutional layer over the last pooling layer, to classify the entire ImageNet Fall 2011 release (15M images, 22K categories), and then “fine-tuning” it on ILSVRC-2012 gives an error rate of $16.6%$ . Averaging the predictions of two CNNs that were pre-trained on the entire Fall 2011 release with the aforementioned five CNNs gives an error rate of $15.3%$ . The second-best contest entry achieved an error rate of $26.2%$ with an approach that averages the predictions of several class if i ers trained on FVs computed from different types of densely-sampled features [7].

五个相似的CNN (Convolutional Neural Network) 组合的错误率为 $16.4%$。在最后一个池化层上增加第六个卷积层，训练单个CNN对整个ImageNet 2011秋季版(1500万图像，2.2万类别)进行分类，然后在ILSVRC-2012上"微调(fine-tuning)"，其错误率为 $16.6%$。将两个在整个2011秋季版上预训练的CNN与前述五个CNN的预测结果取平均，错误率降至 $15.3%$。竞赛第二名的方案通过平均多个分类器(基于不同类型密集采样特征计算的FV[7])的预测结果，取得了 $26.2%$ 的错误率。

Finally, we also report our error rates on the Fall 2009 version of ImageNet with 10,184 categories and 8.9 million images. On this dataset we follow the convention in the literature of using half of the images for training and half for testing. Since there is no established test set, our split necessarily differs from the splits used by previous authors, but this does not affect the results appreciably. Our top-1 and top-5 error rates on this dataset are $67.4%$ and

最后，我们还报告了在2009年秋季版ImageNet数据集上的错误率，该版本包含10,184个类别和890万张图像。在此数据集上，我们遵循文献惯例，将一半图像用于训练，另一半用于测试。由于没有固定的测试集划分，我们的分割方式必然与之前作者采用的方案不同，但这不会显著影响结果。我们在该数据集上的top-1和top-5错误率分别为$67.4%$和

Table 2: Comparison of error rates on ILSVRC-2012 validation and test sets. In italics are best results achieved by others. Models with an asterisk* were “pre-trained” to classify the entire ImageNet 2011 Fall release. See Section 6 for details.

Model	Top-1 (val)	Top-5 (val)	Top-5 (test)
SIFT+FVs [7]			26.2%
1 CNN	40.7%	18.2%
5 CNNs	38.1%	16.4%	16.4%
1CNN*	39.0%	16.6%
7CNNs*	36.7%	15.4%	15.3%

表 2: ILSVRC-2012验证集和测试集的错误率对比。斜体表示其他方法取得的最佳结果。带星号* 的模型经过"预训练"可对整个ImageNet 2011 Fall版本进行分类。详见第6节。

模型	Top-1 (验证)	Top-5 (验证)	Top-5 (测试)
SIFT+FVs [7]			26.2%
1 CNN	40.7%	18.2%
5 CNNs	38.1%	16.4%	16.4%
1CNN*	39.0%	16.6%
7CNNs*	36.7%	15.4%	15.3%

$40.9%$ , attained by the net described above but with an additional, sixth convolutional layer over the last pooling layer. The best published results on this dataset are $78.1%$ and $60.9%$ [19].

$40.9%$，由上述网络实现，但在最后一个池化层上额外增加了第六个卷积层。该数据集上已发布的最佳结果为$78.1%$和$60.9%$[19]。

6.1 Qualitative Evaluations

6.1 定性评估

Figure 3 shows the convolutional kernels learned by the network’s two data-connected layers. The network has learned a variety of frequency- and orientation-selective kernels, as well as various colored blobs. Notice the specialization exhibited by the two GPUs, a result of the restricted connectivity described in Section 3.5. The kernels on GPU 1 are largely color-agnostic, while the kernels on on GPU 2 are largely color-specific. This kind of specialization occurs during every run and is independent of any particular random weight initialization (modulo a renumbering of the GPUs).

图 3: 展示了网络两个数据连接层学习到的卷积核。该网络已习得多种频率和方向选择性核 (kernel) ，以及各种彩色斑点。请注意两块 GPU 呈现出的专业化特征，这是 3.5 节所述受限连接性的结果。GPU 1 上的核基本与颜色无关，而 GPU 2 上的核则主要针对特定颜色。这种专业化现象在每次运行中都会出现，且与任何特定的随机权重初始化无关 (除 GPU 编号变化外) 。

Figure 4: (Left) Eight ILSVRC-2010 test images and the five labels considered most probable by our model. The correct label is written under each image, and the probability assigned to the correct label is also shown with a red bar (if it happens to be in the top 5). (Right) Five ILSVRC-2010 test images in the first column. The remaining columns show the six training images that produce feature vectors in the last hidden layer with the smallest Euclidean distance from the feature vector for the test image.

图 4: (左) 八张ILSVRC-2010测试图像及我们的模型认为最可能的五个标签。每张图像下方标注了正确标签，并以红色条形图显示正确标签的概率(若其位于前五)。(右) 第一列为五张ILSVRC-2010测试图像，其余列展示了与测试图像在最后一隐藏层特征向量欧氏距离最小的六张训练图像。

In the left panel of Figure 4 we qualitatively assess what the network has learned by computing its top-5 predictions on eight test images. Notice that even off-center objects, such as the mite in the top-left, can be recognized by the net. Most of the top-5 labels appear reasonable. For example, only other types of cat are considered plausible labels for the leopard. In some cases (grille, cherry) there is genuine ambiguity about the intended focus of the photograph.

在图4的左面板中，我们通过计算网络对八张测试图像的前5预测结果，定性评估了网络所学内容。值得注意的是，即使是不居中的物体(如左上角的螨虫)也能被网络识别。大多数前5标签看起来是合理的。例如，对于豹子图像，只有其他类型的猫科动物被视为合理标签。在某些情况下(格栅、樱桃)，照片的实际焦点确实存在歧义。

Another way to probe the network’s visual knowledge is to consider the feature activation s induced by an image at the last, 4096-dimensional hidden layer. If two images produce feature activation vectors with a small Euclidean separation, we can say that the higher levels of the neural network consider them to be similar. Figure 4 shows five images from the test set and the six images from the training set that are most similar to each of them according to this measure. Notice that at the pixel level, the retrieved training images are generally not close in L2 to the query images in the first column. For example, the retrieved dogs and elephants appear in a variety of poses. We present the results for many more test images in the supplementary material.

另一种探究网络视觉知识的方法是分析图像在最后一个4096维隐藏层所激发的特征激活。若两幅图像生成的特征激活向量欧氏距离较小，则表明神经网络高层认为它们相似。图4展示了测试集中的五幅图像，以及根据该度量标准与之最相似的六幅训练集图像。值得注意的是，在像素级别上，检索到的训练图像与首列查询图像的L2距离通常并不接近。例如，检索到的狗和大象呈现多种姿态。我们在补充材料中展示了更多测试图像的实验结果。

Computing similarity by using Euclidean distance between two 4096-dimensional, real-valued vectors is inefficient, but it could be made efficient by training an auto-encoder to compress these vectors to short binary codes. This should produce a much better image retrieval method than applying autoencoders to the raw pixels [14], which does not make use of image labels and hence has a tendency to retrieve images with similar patterns of edges, whether or not they are semantically similar.

通过计算两个4096维实值向量之间的欧氏距离来衡量相似度效率较低，但可以通过训练自编码器（auto-encoder）将这些向量压缩为短二进制码来提高效率。这种方法应能产生比将自编码器应用于原始像素[14]更好的图像检索方法，后者未利用图像标签，因此倾向于检索具有相似边缘模式的图像，无论它们在语义上是否相似。

7 Discussion

7 讨论

Our results show that a large, deep convolutional neural network is capable of achieving recordbreaking results on a highly challenging dataset using purely supervised learning. It is notable that our network’s performance degrades if a single convolutional layer is removed. For example, removing any of the middle layers results in a loss of about $2%$ for the top-1 performance of the network. So the depth really is important for achieving our results.

我们的结果表明，一个大型深度卷积神经网络(Convolutional Neural Network)能够在极具挑战性的数据集上仅通过纯监督学习就取得破纪录的成绩。值得注意的是，如果移除单个卷积层，我们的网络性能就会下降。例如，移除任何中间层都会导致网络top-1性能损失约$2%$。因此，深度确实对我们的成果至关重要。

To simplify our experiments, we did not use any unsupervised pre-training even though we expect that it will help, especially if we obtain enough computational power to significantly increase the size of the network without obtaining a corresponding increase in the amount of labeled data. Thus far, our results have improved as we have made our network larger and trained it longer but we still have many orders of magnitude to go in order to match the infero-temporal pathway of the human visual system. Ultimately we would like to use very large and deep convolutional nets on video sequences where the temporal structure provides very helpful information that is missing or far less obvious in static images.

为简化实验，我们未采用任何无监督预训练，尽管预期其将带来提升，尤其是在获得足够算力大幅扩展网络规模却无法同步增加标注数据量的情况下。目前结果表明，随着网络规模扩大和训练时长增加，性能持续提升，但要匹配人类视觉系统的颞下通路仍需数个数量级的改进。最终目标是构建超大规模深度卷积网络处理视频序列，其时间结构能提供静态图像中缺失或极不明显的关键信息。

References

参考文献