[论文翻译]单图像超分辨率的细粒度关注和特征共享生成对抗网络


下载PDF:https://arxiv.org/pdf/1911.10773v2.pdf

下载代码:https://github.com/Rainyfish/FASRGAN-and-Fs-SRGAN.

Fine-grained Attention and Feature-sharing Generative Adversarial Networks for Single Image Super-Resolution

细粒度的注意力和特征共享生成的单图像超分辨率

author

Yitong Yan, Chuangchuang Liu, Changyou Chen, Xianfang Sun, Longcun Jin*, Xinyi Peng, and Xiang Zhou

Abstract

Traditional super-resolution (SR) methods by minimize the mean square error usually produce images with over-smoothed and blurry edges, due to the lack of high-frequency details. In this paper, we propose two novel techniques within the generative adversarial network framework to encourage generation of photo-realistic images for image super-resolution. Firstly, instead of producing a single score to discriminate real and fake images, we propose a variant, called Fine-grained Attention Generative Adversarial Network (FASRGAN), to discriminate each pixel of real and fake images. FASRGAN adopts a UNet-like network as the discriminator with two outputs: an image score and an image score map. The score map has the same spatial size as the HR/SR images, serving as the fine-grained attention to represent the degree of reconstruction difficulty for each pixel. Secondly, instead of using different networks for the generator and the discriminator, we introduce a feature-sharing variant (denoted as Fs-SRGAN) for both the generator and the discriminator. The sharing mechanism can maintain model express power while making the model more compact, and thus can improve the ability of producing high-quality images. Quantitative and visual comparisons with state-of-the-art methods on benchmark datasets demonstrate the superiority of our methods. We further apply our super-resolution images for object recognition, which further demonstrates the effectiveness of our proposed method. The code is available at https://github.com/Rainyfish/FASRGAN-and-Fs-SRGAN.

摘要

传统超分辨率(SR)通过最小化均线误差通常产生具有过平滑的图像和模糊边缘,由于缺乏高频细节。在本文中,我们提出了一种在生成的对抗网络框架内提出了两种新颖的技术,以鼓励为图像超分辨率产生照片 - 现实图像。首先,不是制作单一分数来区分实际和假图像,我们提出了一种变种,称为细粒度关注生成的对抗网络(Fasrgan),以区分实际和假图像的每个像素。 Fasrgggn 采用一个像 unet-like 的网络作为具有两个输出的鉴别器:图像分数和图像分数图。得分图具有与 HR / SR 图像相同的空间尺寸,用作致细粒的注意力,以代表每个像素的重建难度。其次,而不是使用不同网络的发电机和鉴别器,我们将发电机和鉴别器介绍一个特征共享变量(表示为 FS-SRGAN)。共享机制可以在使模型更紧凑的同时保持模型表达功率,因此可以提高生产高质量图像的能力。在基准数据集上具有最先进方法的定量和视觉比较,证明了我们方法的优越性。我们进一步应用我们的超分辨率图像进行对象识别,进一步展示了我们所提出的方法的有效性。代码:https:/github.com/rainyfish/fasrgan-and-fs-srgan。

Introduction

image super-resolution (SISR), which aims to recover a high-resolution (HR) image from its low-solution (LR) version, has been an active research topic in computer graphic and vision for decades. SISR has also attracted increasing attention in both academia and industry, with applications in various fields such as medical imaging, security surveillance, object recognition and so on. However, SISR is a typically ill-posed problem due to the irreversible image degradation process, , multiple HR images can be generated from one single LR image. Learning the mapping between HR and LR images plays an important role in addressing this problem. Recently, deep convolution neural networks (CNNs) have been shown great success in many vision tasks, such as image classification, object detection, and image restoration. Dong et al. first proposed a three-layer CNN for single image super-resolution (SRCNN) to directly learn the mapping from LR to HR images. Since then the CNN-based methods have been dominant for the SR problem because they greatly improved the reconstruction performance. Kumar et al. tapped into the ability of polynomial neural networks to hierarchically learn refinements of a function that maps LR to HR patches. VDSR obtained remarkable performance by increasing the depth of the network to 20, proving the importance of the network depth for detecting effective features of images. EDSR removed unnecessary batch normalization layer in the ResNet architecture and widened the channels, significantly improving the performance. RCAN applied residual in residual structure to construct a very deep network and used a channel attention mechanism to adaptively rescale features. The aforementioned methods use the optimization idea of minimizing the mean squared error (MSE) between the recovered SR image and the corresponding HR image. Such methods are designed to maximize the peak signal-to-noise ratio (PSNR). However, they typically produce over-smoothed edges and lack tiny textures. To produce photo-realistic SR images, Ledig et al. first used the generative adversarial network (GAN) to match the underlying distributions of HR and SR images. ESRGAN further extended the generator network and used the Relativistic Discriminator to produce more photo-realistic images. However, as shown in Fig.GAN_based, the discriminator in these GAN-based methods only outputs a score of the whole input SR/HR image, which is a coarse way to guide the generator. Furthermore, the generator and discriminator of these works are considered to be independent, while we believe there should be significant information to be shared. For example, the lower-level parts of the two networks both aim at extracting tiny features such as corners and edges, which we believe should be correlated.

简介

单图像超分辨率(SISR)旨在从其低分辨率(LR)版本恢复高分辨率(HR)图像,数十年来一直是计算机图形和视觉领域的活跃研究主题。SISR 在医学成像,安全监控,物体识别等各个领域的应用也引起了学术界和工业界的越来越多的关注。但是,由于不可逆的图像降级过程,SISR 通常是一个不均衡的问题,即可以从一个单一的 LR 图像生成多个 HR 图像。学习 HR 和 LR 图像之间的映射在解决此问题方面起着重要作用。

最近,深度卷积神经网络(CNN)在许多视觉任务(例如图像分类,目标检测和图像恢复)中已显示出巨大的成功。董等。[ dong2015image ]首先提出了一种用于单图像超分辨率(SRCNN)的三层 CNN,以直接学习从 LR 到 HR 图像的映射。从那时起,基于 CNN 的方法 [ 9044873 ]一直是 SR 问题的主导,因为它们极大地提高了重建性能。Kumar 等。 [ kumar2016fast ]充分利用了多项式神经网络的功能,可以分层学习将 LR 映射到 HR 补丁的函数的细化。VDSR [kim2016accurate ]通过增加网络的深度 20,证明用于检测图像的有效特征的网络深度的重要性取得了显着的性能。EDSR [ lim2017enhanced ]删除了 ResNet [ he2016deep ]体系结构中不必要的批处理规范化层, 并拓宽了渠道,从而显着提高了性能。RCAN [ zhang2018image ]在残差结构中应用了残差以构建非常深的网络,并使用通道注意机制来自适应地重新缩放特征。

前述方法使用使恢复的 SR 图像和对应的 HR 图像之间的均方误差(MSE)最小的优化思想。此类方法旨在最大程度地提高峰值信噪比(PSNR)。但是,它们通常会产生过度平滑的边缘,并且缺少微小的纹理。为了产生照片般逼真的 SR 图像,Ledig 等人。[ ledig2017photo ]首先使用生成对抗网络(GAN) [ goodfellow2014generative ]来匹配 HR 和 SR 图像的基本分布。ESRGAN [ wang2018esrgan ]进一步扩展了生成器网络,并使用了相对论判别器 [ jolicoeur-martineau2018]产生更多逼真的图像。但是,如图 1 所示,这些基于 GAN 的方法中的鉴别器仅输出整个输入 SR / HR 图像的分数,这是引导生成器的粗略方法。此外,这些作品的产生者和区分者被认为是独立的,而我们认为应该分享重要的信息。例如,两个网络的较低层部分都旨在提取微小的特征,例如角和边缘,我们认为应该将它们关联起来。

To address these limitations, we propose two novel techniques based on the GAN framework for image super-resolution, a fine-grained attention mechanism for the discriminator and a feature-sharing network component for both the generator and the discriminator. Specifically, we use a UNet-like discriminator (Fig.FASRGAN_D) to introduce a fine-grained attention in the GAN (FASRGAN). Our discriminator produces two outputs, a score of the whole input image and a fine-grained score map of every pixel in the image. The score map shares the same spatial size as the input image, and measures the degree of differences at each pixel between the generated and the true distributions. To produce better visual quality images, we incorporate the score map into the loss function with an attention mechanism to make the generator pay more attention on the hard parts of an image, instead of treating all parts equally. In addition, we propose a feature-sharing mechanism (Fig.Co-SRGAN) to align the low-level feature extraction of both the generator and the discriminator (Fs-SRGAN). This novel structure can significantly reduce the number of parameters and improve the performance. Overall, our main contributions are three-fold: \item We propose a novel UNet-like discriminator to generate a single score for the whole image and a pixel-wise score map of the input image. We further incorporate the score map into the loss function with an attention mechanism to define the generator. This attention mechanism makes the generator focus on hard parts of an image for generation. \item We introduce a feature-sharing mechanism to define the shared low-level feature extraction for the generator and the discriminator. This reduces the number of model parameters and helps the generator and the discriminator extract more effective features. \item The proposed two components are general, and can be applied to other GAN-based SR models. %In this paper, we also combine the two techniques into one model. Extensive experiments on benchmark datasets illustrate the superiority of our proposed methods compared with current state-of-the-art methods. The remainder of the paper is organized as follows. Section II describes related works. The proposed GAN-based methods are presented in Section III. Experimental results are discussed in Section IV. Finally, the conclusions are drawn in Section V.

image
图 1:基于 GAN 的超分辨率方法的体系结构。生成器旨在重建逼真的 SR 图像,而鉴别器则将 SR 图像与真实的 HR 图像区分开。

为了解决这些局限性,我们提出了两种基于 GAN 框架的图像超分辨率新技术,一种用于鉴别器的细粒度关注机制,以及一种用于生成器和鉴别器的特征共享网络组件。具体来说,我们使用类似 UNet 的[ ronneberger2015u ]鉴别器(图 2),以在 GAN(FASRGAN)中引入细粒度的关注。我们的鉴别器产生两个输出,一个是整个输入图像的得分,另一个是图像中每个像素的细粒度得分图。分数图与输入图像共享相同的空间大小,并测量所生成的分布与真实分布之间的每个像素的差异程度。为了产生更好的视觉质量图像,我们将得分图结合到具有关注机制的损失函数中,以使生成器更加关注图像的硬部分,而不是平等对待所有部分。此外,我们提出了一种功能共享机制(图[3]),以对齐生成器和鉴别器(Fs-SRGAN)的低级特征提取。这种新颖的结构可以显着减少参数数量并提高性能。

总体而言,我们的主要贡献是三方面的:

  • 我们提出了一种新颖的类似于 UNet 的鉴别器,以生成整个图像的单个分数以及输入图像的像素级分数图。我们进一步将得分图结合到具有关注机制的损失函数中,以定义生成器。这种关注机制使生成器专注于生成图像的硬部分。
  • 我们引入了一种特征共享机制来定义生成器和鉴别器的共享低级特征提取。这减少了模型参数的数量,并有助于生成器和鉴别器提取更有效的特征。
  • 所提议的两个组件是通用的,并且可以应用于其他基于 GAN 的 SR 模型。在基准数据集上进行的大量实验表明,与当前的最新技术相比,我们提出的方法具有优越性。

在本文的其余部分安排如下。第二节介绍相关工作。第三节介绍了基于 GAN 的建议方法。实验结果在第四节中讨论。最后,在第五节中得出结论。

Traditional SISR methods are exemplar or dictionary based. However, these methods are limited by the size of datasets or dictionaries, and are usually time-consuming. These shortcomings can be greatly alleviated by the recent CNN-based methods [9044873].

In their pioneer work, Dong et al. [dong2015image] applied convolutional neural networks with three layers for SISR, namely SRCNN, to learn a mapping from LR to HR images in an end-to-end manner. Kim et al. [kim2016accurate] increases the depth of the network to 20, achieving great improvement in accuracy compared to SRCNN. Instead of using the interpolated LR images as the inputs of network, FSRCNN [dong2016accelerating] extracted features from the origin LR images and upscaled the spatial size by upsampling layers at the tail of the network. This architecture is widely used in the subsequent SR methods. Various advanced upsampling structures have been proposed recently, for instance, deconvolutional layer [zeiler2010deconvolutional] , sub-pixel convolution [shi2016real], and EUSR [kim2018deep]. LapSRN [lai2017deep] progressively reconstructed an HR image with increasing scales of an input image by the Laplacian pyramid structure. Lim et al. [lim2017enhanced] proposed a very large network (EDSR) and its multi-scale version (MDSR), which removed the unnecessary batch normalization layer in the ResNet [he2016deep] and greatly improved super-resolution performance. D-DBPN [haris2018deep] introduced an error-correcting feedback mechanism to learn relationships between LR features and SR features. ZSSR [shocher2018zero] used a unsupervised method to learn the mapping between HR images and LR images. DIP [ulyanov2018deep] showed that the structure of a generator network can capture a large amount of low-level image statistics before any learning is performed, which can be used as a handcrafted prior with excellent results in super-resolution and other standard inverse problems. To address the real-world LR image problem, Fritsche et al. [fritsche2019frequency] proposed to separate the low and high image frequencies and treat them in different ways during training. Adversarial training is used to modify only the high, not the low frequencies. RDN [zhang2018residual] combined dense and residual connections to make full use of information of LR images. Different from RDN, MS-RHDN [liu2019multi] proposed multi-scale residual hierarchical dense networks to extract multi-scale and hierarchical feature maps. Yang et al. [yang2018drfn] proposed a deep recurrent fusion network (DRFN) for SR with large-scale factors, which used transposed convolution to jointly extract and upsample raw features from the input and used multi-level fusion for reconstruction. SeaNet [fang2020soft] proposed a Soft-edge assisted Network to reconstruct the high-quality SR images with the help of image soft-edge. Zhang et al. [zhang2020adaptive] proposed an adaptive importance learning scheme to enhance the performance of the lightweight SISR network architecture. RCAN [zhang2018image] applied channel-attention mechanism to adaptively rescale channel-wise features. SAN [Dai_2019_CVPR] further proposed a second-order channel attention (SOCA) module to rescale the features instead of global average pooling.

传统的 SISR 方法是基于示例或字典的。但是,这些方法受数据集或字典大小的限制,通常很耗时。这些缺点可以通过最近的基于 CNN 的方法来大大缓解 [ 9044873 ]。

在他们的开创性工作中,Dong 等人[ dong2015image ]将具有三层的卷积神经网络用于 SSR,即 SRCNN,以端对端的方式学习从 LR 到 HR 图像的映射。Kim 等。 [ kim2016accurate ]将网络的深度增加到 20,与 SRCNN 相比,准确性得到了极大的提高。FSRCNN [ dong2016accelerating ]而不是将插值的 LR 图像用作网络的输入。 从原始 LR 图像中提取特征,并通过对网络尾部的图层进行上采样来扩大空间大小。此体系结构在随后的 SR 方法中被广泛使用。最近已经提出了各种先进的上采样结构,例如,反卷积层 [ zeiler2010deconvolutional ],子像素卷积 [ shi2016real ]和 EUSR [ kim2018deep ]。LapSRN [ lai2017deep ]通过拉普拉斯金字塔结构,随着输入图像比例的增加,逐渐重建了 HR 图像。Lim 等。 [ lim2017enhanced ]提出了一个超大型网络(EDSR)及其多尺度版本(MDSR),该网络删除了 ResNet [ he2016deep ]中不必要的批处理规范化层, 并极大地提高了超分辨率性能。D-DBPN [ haris2018deep ]引入了一种纠错反馈机制,以了解 LR 功能和 SR 功能之间的关系。ZSSR [ shocher2018zero ]使用一种无​​监督的方法来学习 HR 图像和 LR 图像之间的映射。DIP [ ulyanov2018deep ]结果表明,生成器网络的结构可以在进行任何学习之前捕获大量的低级图像统计信息,可以用作手工制作的方法,在超分辨率和其他标准逆问题中具有出色的效果。为了解决现实世界中的 LR 图像问题,Fritsche 等人。 [ fritsche2019frequency ]建议将低图像频率和高图像频率分开,并在训练期间以不同的方式对其进行处理。对抗训练仅用于修改高频,而不能修改低频。RDN [ zhang2018residual ]结合了密集连接和残余连接以充分利用 LR 图像的信息。与 RDN 不同,MS-RHDN [ liu2019multi ]提出了多尺度残差分层稠密网络,以提取多尺度和层次特征图。Yan [yang2018drfn]提出了一种具有大规模因子的 SR 深度递归融合网络(DRFN),该算法利用转置卷积从输入中共同提取和上采样原始特征,并使用多级融合进行重构。SeaNet [ fang2020soft ]提出了一种软边缘辅助网络,借助图像软边缘来重建高质量的 SR 图像。[zhang2020adaptive]提出了一种自适应重要性学习方案,以增强轻量级 SISR 网络体系结构的性能。RCAN[ zhang2018image ]应用了通道注意机制来自适应地重新缩放通道级特征。SAN [ Dai_2019_CVPR ]进一步提出了一个二阶通道关注(SOCA)模块,以重新调整功能而不是全局平均池。

The aforementioned methods aim to achieve high PSNR and SSIM values. However, these criteria usually cause heavy over-smoothed edges and artifacts. Images generated by these MSE-based SR methods lose various high-frequency details and have a bad perceptual quality. To generate more photo-realistic images, Ledig et al. firstly introduced generative adversarial network into image super-resolution, called SRGAN. SRGAN combined a perceptual loss and an adversarial loss to improve the reality of generated images. But some visually implausible artifacts still could be found in some generated images. To reduce the artifacts, EnhanceNet combined a pixel-wise loss in the image space, a perceptual loss in the feature space, a texture matching loss and an adversarial loss. The contextual loss was a kind of perceptual loss to make the generated images as similar as possible to ground-truth images. Yan et al. firstly trained a novel full-reference image quality assessment (FR-IQA) approach for SISR, then employed the proposed loss function (SR-IQA) to train their SR network which contains their proposed highway unit. In addition, they also integrated SR-IQA loss to the GAN-based SR method to achieve better results for both accuracy and perceptual quality. Based on SRGAN, ESRGAN $ RN{1} $ substituted the standard residual block with a residual-in-residual dense block, $ RN{2} $ removed batch normalization layers, $ RN{3} $ utilized VGG feature before activated as perceptual loss, and $ RN{4} $ replaced the standard discriminator with Relativistic Discriminator proposed in RaGAN. Noteworthily, ESRGAN won the first place in the 2018 PIRM Challenge on Perceptual Image Super-Resolution, which pursued the high perceptual-quality images. RankSRGAN firstly trained a ranker to learn the behavior of perceptual metrics and then introduced a rank-content loss to optimize the perceptual quality.

image
图 2: FASRGAN 的鉴别器架构,其中 K,S,G 分别代表 Conv 层的内核大小,步幅和过滤器编号。FC 代表完全连接的层。面具是一张得分图[0,1 个],弥补了图像中每个像素重建的困难。

上述方法旨在实现高 PSNR 和 SSIM 值。然而,这些标准通常会导致重型过平滑的边缘和伪影。基于 MSE 的 SR 方法生成的图像失去了各种高频细节,并且具有糟糕的感知质量。生成更多照片逼真的图像,LEDIG *等。*首先将生成的对抗网络引入图像超分辨率,称为 SRAN。 SRGAN 联合感知损失和对抗的丧失丧失,以改善生成的图像的现实。但是一些视觉难以置信的伪像在一些生成的图像中仍然可以找到。为了减少伪影,增强 eneT 组合在图像空间中的像素明显的损失,特征空间中的感知损失,纹理匹配损失和越野损失。语境损失是一种感知损失,使生成的图像尽可能相似地与地面真相类似。 Yan 首先培训了 SISR 的新型全参考图像质量评估(FR-IQA)方法,然后雇用了拟议的损失函数(SR-IQA)培训他们的 SR 网络,其中包含其所提出的公路单位。此外,它们还将 SR-IQA 损失集成到基于 GAN 的 SR 方法,以实现准确性和感知质量的更好结果。基于 SRGAN,ESRGAN $RN{1} $ 用 residual-in-residual dense block 代替标准残差块,$ RN{2} $拆除批量归一化层,$RN{3} $在激活为感知损失之前使用 VGG 功能,以及$ RN{4} $用 Ragan 中提出的相对论判别器取代了标准鉴别器。值得注意的是,Esrgan 获得了 2018 年 PIRM 挑战感知形象超分辨率的第一名,生成了高感知形象。 RanksRggan 首先训练了一个 Ranker 来学习感知度量的行为,然后介绍了排名内容损失以优化感知质量。

Proposed Methods 建议方法

Overview 概述

Our methods aim to reconstruct a high-resolution image $ I^{\text{SR}}\in R^{Wr\times Hr\times C} $ from a low-resolution image $ I^{\text{LR}}\in R^{W\times H\times C} $ , where $ W $ and $ H $ are the width and height of the LR image, $ r $ is the upscaling factor, and $ C $ is the number of channels of the color space. This section details our two strategies within the GAN framework for image super-resolution in order: FASRGAN and Fs-SRGAN. Specifically, we propose a fine-grained attention mechanism in FASRGAN to make the generator focus on the difficult parts of image reconstruction based on the score map from the UNet-like discriminator, instead of treating each part equally. We further propose a feature-sharing mechanism in Fs-SRGAN by sharing the low-level feature extractor of the generator and the discriminator. Both networks update the gradient of the shared low-level feature extractor in the training phase, which could make the model more compact while keeping enough representation power. These two strategies contribute to the overall perceptual quality for SR, respectively. For simplicity, we use the same network architecture as ESRGAN for the generator to generate the SR images from the input LR images.

我们的方法旨在从低分辨率图像$ I^{\text{LR}}\in R^{W\times H\times C} $ 中重建高分辨率图像 $ I^{\text{SR}}\in R^{Wr\times Hr\times C} $ ,其中$W $和$H $是宽度和高度 LR 图像,$r $是 Upsoging 因子,$C $是颜色空间的通道数。本节按顺序详细介绍了 GAN 框架中用于图像超分辨率的两种策略:FASRGAN 和 Fs-SRGAN。具体来说,我们在 FASRGAN 中提出了一种细粒度的关注机制,以使生成器基于类似 UNet 的鉴别器的得分图,专注于图像重建的困难部分,而不是平等地对待每个部分。通过共享生成器和鉴别器的低级特征提取器,我们进一步提出了 Fs-SRGAN 中的特征共享机制。两个网络都在训练阶段更新共享的低级特征提取器的梯度,这可以使模型更紧凑,同时保持足够的表示能力。这两种策略分别有助于提高 SR 的整体感知质量。为简单起见, [ wang2018esrgan ]用于生成器从输入的 LR 图像生成 SR 图像。

Fine-grained Attention Generator Adversarial Networks 细粒度的注意力生成器对抗网络

Our proposed fine-grained attention GAN (FASRGAN) designs a specific discriminator for SISR. As discussed above and shown in Fig.GAN_based, the discriminator in a standard GAN-based SR model outputs a score of the whole input SR/HR image. This can be considered as a coarse way to judge an input image and cannot discriminate local features of inputs. To tackle this problem, the proposed FASRGAN defines a UNet-like discriminator contained two outputs, corresponding to a score of the whole image and a fine-grained score map. The score map has the same size as the input image and is used for pixel-wise discrimination. The proposed discriminator is illustrated in Fig.FASRGAN_D.

我们提出的细粒度注意 GAN(FASRGAN)设计了 SISR 的特定区分符。如上文所讨论并如图 1 所示,基于 GAN 的标准 SR 模型中的鉴别器输出整个输入 SR / HR 图像的分数。这可以被认为是判断输入图像的粗略方法,并且不能区分输入的局部特征。为了解决这个问题,提出的 FASRGAN 定义了一个类似 UNet 的鉴别器,其中包含两个输出,分别对应于整个图像的分数和细粒度的分数图。得分图具有与输入图像相同的大小,并用于按像素区分。所提出的鉴别器在图 2 示出 。

A UNet-like Discriminator 类似于 UNet 的鉴别器

The UNet-like discriminator with two outputs can be divided into two parts: an encoder and a decoder. Similar to the standard discriminator D in ESRGAN, the encoder part of the proposed UNet-like discriminator uses a standard max-pooling layer with a stride of 2 to reduce the spatial size of a feature map and increase receptive fields. At the same time, the number of channels is increased for improving representative ability. At the end of the encoder, two fully connected layers are added to output a score, measuring the overall probability of an input image $ x $ being real or fake. We further enhance the discriminator based on the Relativistic GAN, which has also been used in ESRGAN. The loss function is defined as:

具有两个输出的类 UNet 鉴别器可以分为两部分:编码器和解码器。

编码器。 与 ESRGAN 中的标准鉴别器 D 相似,拟议的类似 UNet 的鉴别器的编码器部分使用步长为 2 的标准最大池化层来减小特征图的空间大小并增加接收场。同时,增加了通道数以提高代表能力。在编码器的末尾,添加了两个完全连接的层以输出分数,从而测量输入图像的总体概率 X 是真实的还是假的。我们基于相对论 GAN [ jolicoeur-martineau2018 ]进一步增强了鉴别器,该相对论 也已在 ESRGAN [ wang2018esrgan ]中使用。损失函数定义为:

$$
L_{adv}^D= D_{Ra}(x_r,x_f)+D_{Ra}(x_f,x_r)= \mathbb{E}[\log(1-\sigma(C(x_r)-\mathbb{E}[C(x_f)]))] + \mathbb{E}[\log(\sigma(C(x_f)-\mathbb{E}[C(x_r)]))],
$$

$$
L_{adv}^D = \mathbb{E}_{x_r}[\log(1-D_{Ra}(x_r,x_f))]+ \mathbb{E}_{x_f}[\log(D_{Ra}(x_f,x_r))]= \mathbb{E}_{x_r}[\log(1-\sigma(C(x_r)-\mathbb{E}_{x_f}[C(x_f)]))]+ \mathbb{E}_{x_f}[\log(\sigma(C(x_f)-\mathbb{E}_{x_r}[C(x_r)]))],
$$

image.png

where $ x_r $ and $ x_f $ stand for the ground-truth image and the generated SR image, respectively. $ D_{Ra}(\cdot) $ refers to the function of the relativistic discriminator, which tries to predict the probability that a real image $ x_r $ is more realistic than a fake one $ x_f $ ; $ C(x) $ is the discriminator output before sigmoid function and $ \sigma $ is the sigmoid function. We exploit an upsampling layer to gradually extend the spatial size of feature maps, followed by two convolutional layers to extract more information. To make full use of features, we concatenate the previous outputs from the encoder, which have the same spatial size as current ones. As shown in Fig.FASRGAN_D, the feature maps at the end of the decoder have the same spatial size as input images. Finally, we use the sigmoid function to produce a score map $ M\in R^{Wr\times Hr \times C} $ that provides pixel-wise discrimination between real and fake pixels of an input image. The fine-grained adversarial loss function $ L_{M}^D $ for the discriminator is defined as: where $ M_r $ and $ M_f $ represent the score maps of the HR image and the generated SR image, respectively. Finally, the loss function for the discriminator is defined as:

其中$x_r $和$x_f $分别代表真实图像和生成的 SR 图像。 $D_{Ra}(\cdot) $是指相对论鉴别器的功能,这试图预测真实图像$x_r $比假一个$x_f $更真实的概率; $C(x) $是 SIGMOID 函数之前的鉴别器输出,$\sigma $是 SIGMOID 函数。我们利用上采样层逐渐扩展特征贴图的空间大小,然后是两个卷积层来提取更多信息。为了充分利用功能,我们将先前的 Encoder 输出连接,其具有与当前的空间大小相同。如图 FASRGAN_D 所示,解码器末尾的特征映射具有与输入图像相同的空间尺寸。最后,我们使用 SIGMOID 函数来产生得分图$M\in R^{Wr\times Hr \times C} $,其提供输入图像的实际和虚假像素之间的像素明智的辨别。用于鉴别器的细粒逆势损失函数$L_{M}^D $定义为:其中$M_r $和$M_f$分别表示 HR 图像的得分图和生成的 SR 图像。最后,鉴别器的损耗函数定义为:

$ L^D = L^D_{adv} + L^D_M $ .

image

Generator Objective Function

In the GAN-based SR methods, the generator is generally used to generate the SR images from the LR images. ESRGAN introduced Residual-in-Residual Dense Block (RRDB) without batch normalization as the basic network building unit, which is of higher capacity and easier to train compared with the ResBlock in SRGAN. In this paper, we also adopt RRDB to construct our generator for a fair comparison with ESRGAN. The generator is trained by several losses, defined as following: Following, we use an $ L_1 $ loss function to constrain the content of a generated SR image to be close to the HR image. The loss is defined in Eq.L1. where $ F_\theta^G(\cdot) $ represents the function of the generator, $ \theta $ is the parameters of the generator and $ I_i $ means the $ i $ -th image. The perceptual loss aims to make the SR image close to the corresponding HR image based on high-level features extracted from a pre-trained network. Similar to , we consider both the SR and HR images as the input to the pre-trained VGG19 and extract the VGG19-54 layer features. The perceptual loss is defined as:

$$ L_{percep} = \parallel F_{\theta}^{VGG}(G(I_i^{LR}))-F_{\theta}^{VGG}(I_i^{HR})\parallel_1, $$

where $ F_{\theta}^{VGG}(\cdot) $ is the function of VGG and $ I_i $ is the $ i $ -th image, $ G(\cdot $ ) is the function of the generator. The discriminator contains two outputs, a whole estimation of the entire image and the pixel-wise fine-grained estimations of an input image. The total adversarial loss function for the generator is defined as:

$$ L_{adv}^G = L_{entire}^G+L_{fine}^G, $$

As shown in Eq.mask_d, the discriminator tries to distinguish the real and fake image in a fine-grained way, while the generator aims to fool the discriminator. Thus the fine-grained adversarial loss function $ L^G_{fine} $ for the generator is the symmetrical form of Eq.mask_d: $ L_{entire}^G $ is also the symmetrical form of Eq.discriminator_adversarial and defined as:
$$
L_{entire}^G=\mathbb{E}[\log(\sigma(C(x_r)-\mathbb{E}[C(x_f)]))]
+\mathbb{E}[\log(1-\sigma(C(x_f)-\mathbb{E}[C(x_r)]))].
$$

The score map generated by the UNet-like discriminator is represented as pixel-wise discrimination scores of an input image, with values $ M(w,h,c) $ among $ [0,1] $ . A higher score means the corresponding pixel of the input image is closer to that of the ground-truth image. In this manner, the score map can indicate which parts of an image are more difficult to generate and which parts are easier. For instance, the structure background part of an image is sometimes simpler, and thus it would expect the discriminator reflects this to the generator when updating the generator. In other words, the part with lower scores (more difficult to generate) should receive more attention when updating the generator. As a result, we incorporate the score map as the fine-grained attention mechanism into a $ L_1 $ loss function, constituting a weighted attention loss function:

$$
L_{attention}=\frac{1}{Wr\times Hr \times C}\sum_{w=1}^{Wr}\sum_{h=1}^{Hr}\sum_{c=1}^C (1-M_f(w,h,c))
\times \parallel F_{\theta}^{G} (I_i^{LR})(w,h,c)-I_i^{HR}(w,h,c),
$$

where $ M_f(w,h,c) $ is the score map of the generated image given by the discriminator. Instead of treating every pixel of an image equally, $ L_{attention} $ contributes to pay more attention in the hard-to-recovered part of an image, such as the textures with rich semantic information. Combining the above losses with different weights, the total loss of the generator is:
$$ \label{eq:generator_loss} L^{G}= L_{percep} + \lambda_1 L_{adv}^G + \lambda_2 L_{attention} + \lambda_3 L_1, $$
where $ \lambda_1 $ , $ \lambda_2 $ , $ \lambda_3 $ are the coefficients to balance different loss terms.

发生器物镜函数

在基于 GAN 的 SR 方法中,发电机通常用于从 LR 图像生成 SR 图像。 ESRGAN 引入了残留的残余密集块(RRDB),无批量标准化为基本网络建筑单元,与 SRAN 中的 RESBLOCK 相比,该容量更高,更容易训练。在本文中,我们还通过 RRDB 构建我们的发电机,以便与 ESRAN 进行公平比较。发电机由几个损耗培训,定义如下:以下,我们使用$L_1 $丢失功能来限制生成的 SR 图像的内容以接近 HR 图像。损失在 EQ.L1 中定义。其中$F_\theta^G(\cdot) $表示发电机的功能,$\theta $是发电机的参数,$I_i $表示$i $ -th 图像。感知损失旨在使 SR 图像基于从预先训练的网络中提取的高级功能,使得靠近相应的 HR 图像。类似于,我们将 SR 和 HR 图像视为预先训练的 VGG19 的输入并提取 VGG19-54 层特征。感知损失定义为:
$$ \ label {eq:percep} l_ {percep} = \ parents f _ {\ theta} ^ {vgg}(g(i_i ^ {lr})) - f _ {\ theta} ^ { vgg}(i_i ^ {hr})\ parallel_1,$$
其中$F_{\theta}^{VGG}(\cdot) $是 vgg 和$I_i $的功能是$i $ -th 图像,$G(\cdot $)是函数发电机。鉴别器包含两个输出,整个图像的整体估计和输入图像的像素明智的细粒估计。发电机的总体对冲丢失函数定义为:
$$ \ label {eq:avd} l_ {adv} ^ g = {entire} ^ g + l_ {fine} ^ g,$$
,如 eq.mask_d 所示,鉴别者试图以细粒度的方式区分真实和假图像,而发电机则旨在欺骗鉴别者。因此,发电机的细粒度逆势丢失函数$L^G_{fine} $是 eq.mask_d:$L_{entire}^G $的对称形式也是 eq.discriminator_addersarial 的对称形式,并定义为:

$$ \ label {eq:ground_adv }
L_{entire}^G=\mathbb{E}[\log(\sigma(C(x_r)-\mathbb{E}[C(x_f)]))]
+\mathbb{E}[\log(1-\sigma(C(x_f)-\mathbb{E}[C(x_r)]))].
$$

由非唯一鉴别器生成的分数图表示为输入图像的像素 - 方向辨别得分,其中$[0,1] $中值$M(w,h,c)$。较高的分数是指输入图像的相应像素更靠近地面图的图像。以这种方式,得分图可以指示图像的哪些部分更难以产生,并且哪个部件更容易。例如,图像的结构背景部分有时是更简单的,因此在更新发电机时,它将期望鉴别器将其反射到发电机。换句话说,在更新发电机时,具有较低分数(更难以生成)的部件应更加关注。因此,我们将分数图作为细长的注意机制纳入$L_1 $损耗功能,构成了加权注意力损失功能:

$$
L_{attention}=\frac{1}{Wr\times Hr \times C}\sum_{w=1}^{Wr}\sum_{h=1}^{Hr}\sum_{c=1}^C (1-M_f(w,h,c))
\times \parallel F_{\theta}^{G} (I_i^{LR})(w,h,c)-I_i^{HR}(w,h,c)\parallel _1,
\label{EQ:ATTEN}
$$

其中$M_f(w,h,c) $是通过鉴别器给出的生成图像的分数映射。代替将图像的每个像素同样处理,$L_{attention} $有助于在图像的难以恢复的部分中提供更多关注,例如具有丰富语义信息的纹理。将上述损失与不同权重结合,发电机的总损失是:
$$ \ label {eq:generator_loss} l ^ {g} = l_ {percep} + lambda_1 l_ {adv} ^ g + \ lambda_2 l_ {attention} + \ lambda_3 l_1,$$
其中$\lambda_1 $,$\lambda_2 $,$\lambda_3 $是平衡不同损耗术语的系数。

Feature-sharing Generator Adversarial Networks

In the standard GANs, the generator and the discriminator are usually defined as two independent networks. Based on the observation that the shallow parts of the two networks always extract low-level textures such as edges and corners, we propose a new network structure (Fs-SRGAN) to enable low-level feature sharing between the generator and the discriminator. This can reduce the number of parameters and help both networks extract more effective features. Consequently, our Fs-SRGAN contains three parts: a shared feature extractor, a generator, and a discriminator, as shown in Fig.Co-SRGAN.

功能共享生成发生器对冲网络

在标准 GAN 中,发电机和鉴别器通常被定义为两个独立网络。基于观察到两个网络的浅部分总是提取边缘和角落等低级纹理,我们提出了一种新的网络结构(FS-SRGAN),以在发电机和鉴别器之间启用低级功能共享。这可以减少参数的数量并帮助两个网络提取更有效的功能。因此,我们的 FS-SRGAN 包含三个部分:共享特征提取器,发电机和鉴别器,如图所示。

Shared Feature Extractor

We first use a share feature extractor to transform an input image from color space to feature space, before extracting low-level feature maps. The feature-sharing mechanism allows the generator and the discriminator to jointly optimize the low-level feature extractor. Similar to FASRGAN, we adopt RRDB, the basic block of ESRGAN, as the basic structure. The shared feature extractor contains $ E $ RRDBs to extract helpful feature maps for both the generator and the discriminator, described as following:
$$ \label{eq:shared_part} H_{shared} = F_{shared}(x), $$
where $ H_{shared} $ is the low-level feature maps extracted by the shared part, $ F_{shared} $ represents the function of the shared feature extractor, and $ x $ is the input. For the generator, the input is an LR image, while for the discriminator it is a SR image or a HR image. Considering the input sizes of the generator and the discriminator are different, we apply a fully Convolutional Neural Network with invariant size of feature maps to extract features so that the different input sizes do not matter.

shared feature extractor

我们首先使用共享特征提取器将输入图像从颜色空间转换为特征空间,然后提取低级特征映射。特征共享机制允许发电机和鉴别器共同优化低级特征提取器。类似于 Fasrgan,我们采用 RRDB,ESRGAN 的基本块,作为基本结构。共享特征提取器包含$E $ rrdbs,用于提取有关的发电机和鉴别器的有用特征映射,如下所示:
$$ \ label {eq:shared_pa​​rt} h_ {shared} = f_ {shared}(x),$$
其中$H_{shared} $是由共享部分提取的低级特征映射,$F_{shared} $表示共享特征提取器的功能,而$x $是输入。对于发电机,输入是 LR 图像,而对于鉴别器,它是 SR 图像或 HR 图像。考虑到发电机的输入大小和鉴别器是不同的,我们应用了一个完全卷积的神经网络,具有不变大小的特征映射来提取功能,以便不同的输入大小无关紧要。

The Generator and the Discriminator

The rest parts of the generator and the discriminator are the same as those in standard GAN-based methods, except that the inputs are feature maps instead of images as shown in Fig.Co-SRGAN. The generator contains three parts: low-level feature extraction, deep feature extraction, and reconstruction, which are used for transforming the input image to the feature space from the color space and extracting low-level information, extracting high-level semantic features and reconstructing SR image, respectively. Note that the generator in our Fs-SRGAN only contains the latter two parts. Similar to the shared low-level feature extraction, we adopt RRDB as the basic part of deep feature extraction, except that more RRDBs are used to increase the depth of the network with the purpose of extracting more high-frequency feature for reconstruction. The reconstruction part utilizes an upsampling layer to upscale the high-level feature maps and a Conv layer to reconstruct an SR image. The loss function of the generator is same as that of ESRGAN, which includes perceptual loss, adversarial loss, and pixel-based loss:
$$ \label{eq:fs_generator} L^{G}= L_{percep} + \lambda_1 L_{adv}^G + \lambda_2 L_1, $$
where $ \lambda_1 $ , $ \lambda_2 $ are the coefficients to balance different loss terms, $ L_{percep} $ and $ L_1 $ are defined in Eq.percep and Eq.L1, respectively, $ L_{adv}^G $ is the adversarial loss with the same definition as $ L^G_{entire} $ in Eq.entire_adv. Because the discriminator is a classification network that distinguishes the input as SR or HR image, we apply a structure similar to the VGG network as the discriminator. To reduce information loss, we substitute the pooling layer (used in the encoder of the UNet-discriminator) for a Conv layer with a stride of 2 to decrease the size of feature map. At the tail of the discriminator, we use a Conv layer to transform the feature map into a one-dimensional vector, then use two fully connected layers to output the classification score $ s $ among $ [0, 1] $ . The value of $ s $ closer to 1 means more real, otherwise more fake. The loss function of the discriminator is same as $ L^D_{adv} $ defined in Eq.discriminator_adversarial.

生成器和鉴别器

发电机的静止部分和鉴别器的基于标准 GaN 的方法的静止部分相同,不同之处在于输入是特征映射而不是图中所示的图像 SRGAN。发电机包含三个部分:低级特征提取,深度特征提取和重建,用于从颜色空间转换到要素空间的输入图像并提取低级信息,提取高电平语义特征和重建 SR 图像分别。请注意,我们的 FS-SRGAN 中的发电机只包含后两部分。类似于共享低级特征提取,我们采用 RRDB 作为深度特征提取的基本部分,除了使用更多的 RRDB 来增加网络的深度,目的是提取更多的高频功能来重建。重建部分利用上采样层来高档高级特征映射和经常层以重建 SR 图像。发电机的损耗功能与 ESRAN 的损失功能相同,包括感性损失,对抗性丢失和基于像素的损失:
$$ \label{eq:fs_generator} l ^ {g} = l_ {percep} + \ lambda_1 l_ {adv} ^ g + \ lambda_2 l_1,$$
,其中$\lambda_1 $,$\lambda_2 $是平衡不同损失术语的系数,$L_{percep} $和$L_1 $在 eq.percep 和 eq.l1 中定义,分别,$L_{adv}^G $是具有与 eq.entire_adv 的$L^G_{entire} $相同的对手丢失。因为鉴别器是将输入作为 SR 或 HR 图像的分类网络,所以我们将类似于 VGG 网络的结构应用于鉴别器。为了减少信息丢失,我们将汇集层(在 UNET-Coldiminator 的编码器中使用)替换为 CONC 层,其步幅为 2,以减小特征图的大小。在鉴别器的尾部,我们使用经常层将特征映射转换为一维向量,然后使用两个完全连接的层输出$[0, 1] $中的分类得分$s $。 $s $更接近 1 的值意味着更真实,否则更假。鉴别器的损耗函数与在 eq.discriminator_addersarial 中定义的$L^D_{adv} $相同。

Experimental Results

In this section, we first describe our model training details, then provide quantitative and visual comparisons with several state-of-the-art methods on benchmark datasets for our two proposed novel methods, FASRGAN and Fs-SRGAN. We further combine the fine-grained attention and the feature-sharing mechanisms into one single model, termed FA $ + $ Fs-SRGAN.

Dataset Metric EnhanceNet SRGAN ESRGAN RankSRGAN FASRGAN Fs-SRGAN FA+Fs-SRGAN
PSNR 28.57 29.91 30.46 29.73 30.15 30.28 30.19
SSIM 0.8102 0.8510 0.8515 0.8398 0.8450 0.8588 0.8571
Set5 PI 2.8466 3.4322 3.5463 2.9867 3.1685 3.9143 3.7455
LPIPS 0.0488 0.0389 0.0350 0.0348 0.0325 0.0330 0.0344
PSNR 23.54 24.39 24.36 24.49 24.51 24.55 24.67
SSIM 0.6933 0.7309 0.7341 0.7319 0.7380 0.7509 0.7466
Urban100 PI 3.4543 3.4814 3.7312 3.3253 3.5173 3.5940 3.5819
LPIPS 0.0777 0.0693 0.0591 0.0667 0.0588 0.0591 0.0625
PSNR 24.94 25.50 25.32 25.51 25.41 25.61 25.87
SSIM 0.6266 0.6528 0.6514 0.6530 0.6523 0.6726 0.6747
BSD100 PI 2.8467 2.3054 2.4150 2.0768 2.2783 2.4056 2.3749
LPIPS 0.0982 0.0887 0.0798 0.0850 0.0796 0.0801 0.0855
PSNR 27.28 28.16 28.17 28.10 28.15 28.15 28.23
SSIM 0.7460 0.7753 0.7759 0.7710 0.7768 0.7903 0.7891
DIV2K val PI 3.4953 3.1619 3.2572 3.0130 3.1034 3.3303 3.3092
LPIPS 0.0753 0.0605 0.0550 0.0576 0.0539 0.0542 0.0576
PSNR 25.47 25.61 25.18 25.65 25.38 25.75 26.00
SSIM 0.6569 0.6757 0.6596 0.6726 0.6648 0.6907 0.6934
PIRM val PI 2.6762 2.2254 2.5548 2.0183 2.2476 2.3311 2.2482
LPIPS 0.0838 0.0718 0.0714 0.0675 0.0685 0.0651 0.0677
Quantitative results with the Bicubic degradation model for $4\times$ SR. Best and second best results are highlighted

实验结果

在本节中,我们首先描述了我们的模型培训细节,然后提供了对我们两个提出的新颖方法,Fasrgan 和 FS-SRGAN 的基准数据集的数量和视觉比较。我们进一步将细粒度的注意力和特征共享机制结合在一起,称为 FA $+ $ FS-SRGAN。

Training Details

In training, we use the training set from DIV2K as the training set to train our models. The LR images are obtained by bicubic downsampling (BI) from the source high-resolution images. Images are augmented by rotating and flipping. We also randomly crop $ 48\times48 $ patches from LR images as the input of the network. Our networks are optimized with the ADAM optimizer. The hyper-parameters $ \beta_1 $ and $ \beta_2 $ in the ADAM optimizer are set to $ \beta_1 = 0.9 $ and $ \beta_2=0.999 $ . The batch size is set to 16. The generator is pre-trained by the $ L_1 $ loss function, followed by generator and the discriminator training with the corresponding loss functions. Following, the initial learning rate is set to $ 1\times10^{-4} $ , and then decays to half every $ 2\times10^5 $ iterations. In FASRGAN, the coefficients in Eq.generator_loss are set as $ \lambda_1 = 5e $ - $ 3 $ , $ \lambda_2 = 1e $ - $ 2 $ and $ \lambda_3 = 1e $ - $ 2 $ . Similar to ESRGAN, the number of RRDBs in the generator is set as 23. In Fs-SRGAN, we set the number of RRDBs in the shared feature extractor as $ E=1 $ , and in the deep feature extractor as 16. The coefficients in Eq.fs_generator are set as $ \lambda_1 = 5e $ - $ 3 $ and $ \lambda_2 = 1e $ - $ 2 $ . In FA+Fs-SRGAN, the number of RRDBs in the share part is set as 2, while in the deep feature extraction part is 15. The discriminator and the coefficients of the loss function are the same as those of FASRGAN. We implement our models with the PyTorch framework on two NVIDIA GeForce RTX 2080Ti GPUs.

培训详情

在培训中,我们使用 DIV2K 的培训设置为培训我们的模型。通过来自源高分辨率图像的双向下采样(BI)获得 LR 图像。通过旋转和翻转来增强图像。我们还随机从 LR 图像随机作物$48\times48 $修补程序作为网络的输入。我们的网络与 ADAM Optimizer 进行了优化。 ADAM 优化器中的超参数$\beta_1 $和$\beta_2 $设置为$\beta_1 = 0.9 $和$\beta_2=0.999 $。批量大小设置为 16.发电机由$L_1 $损耗功能预先培训,然后是发电机和具有相应损耗功能的鉴别器训练。以下,初始学习速率设置为$1\times10^{-4} $,然后衰减为每一个$2\times10^5 $迭代。在 Fasrgan 中,EQ.Generator_Loss 中的系数被设置为$\lambda_1 = 5e $-$3 $,$\lambda_2 = 1e $-$2 $和$\lambda_3 = 1e $-$2 $。类似于 ESRAN,发电机中的 RRDB 的数量设定为 23.在 FS-SRGAN 中,我们将共享特征提取器中的 RRDB 中的 RRDB 数量设置为$E=1 $,深度特征提取器为 16. eq 中的系数。fs_generator 被设置为$\lambda_1 = 5e $-$3 $和$\lambda_2 = 1e $-$2 $。在 FA + FS-SRGAN 中,共享部分中的 RRDB 的数量设置为 2,而在深度特征提取部分中为 15.判别器和损失函数的系数与 FasRGAN 的判别器相同。我们使用两个 NVIDIA GeForce RTX 2080TI GPU 的 Pytorch 框架实现我们的模型。

Datasets and Evaluation Metrics

In the testing phase, we use seven standard benchmark datasets to evaluate the performance of our proposed methods: Set5, Set14, BSD100, Urban100, Manga109, DIV2K validation, PIRM validation and test dataset. Blau et al. proved mathematically that perceptual quality is not always improved with the increase of PSNR value and there is a trade-off between average distortion and perceptual quality. Hence, we not only use PSNR and SSIM to measure the reconstruction accuracy, but also adopt the learned perceptual image patch similarity (LPIPS) and perceptual index (PI) to evaluate the perceptual quality of SR images. LPIPS firstly adopts a pre-trained network $ \mathcal{F} $ to extract patches
$ y, y_0 $ from the reference and target images $ x, x_0 $ . The network $ \mathcal{F} $ computes the activations of the image patches, each is scaled by a learned weight $ w_l $ and then sums up the $ L_2 $ distances across all layers. Finally, it computes a perceptual real/fake prediction as follows:
$$
\label{eq:LPIPS} d(x, x_0) = \sum_l \frac{1}{H_lW_l}\sum_{h,w}\parallel w_l \odot(y_l^{hw} - \hat{y}^{hw}_{0l})\parallel_2^2,
$$
where $ y_l, y_{0l}\in\mathbb{R}^{H_l \times W_l \times C_l} $ represent the reference or target features from layer $ l $ . The HR images from the public datasets are regarded as the reference images, the SR images generated by our methods or the compared methods as the target images. We use the public codes and pre-trained network (AlexNet from version 0.0) for evaluation. While PI is based on the non-reference image quality measures of Ma et al. and NIQE: PI= $ \frac{1}{2} $ ((10-Ma)+NIQE). PSNR and SSIM are calculated on the luminance channel in the YCbCr color space. We also use LPIPS and root mean square error (RMSE) to measure the trade-off between perceptual quality and reconstruction accuracy. Using LPIPS/RMSE rather than LPIPS/PSNR to evaluate the trade-off is for better observation, where lower LPIPS/RMSE means a better result. Higher PSNR/SSIM and lower RMSE mean better results in reconstruction accuracy, while lower scores of LPIPS/PI imply that the images are more similar to the HR images. As shown in Fig.78004, the SR image of our FASRGAN has less artifacts than that of ESRGAN and is clearer than that of RankSRGAN. But the PI value of the SR image produced by RankSRGAN is lower than that of our FASRGAN, and even lower than that of the original HR image. In terms of LPIPS, our method attains the lowest value, which is more consistent with human observation. Hence, we use LPIPS as our first perceptual quality metric and PI as the second one.

image

数据集和评估指标

在测试阶段,我们使用七个标准基准数据集来评估我们所提出的方法的性能:SET5,SET14,BSD100,URBAN100,MANGA109,DIV2K 验证,PIRM 验证和测试数据集。 Blau *等。*在数学上证明,随着 PSNR 价值的增加,不断提高了感知质量,平均失真与感知质量之间存在权衡。因此,我们不仅使用 PSNR 和 SSIM 来测量重建准确性,而且还采用学习的感知图像贴片相似度(LPIP)和感知指数(PI)来评估 SR 图像的感知质量。 LPIPS 首先采用预先训练的网络$\mathcal{F} $来从参考和目标图像$x, x_0 $中提取补丁$y, y_0 $。网络$\mathcal{F} $计算图像修补程序的激活,每个激活由学习权重$w_l $缩放,然后在所有层上施加$L_2 $距离。最后,它计算了一个感知真/假的预测。$y_l, y_{0l}\in\mathbb{R}^{H_l \times W_l \times C_l} $表示来自图层$l $的参考或目标特征。来自公共数据集的 HR 图像被视为参考图像,由我们的方法或比较方法作为目标图像生成的 SR 图像。我们使用公共代码和预先训练的网络(0.0 版本的 AlexNet)进行评估。虽然 PI 基于 MA *等人的非参考图像质量测量。*和 NIQE:PI = $\frac{1}{2} $((10-MA)+ NIQE)。 PSNR 和 SSIM 在 YCBCR 颜色空间中的亮度通道上计算。我们还使用 LPIP 和均方根误差(RMSE)来测量感知质量与重建准确性之间的权衡。使用 LPIPS / RMSE 而不是 LPIPS / PSNR 来评估权衡是为了更好的观察,下 LPIP / RMSE 意味着更好的结果。较高的 PSNR / SSIM 和较低的 RMSE 意味着改善重建精度的结果,而 LPIPS / PI 的较低分数意味着图像与 HR 图像更类似于图像。如图 78004 所示,我们的 FasRggan 的 SR 图像具有比 Esrgan 的少于伪像,而且比 RanksRAN 的更清晰。但是 RankSrgan 产生的 SR 图像的 PI 值低于我们的 FasRgan,甚至低于原始 HR 图像的 SR 图像。就 LPIP 而言,我们的方法达到了最低值,这与人类观察更符合。因此,我们使用 LPIPS 作为我们的第一个感知质量指标和 PI 作为第二个。

Quantitative Comparisons

We present the quantitative comparisons between our methods and the state-of-the-art perceptual image SR methods on several benchmark datasets. As shown in Tablequantitative, in most cases, RankSRGAN obtains the lowest PI values among these methods, benefiting from using the loss function with the newly added ranker to optimize the generator. However, our FASRGAN obtains the best LPIPS on most datasets, and both the LPIPS and PI values are better than that of ESRGAN, whose structure of the generator is the same as ours. It demonstrates that the fine-grained attention in our FASRGAN can transfer more information to the generator to produce better results. With less RRDBs in the generator, our Fs-SRGAN obtains best SSIM results and comparable, sometimes even better LPIPS results than those of ESRGAN and FASRGAN. In other words, our Fs-SRGAN extracts features more effectively and efficiently, benefiting from the feature-sharing mechanism. The combined model FA+Fs-SRGAN obtains the highest PSNR except for Set5, indicating that it can recover more contents in the SR images. We also compare our methods with the state-of-the-art methods on the trade-off between reconstruction accuracy and visual quality. The results are shown in Fig.PI_RMSE. Methods in the top-left part are almost MSE-based with low RMSE and high LPIPS scores due to the over-smoothed edges and lack of high-frequency details. The bottom-right category includes the GAN-based methods, such as SRGAN, EnhanceNet, ESRGAN, RankSRGAN, and our methods. These methods usually gain high-visual quality images even if their RMSE values are larger than those of the MSE-based methods. Our FASRGAN gets better visual quality and reconstruction accuracy compared with EnhanceNet, SRGAN and ESRGAN, and lower LPIPS than RankSRGAN. Our Fs-SRGAN attains comparable LPIPS with ESRGAN but lower RMSE, and better visual quality and reconstruction accuracy than RankSRGAN. The combined model FA+Fs-SRGAN obtains the lowest RMSE among the GAN-based methods. These demonstrate the effectiveness of our fine-grained attention and feature-sharing mechanism. To further demonstrate the effectiveness of our FASRGAN and Fs-SRGAN, we conduct a user study to calculate the Mean Opinion Score (MOS) against the state-of-the-art SR methods, i.e. SRGAN, ESRGAN and RankSRGAN. Ten candidates are shown with a side-by-side comparison of the generated SR image and the corresponding ground-truth. They are then asked to evaluate the difference of the two images on a 5-level scale defined as: 0 - almost identical, 1 - mostly similar, 2 - similar, 3 - somewhat similar and 4 - mostly different. We randomly select 10 images from PIRM val dataset, and invite 10 participants to give a score on each image according to the 5-level scale. For a better comparison, one small patch from the image is zoomed in. The average scores of all images are considered as the final results. As suggested in TableMos, our FASRGAN and Fs-SRGAN achieve better performance than all the compared methods, proving the effectiveness of our proposed fine-grained attention and feature-sharing mechanism.

Methods PIRM Val
LPIPS Mos
SRGAN 0.0718 1.98
ESRGAN 0.0714 1.88
RankSRGAN 0.0675 1.84
FASRGAN (ours) 0.0685 1.46
Fs-SRGAN (ours) 0.0651 1.46
The comparison of LPIPS and MOS between our methods and the state-of-the-art methods on PIRM Val, where the lower values mean more similar with the HR image. The LPIPS is tested on the whole dataset, while MOS is calculated on 10 randomly selected images.

定量比较

我们在几个基准数据集中介绍了我们的方法和最先进的感知图像 SR 方法之间的定量比较。如下所示,在大多数情况下,RanksRGAN 在这些方法中获得最低的 PI 值,从使用新添加的 Ranker 使用损耗功能来优化发电机。但是,我们的 FasRggan 获得了大多数数据集的最佳 LPIP,LPIP 和 PI 值都比 ESRGAN 的 LPIP 和 PI 值均优于发电机的结构与我们的结构相同。它表明,我们的 Fasrgan 中的细粒度关注可以将更多信息传送给发电机以产生更好的结果。由于发电机中较少的 RRDB,我们的 FS-SRGAN 获得最佳 SSIM 结果和可比性,有时甚至更好的 LPIPS 结果而不是 ESRAN 和 FASRGAN。换句话说,我们的 FS-SRGAN 更有效,有效地提取功能,从特征共享机制中受益。组合模型 FA + FS-SRGAN 获得除 SET5 之外的最高 PSNR,表示它可以在 SR 图像中恢复更多内容。我们还将我们的方法与最先进的方法进行了重建准确性和视觉质量之间的权衡。结果如图所示。由于过平滑的边缘和缺乏高频细节,左上部分的方法几乎基于低 RMSE 和高 LPIPS 分数。右下类别包括基于 GaN 的方法,例如 SRGAN,ENHANCENET,ESRAN,RANKSRGAN 和我们的方法。即使其 RMSE 值大于基于 MSE 的方法的方法,这些方法通常也会获得高视觉质量图像。与 Enhanceenet,SRANG 和 ESRAN,比 RANKSRAN 相比,我们的 FASRGAN 获得更好的视觉质量和重建准确性。我们的 FS-SRGAN 获得了与 ESRAN 的相当的 LPIP,但较低的 RMSE,以及比 RANKSRGAN 更好的视觉质量和重建准确性。组合模型 FA + FS-SRGAN 获得基于 GAN 的方法中的最低 RMSE。这些展示了我们细粒度关注和特征共享机制的有效性。为了进一步展示我们的 Fasrgan 和 FS-SRGAN 的有效性,我们开展用户学习,以计算针对最先进的 SR 方法的平均意见分数(MOS),即 SRGAN,ESRAN 和 RANKSRAN。显示十个候选者,并以生成的 SR 图像和相应的地面真理的并排比较。然后要求它们在 5 级规模上评估两个图像的差异定义为:0 - “几乎相同”,1 - “大多是相似”,2 - “相似”,3 - “有些类似”和 4 - “大多数不同”。我们从 PIRM Val 数据集中随机选择 10 个图像,并邀请 10 名参与者根据 5 级比例在每个图像上进行分数。有更好的比较,从图像中放大一个小贴片。所有图像的平均分数被视为最终结果。如 Tablemos 所示,我们的 Fasrgan 和 FS-SRGAN 获得比所有比较方法更好的性能,证明了我们提出的细粒度关注和特征共享机制的有效性。

Qualitative Results

We compare our final models on several public benchmark datasets with the state-of-the-art MSE-based methods: SRCNN, EDSR, RDN, RCAN, and GAN-based approaches: SRGAN, EnhanceNet, ESRGAN, RankSRGAN.

定性结果

我们将我们的最终模型与基于最先进的 MSE 的方法进行比较:SRCNN,EDSR,RDN,RCAN 和基于 GAN 的方法:SRGAN,ENHANCENET,ESRAN,RANKSRAN 。

Visual Comparisons of FASRGAN

Some representative quality results are presented in Fig.visual_results_ASRGAN. PSNR , PI and LPIPS are also provided for reference. As shown in Fig.visual_results_ASRGAN, our proposed FASRGAN outperforms previous methods by a large margin. Images generated by FASRGAN contain more fine-grained textures and details. For example, the cropped parts of image 0828 are full of textures. All the compared MSE-based methods suffer from heavy blurry artifacts, failing to recover the structure and the gap of the stripes. SRGAN, EnhanceNet, ESRGAN and RankSRGAN generate high-frequency noise and wrong textures; while our FASRGAN can reduce noise and recover them more correctly, producing more faithful results and being closer to the HR images. For image img_093 in Urban100, the cropped parts of the images generated by the compared methods contain heavily blurry artifacts and lines with wrong directions. Although the LPIPS of ESRGAN is a little lower than our FASRGAN, our FASRGAN can alleviate the artifacts better and recover zebra crossing in the right direction. More results can be seen in the supplemental material. These comparisons demonstrate the strong ability of FASRGAN for producing more photo-realistic and high-quality SR images.

fasrgan 的视觉比较

一些代表性质量结果在图 visup_results_asrgan 中呈现。还提供 PSNR,PI 和 LPIP 参考。如图所示,我们提出的 Fasrgan 优于以前的方法。 Fasrgan 生成的图像包含更细粒度的纹理和细节。例如,图像 0828 的裁剪部分充满了纹理。所有基于 MSE 的方法都遭受重型模糊的伪像,未能恢复条纹的结构和间隙。 SRGAN,ENHANCENET,ESRGAN 和 RANKSRGAN 产生高频噪音和错误的纹理;虽然我们的 Fasrgan 可以降低噪音并更好地恢复它们,产生更忠实的结果并更接近人力资源图像。对于 Urban100 中的图像“IMG_093”,比较方法产生的图像的裁剪部分包含严重模糊的伪像和具有错误方向的线。虽然 ESRGAN 的 LPIPS 比我们的 FASRGAN 低一点,但我们的 FASRGAN 可以更好地缓解伪影,并在正确的方向上恢复斑马交叉。可以在补充材料中看到更多结果。这些比较展示了 Fasran 以生产更多照片现实和高质量 SR 图像的强大能力。

Visual Comparisons of Fs-SRGAN

We also compare our Fs-SRGAN with state-of-the-art methods in Fig.visual_results_Co-SRGAN. Our Fs-SRGAN obtains better performance than other methods in producing SR images, in terms of sharpness and details. For image baboon, the cropped parts of the images generated by the MSE-based methods are over-smoothed. Previous GAN-based methods not only fail to produce clear whiskers but also introduce lots of unpleasing noise. Despite having lower LPIPS value, ESRGAN generates too many whiskers, which have not appeared in the original HR image. While our Fs-SRGAN produces more correct whiskers. For image 0812 , MSE-based methods still suffer from heavy blurry artifacts and generate unnatural results. GAN-based methods cannot maintain the structures of the stairs or the train tracks and introduce artifacts. Our proposed Fs-SRGAN outperforms the compared methods, reducing the artifacts and recovering the correct textures. More results can be seen in the supplemental material. These also indicate that the shared low-level feature extractor of the generator and the discriminator is beneficial.

FS-SRGAN 的可视比较

我们还将我们的 FS-SRGAN 与图 VISUP_RESULTS_CO-SRGAN 中最先进的方法进行比较。在清晰度和细节方面,我们的 FS-SRGAN 获得的性能比其他方法在生产 SR 图像方面。对于图像“狒狒”,基于 MSE 的方法生成的图像的裁剪部分过度平滑。以前的基于 GAN 的方法不仅无法产生清晰的晶须,而且还引入了许多令人难以释放的噪音。尽管 LPIPS 值较低,ESRGAN 产生太多晶须,其尚未出现在原始的 HR 图像中。虽然我们的 FS-SRGAN 生产更正确的晶须。对于图像 0812,基于 MSE 的方法仍然患有沉重的模糊伪像并产生不自然的结果。 GaN 的方法不能维持楼梯的结构或火车轨道并引入伪影。我们提出的 FS-SRGAN 优于比较方法,减少伪影并恢复正确的纹理。可以在补充材料中看到更多结果。这些还表明发电机的共享低级特征提取器和鉴别者是有益的。

Visual Comparisons of FA+Fs-SRGAN

We further present the visual results of our FA+Fs-SRGAN compared with ESRGAN, FASRGAN and Fs-SRGAN. As shown in Fig.FA+Fs-SRGAN, for image 57 and 'OhWareraRettouSeitokai', the results from FA+Fs-SRGAN are better than those of FASRGAN, and contain more correct textures. The PSNR and LPIPS values are both the best for FA+Fs-SRGAN. More results can be seen in the supplemental material. These results illustrate that the combined method can restore more contents for the SR images and obtains comparable or even better visual results compared with FASRGAN and Fs-SRGAN.

FA + FS-SRGAN 的视觉比较

我们进一步提出了与 ESRAN,FASRAN 和 FS-SRGAN 相比我们的 FA + FS-SRGAN 的视觉结果。如图 FA + FS-SRGAN,用于图像 57 和'OHWARERARTERESITOKAI'所示,FA + FS-SRGAN 的结果优于 FASRGAN 的结果,并包含更正确的纹理。 PSNR 和 LPIPS 值都是 FA + FS-SRGAN 的最佳状态。可以在补充材料中看到更多结果。这些结果说明了组合方法可以恢复 SR 图像的更多内容,并与 Fasrgan 和 FS-SRGAN 相比获得可比或甚至更好的视觉结果。

Model Analysis

This section compares the sizes and the time complexity of the generators between our methods and ESRGAN, which use RRDB as the basic block to construct the generators. We are not comparing our methods with SRGAN and RankSRGAN in these aspects as clearly they use 16 Resblocks to build their generators, so their parameters and inference time are less than ours. In the aspect of the numbers of parameters, both ESRGAN and our FASRGAN have 23 RRDBs and 16.7M parameters, while our Fs-SRGAN and FA+Fs-SRGAN have 17 RRDBs and 12.46M parameters in their generators. In the aspect of inference time, we run our models and the public official test code and model from ESRGAN on Urban100 using a machine with 4.2GHz Inter i7 CPU (32G RAM) and Nvidia RTX 2080 platform. We conduct five times of inference on Urban100 and take the mean as the inference time. Our Fs-SRGAN and FA+Fs-SRGAN run much faster than ESRGAN, where Fs-SRGAN has the average time of 0.1377 seconds and FA+Fs-SRGAN 0.1364 seconds, while ESRGAN has 0.3573 seconds. Even our FASRGAN runs a little faster than ESRGAN, with average time 0.3160 seconds. From Tablequantitative we can see that our Fs-SRGAN has comparable or even better results than ESRGAN, which demonstrates the efficiency of our feature-sharing mechanism. Fig.PI_process plots the curves of PI values in the training process of our proposed methods on Set14. We observe that the training process of FASRGAN is more stable and the PI value is the lowest. The average PI value of Fs-SRGAN is higher than FASRGAN. As described in, the deep model has a stronger representation capacity to capture semantic information and reduce unpleasing noises. And as mentioned above, Fs-SRGAN contains fewer RRDBs than FASRGAN. Hence, we speculate that compared with FASRGAN, Fs-SRGAN with fewer RRDBs captures less information for reconstruction but brings more noises, causing higher PI values. FA $ + $ Fs-SRGAN, which combines the fine-grained attention mechanism into Fs-SRGAN, obtains the lower PI values than Fs-SRGAN, which demonstrates the effectiveness of our fine-grained attention mechanism. However, the training of the FA+Fs-SRGAN is not stable, which is the concern we need to focus on in our future work.

image

Evaluation Top-1 error Top-5 error
Bicubic 0.526 0.277
SRCNN 0.464 0.230
FSRCNN 0.488 0.252
SRGAN 0.410 0.191
EnhanceNet 0.454 0.224
ESRGAN 0.334 0.132
RankSRGAN 0.342 0.136
Fs-SRGAN (ours) 0.338 0.136
FA+Fs-SRGAN (ours) 0.337 0.134
FASRGAN (ours) 0.323 0.124
Baseline 0.241 0.071
The result of object recognition between our methods and the state-of-the-art methods for 4 times SR. The baseline uses the original HR image as the input of ResNet-50 model.

型号分析

本节比较了我们的方法与 ESRGAN 之间的发电机的大小和时间复杂度,它使用 RRDB 作为构建发电机的基本块。我们并没有将我们的方法与 SRGAN 和 RANKSRAN 进行比较,因为他们使用 16 个 RESBLOCKS 构建其发电机,因此他们的参数和推理时间少于我们的。在参数数量的方面,ESRAN 和 FASRGAN 都有 23 个额定 RRDB 和 16.7M 参数,而我们的 FS-SRGAN 和 FA + FS-SRGAN 在其发电机中有 17 个 RRDB 和 12.46M 参数。在推理时间的方面,我们使用带有 4.2GHz INTE I7 CPU(32G RAM)和 NVIDIA RTX 2080 平台的机器运行我们的模型和公共官方测试代码和型号。我们对城市 100 进行了五次推断,并将其平均值作为推理时间。我们的 FS-SRGAN 和 FA + FS-SRGAN 比 ESRGAN 更快,FS-SRGAN 的平均时间为 0.1377 秒,FA + FS-SRGAN 0.1364 秒,而 ESRAN 有 0.3573 秒。即使我们的 Fasrggg 也比 ESRGAN 更快地运行,平均时间为 0.3160 秒。从 REBUTQUNITIVATIVE,我们可以看到我们的 FS-SRGAN 具有比 ESRAGAG 的可比性甚至更好,这表明了我们的特征共享机制的效率。图 Pi_Process 在我们在 Set14 上提出的方法的培训过程中绘制 PI 值的曲线。我们观察到 Fasrgan 的培训过程更稳定,PI 值是最低的。 FS-SRGAN 的平均 PI 值高于 FASRGAN。如上所述,深层模型具有更强的表示能力来捕获语义信息并减少令人难以释放的噪声。如上所述,FS-SRGAN 包含比 FASRGAN 更少的 RRDB。因此,我们推测与 Fasrgan 相比,FS-SRGAN 与较少 RRDB 的 FS-SRGAN 捕获更少的重建信息,但带来更多噪音,导致更高的 PI 值。 FA $+ $ FS-SRGAN,将细粒度注意机制与 FS-SRGAN 相结合,获得比 FS-SRGAN 更低的 PI 值,这证明了我们的细粒度关注机制的有效性。但是,FA + FS-SRGAN 的培训并不稳定,这是我们在未来工作中关注的担忧。

Object Recognition Performance

To further demonstrate the quality of our generated SR images, we treat them as a pre-processing step for object recognition. We use the same setting as EnhanceNet and evaluate the object recognition performance with the generated images by our methods and other state-of-the-art methods: SRCNN, FSRCNN, SRGAN, EnhanceNet, ESRGAN, RankSRGAN. We use the pre-trained ResNet-50 on imageNet as an evaluation model and fetch the first 1000 images in ImageNet CLS-LOC validation dataset for evaluation. The test images are first down-sampled by bicubic and then upscaled by our methods and the compared methods. These SR images are then used as inputs to the ResNet-50 model to calculate their Top-1 and Top-5 errors for evaluation. As shown in Tableobject_recognition_SRGAN, both two methods we proposed and the variant FA $ + $ Fs-SRGAN achieve considerable accuracy compared to the state-of-the-art methods. Among these three methods, FASRGAN achieves the lowest Top-1 and Top-5 errors, Fs-SRGAN and FA+Fs-SRGAN obtain comparable results with ESRGAN, demonstrating the effectiveness of both the fine-grained attention and the feature-sharing mechanisms.

Model FA mechanism Fs mechanism
w/o FA FASRGAN w/o Fs Fs-SRGAN
PSNR 25.04 25.26 25.44 25.69
PIRM SSIM 0.6454 0.6523 0.6626 0.6785
Test PI 2.4251 2.1160 2.1420 2.2279
LPIPS 0.0751 0.0718 0.0731 0.0695
The ablation study of fine-grained attention (FA) and feature-sharing (Fs) mechanisms for 4 times super-resolution.

对象识别性能

进一步证明了我们生成的 SR 图像的质量,我们将它们视为用于对象识别的预处理步骤。我们使用与 EnhanceNet 相同的设置,并通过我们的方法和其他最先进的方法评估对象识别性能:SRCNN,FSRCNN,SRGAN,ENHANCENET,ESRGAN,RANKSRGAN。我们将预先训练的 Reset-50 上的 ImageNet 作为评估模型,并在 Imagenet CLS-Loc 验证数据集中获取前 1000 个图像进行评估。首先通过双向采样测试图像,然后通过我们的方法和比较方法升高。然后将这些 SR 图像用作 Reset-50 模型的输入,以计算它们的前 1 个和前 5 个误差进行评估。如 TableObject_recognition_SRGAN 所示,与最先进的方法相比,我们提出的两种方法和 VARIANT FA $+ $ FS-SRGAN 获得了相当大的准确性。在这三种方法中,Fasrgan 获得了最低的前 1 和前 5 个错误,FS-SRGAN 和 FA + FS-SRGAN 获得了 ESRAN 的可比结果,证明了细粒度关注的有效性和特征共享机制。

Ablation Study

To study the effects of the two mechanisms in the proposed methods, we conduct ablation experiments by removing the mechanisms and test the differences, respectively. The quantitative results are illustrated in Tableablation_result, overall visual comparisons are presented in Fig.ablation_FASRGAN, Fig.ablation_Co-SRGAN and the supplemental material. A detailed discussion is provided as follows.

消融研究

在提出的方法中研究两种机制的效果,我们通过消除机制和测试差异来进行消融实验。定量结果在桌面中示出,整体可视化比较在图 3 中呈现,FASRGAN,FIG.ablation_CO-SRGAN 和补充材料。详细讨论如下提供。

Removing the Fine-grained Attention Mechanism

We first remove the fine-grained attention (FA) mechanism in FASRGAN. The attention item is removed from the loss functions of the generator in the model without FA. The coefficients of Eq.generator_loss are set as $ \lambda_1 = 5e $ - $ 3 $ , $ \lambda_2 = 0 $ and $ \lambda_3 = 1e $ - $ 2 $ . The fine-grained adversarial loss functions, $ L^D_M $ and $ L^G_{fine} $ are also removed. The generators of FASRGAN and the model without FA have the same parameters, and the difference lies in the loss function in training. From Tableablation_result we can see that FASRGAN surpasses the model without FA in all metrics. An obvious performance decrease can be observed in Fig.ablation_FASRGAN. For image img_009, the model without FA mechanism introduces some unnatural noises and undesired edges, while FASRGAN can maintain the structure and produce high-quality SR images. For image OL_Lunch, the result from the model without FA mechanism contains more artifacts and noise and the letters cannot be well recognized, while FASRGAN reduces the artifacts and noises, whose result looks closer to the original HR images. The visual analysis indicates the effectiveness of the FA mechanism in removing unpleasant and unnatural artifacts.

删除细粒度注意机制

我们首先在 Fasrgan 中删除细粒度的注意力(FA)机制。从没有 FA 的模型中的发电机的损耗功能中移除了注意项。 eq.generator_loss 的系数被设置为$\lambda_1 = 5e $-$3 $,$\lambda_2 = 0 $和$\lambda_3 = 1e $-$2 $。还删除了细粒越野丧失丢失函数$L^D_M $和$L^G_{fine} $。 Fasrgan 的发电机和没有 FA 的模型具有相同的参数,并且差异在于训练中的损失功能。来自桌面_Result,我们可以看到 Fasrgan 在所有指标中超越了没有 FA 的模型。在图 5 中可以观察到明显的性能下降。对于图像'IMG_009',没有 FA 机制的模​​型引入了一些不自然的噪音和不期望的边缘,而 Fasrgan 可以维持结构并产生高质量的 SR 图像。对于图像'OL_luch',没有 FA 机制的模​​型的结果包含更多的工件和噪音,而字母不能很好地识别,而 FASRGAN 减少了伪影和噪声,其结果看起来更接近原始的 HR 图像。视觉分析表明了 FA 机制在去除令人不愉快和不自然伪影方面的有效性。

Removing the Feature-sharing Mechanism

We remove the feature-sharing (Fs) mechanism, so that the generator and discriminator extract their low-level features separately, but the loss function keeps the same as that of Fs-SRGAN. The discriminator and the generator in our Fs-SRGAN use a shared RRDB to extract low-level features, while in the case the Fs mechanism is removed, different RRDBs are used to extract low-level features for them individually. Tableablation_result shows that Fs-SRGAN has lower LPIPS and higher PSNR/SSIM than the model without Fs mechanism. Fig.ablation_Co-SRGAN presents the results of the model without Fs mechanism and Fs-SRGAN. We can observe that Fs-SRGAN outperforms the model without Fs mechanism by a large margin. The removal of Fs mechanism tends to introduce unpleasant artifacts. For image zebra, by employing the Fs mechanism, Fs-SRGAN can alleviate heavy artifacts and noises, recovering the strips of legs more clearly and correctly. For image 292, our Fs-SRGAN generates more textures of the pane. The above results illustrate the effectiveness of our Fs mechanism.

删除要素共享机制

我们删除了功能共享(FS)机制,以便发电机和鉴别器分别提取它们的低级功能,但损耗函数与 FS 的丢失功能保持相同 SRGAN。我们的 FS-SRGAN 中的鉴别器和生成器使用共享的 RRDB 来提取低级功能,而在移除 FS 机制的情况下,使用不同的 RRDB 来单独为它们提取低级别功能。桌布_Result 显示 FS-SRGAN 具有较低的 LPIP 和更高的 PSNR / SSIM 而不是 FS 机制的模型。 Fig.ablation_Co-SRGAN 显示没有 FS 机制和 FS-SRGAN 的模型的结果。我们可以观察到 FS-SRGAN 以大边距不得不使用 FS 机制的模型。去除 FS 机制倾向于引入令人不快的伪影。对于图像“斑马”,通过采用 FS 机制,FS-SRGAN 可以缓解重型伪影和噪音,更清晰且正确地恢复腿部。对于图像 292,我们的 FS-SRGAN 会产生更多窗格的纹理。以上结果说明了我们的 FS 机制的有效性。

Feature Visualization of Fine-Grained Attention and Feature-sharing Mechanisms

To further verify the effectiveness of our proposed fine-grained attention and feature-sharing mechanisms, we present feature visualizations of the first RRDB of the generator (FASRGAN) and the shared feature extraction part (Fs-SRGAN) in Fig.fea_visualization. FASRGAN reduces the noises in the image img_063 and extracts more texture information compared with the model without attention mechanism. The feature maps from Fs-SRGAN also contain more helpful textures, showing that our proposed methods help the networks extract more useful information.

特征可视化的细粒度关注和功能共享机制

进一步验证我们提出的细粒度关注和功能共享机制的有效性,我们提出了发电机的第一个 RRDB 的功能可视化(FasRgan )和图 5 中的共享特征提取部分(FS-SRGAN)。 FasRgan 减少了图像“IMG_063”中的噪声,并与模型相比提取更具纹理信息,而无需注意机制。 FS-SRGAN 的特征映射还包含更多有用的纹理,显示我们所提出的方法帮助网络提取更有用的信息。

The Block Number of the Feature-sharing Part

To further study the effect of the depth of the shared feature extractor in Fs-SRGAN, we vary the number of RRDBs in both the shared low-level feature extractor and the deep feature extractor, keeping the total number of RRDBs in the generator unchanged. As shown in Fig.change, increasing the number of the shared feature extractor leads to performance reduction and increases the burden of the discriminator due to more parameters which makes the model difficult to train. Among them, E1G16 obtains the best results in both visual quality and reconstruction accuracy.

image

Dataset Metric $\lambda_2 = 0.05$ $\lambda_2 =0.01$ $\lambda_2 =0.005$
PSNR 26.44 26.29 26.41
SSIM 0.7103 0.7040 0.7109
PI 2.7639 2.7008 2.8057
LPIPS 0.0662 0.0640 0.0675
Urban100 PSNR 24.35 24.51 24.31
SSIM 0.7359 0.7380 0.7364
PI 3.5091 3.5173 3.5284
LPIPS 0.0608 0.0588 0.0613
DIV2K val PSNR 28.16 28.15 28.19
SSIM 0.7771 0.7903 0.7803
PI 3.1826 3.3303 3.2378
LPIPS 0.0547 0.0542 0.0560
The ablation study of the coefficient

特征共享部分的块编号

进一步研究了 FS-SRGAN 中的共享特征提取器的深度的效果,我们改变了共享低级特征提取器中的 RRDB 的数量和深度特征提取器,保持发电机中的 RRDB 总数不变。如图更改所示,增加共享特征提取器的数量导致性能降低,并且由于更多参数,增加了鉴别者的负担,这使得模型难以训练。其中,E1G16 获得了视觉质量和重建准确性的最佳结果。

Coefficient of the Fine-Grained Attention Loss in the FASRGAN Generator

We also conduct an ablation study to verify the influences of different coefficient $ \lambda_2 $ of the fine-grained attention loss $ L_{attention} $ in the generator of FASRGAN. We set $ \lambda_2 $ as 0.05, 0.01 and 0.005, while the other settings are kept the same. As shown in Tablecoefficient, the model with $ \lambda_2 = 0.01 $ has the best performance in SSIM and LPIPS on Urban100 and DIV2K val, and achieves comparable results in PSNR and PI. The visual results are shown in the supplemental material. When $ \lambda_2 $ is set too small, the fine-grained feedback from the discriminator has less impact on the generator. And when $ \lambda_2 $ is set too large, the training is unstable for both the generator and discriminator and hard to converge. These results indicate that $ \lambda_2 = 0.01 $ is a good setting in practice, which is used in our FASRGAN.

Dataset Metric SRGAN SRGAN_FA SRGAN_Fs
Set5 PSNR 29.91 29.61 29.66
SSIM 0.8510 0.8437 0.8541
PI 3.4322 3.0651 3.4440
LPIPS 0.0389 0.0341 \0.0368
Set14 PSNR 26.56 26.11 26.27
SSIM 0.7093 0.6977 0.7179
PI 2.8549 2.7550 2.7705
LPIPS 0.0696 0.0692 0.0669
Urban100 PSNR 24.39 24.00 24.04
SSIM 0.7309 0.7205 0.7331
PI 3.4814 \3.4252 3.4818
LPIPS 0.0693 0.0688 0.0691
BSD100 PSNR 25.50 25.35 25.46
SSIM 0.6528 0.6506 0.6650
PI 2.3054 2.2503 2.3348
LPIPS 0.0887 0.0856 0.0876
The effect of fine-grained attention (FA) and feature-sharing (Fs) mechanisms on SRGAN.

Fasrgan 发生器中的细粒度注意力损失的系数

我们还开展了一种消融研究,以验证发电机中细粒度注意力$L_{attention} $的不同系数$\lambda_2$的影响 Fasrgan。我们将$\lambda_2 $设置为 0.05,0.01 和 0.005,而其他设置保持相同。如桌面展示所示,具有$\lambda_2 = 0.01 $的模型在城市 100 和 DIV2K VAL 上的 SSIM 和 LPIP 中具有最佳性能,并在 PSNR 和 PI 中实现了可比的结果。可视结果显示在补充材料中。当$\lambda_2 $设置得太小时,鉴别器的细粒度反馈对发电机的影响较小。当$\lambda_2 $设置得太大时,培训对于发电机和鉴别器且难以收敛,训练不稳定。这些结果表明$\lambda_2 = 0.01 $是在实践中的良好设置,用于我们的 FasRgan。

The Fine-grained Attention and Feature-sharing Mechanisms in SRGAN

To verify whether our proposed fine-grained attention (FA) and feature-sharing mechanisms can improve the performance in other GAN-based SR models, we incorporate these two mechanisms into SRGAN, denoting as SRGAN_FA and SRGAN_Fs respectively. The generator in SRGAN_FA is the same as that of SRGAN, and the discriminator adopts our proposed UNet-like structure. We use a convolution layer and a residual block (RB) as the shared low-level feature extraction part for the generator and discriminator in SRGAN_Fs. The rest part of the generator and the discriminator are similar with that of SRGAN, except that the number of RB in the deep feature extraction is 13. Hence, the parameter for SRGAN_Fs is 1.48K, and 1.554K for SRGAN and SRGAN_FA. As shown in Tablesrgan, SRGAN_FA achieves the best performance in terms of PI and LPIPS on most of the test dataset. SRGAN_Fs also outperforms SRGAN in SSIM and LPIPS. These results demonstrate that our proposed FA and Fs mechanisms can be well adapted to the SRGAN model.

SRGAN 的细粒度关注和功能共享机制

验证我们提出的细粒度注意(FA)和功能共享机制是否可以提高其他基于 GAN 的 SR 模型的性能,我们纳入这两个机制进入 SRGAN,分别表示为 SRGAN_FA 和 SRGAN_FS。 SRGAN_FA 中的发电机与 SRGAN 的发电机相同,鉴别器采用我们提出的杂交结构。我们使用卷积层和残余块(RB)作为 SRGAN_FS 中的发电机和鉴别器的共享低级特征提取部件。发生器和鉴别器的静止部分与 SRGAN 的静态部分类似,除了深度特征提取中的 RB 的数量是 13.因此,SRGAN_FS 的参数为 1.48K,SRGAN 和 SRGAN 的 1.554K _F A。如 TablesRgan 所示,SRGAN_FA 在大多数测试数据集中实现了 PI 和 LPIP 的最佳性能。 SRGAN_FS 还优于 SSIM 和 LPIPS 的 SRGAN。这些结果表明,我们所提出的 FA 和 FS 机制可以适应 SRGAN 模型。

Results on the Real-World Dataset

We also benchmark our proposed methods on a publicly available real-world dataset to test the robustness. We adopt the test set from RealSR(V3) as the dataset and PSNR/SSIM/PI/LPIPS as evaluation metrics. As shown in the top part of Tablerealsr, our FASRGAN and Fs-SRGAN obtain better PI and LPIPS than ESRGAN and comparable results with RankSRGAN, demonstrating that our proposed models have better robustness on real-world LR images. In addition, we used the training set from RealSR(V3) to fine-tune ESRGAN and our proposed methods. Both of them have run about 150k iterations, where the learning rate is initially set as $ 10^{-4} $ and decays a half every 50k iterations. We test the fine-tuned (FT) models on the test set, and also compare them with ZSSR and the work from Fritsche et al. proposed for the AIM 2019 Challenge on Real World, denoted as FSSR. ZSSR is the first unsupervised SR method for non-ideal LR images. The codes and models of FSSR are publicly available, and we adopt the model TDSR of AIM for comparison. As shown in the bottom part of Tablerealsr, our FASRGAN and Fs-SRGAN still obtain better results in LPIPS, indicating that our models are robust on the real-world images. Visual results of the fine-tuned models and the compared methods are also presented in Fig.RealSR. We can observe that the results generated by ZSSR and FSSR are heavily blurred, which brings a bad visual effect. The fine-tuned results from ESRGAN contain some artifacts and noises, resulting in unpleasing observation. While our fine-tuned FASRGAN and Fs-SRGAN reduce the artifacts and produce more pleasing results, demonstrating the robustness of our proposed models.

结果在真实世界数据集

上,我们还将我们的建议方法基准测试在公开的真实数据集上以测试稳健性。我们通过 Realsr(v3)作为数据集和 psnr / ssim / pi / lpips 采用测试集作为评估指标。如 Tablerealsr 的顶部所示,我们的 Fasrgan 和 FS-SRGAN 获得的 PI 和 LPIPS 比 ESRGAN 和 RANKSRAN 的比较结果,表明我们所提出的模型对现实世界的 LR 图像具有更好的鲁棒性。此外,我们使用 Realsr(V3)的培训设置为微调 ESRAN 和我们的提出方法。它们都有大约 150k 迭代,其中学习速率最初被设置为$10^{-4} $并衰减每 50k 迭代的一半。我们在测试集上测试微调(FT)模型,并将它们与 ZSSR 和 Fritsche *等人的工作进行比较。*为现实世界的 AIM 挑战提出,表示为 FSSR。 ZSSR 是第一种用于非理想 LR 图像的无保化 SR 方法。 FSSR 的代码和模型是公开可用的,我们采用了旨在比较的型号 TDSR。如 Tablerealsr 的底部所示,我们的 FasRgan 和 FS-SRGAN 仍然可以获得 LPIP 的更好的结果,表明我们的模型对真实世界的图像具有强大。微调模型的视觉结果和比较方法也在图 4 中呈现。我们可以观察到 ZSSR 和 FSSR 产生的结果严重模糊,这带来了糟糕的视觉效果。 ESRGAN 的微调结果包含一些伪影和噪声,导致观察令人难快的观察。虽然我们微调的 Fasrgan 和 FS-SRGAN 减少了伪影,并产生了更令人愉悦的结果,展示了我们所提出的模型的稳健性。

Conclusion

We propose two GAN-based models, FASRGAN and Fs-SRGAN, for SISR to overcome the limitations of existing methods. FASRAGN introduces a fine-grained attention mechanism into the GAN framework, where the discriminator has two outputs: a score for measuring the quality of the whole input and a fine-grained attention estimation for the input. The fine-grained attention delivers a fine-grained supervisor to the generator to ensure generation of pixel-wise photo-realistic images. The Fs-SRGAN shares the low-level feature extractor of the generator and the discriminator, reducing the number of parameters and improving the reconstruction performance. These two mechanisms are general and could be applied to other GAN-based SR models. Comparisons with other state-of-the-art methods on benchmark datasets demonstrate the effectiveness of our proposed methods.

结论

我们提出了两个基于 GAN 的模型,FasRgan 和 FS-SRGAN,用于克服现有方法的局限性。 Fasragn 将一个细粒度的注意机制引入 GaN 框架,其中鉴别器有两个输出:测量整个输入质量的分数和输入的细粒度注意估计。细粒度的注意力为发电机提供了一个细粒度的监督员,以确保产生像素 - 明智的照片逼真图像。 FS-SRGAN 共享发电机的低级特征提取器和鉴别器,减少参数的数量并提高重建性能。这两个机制是通用的,可以应用于其他基于 GaN 的 SR 模型。与基准数据集的其他最先进方法的比较展示了我们所提出的方法的有效性。