Essential Suite — Artwork Producer Assistant
译者
Gingo 白果
Netflix Technology BlogFollow
Feb 7, 2020 · 8 min read
By: Hamid Shahid & Syed Haq
Introduction 介绍
Netflix continues to invest in content for a global audience with a diverse range of unique tastes and interests. Correspondingly, the member experience must also evolve to connect this global audience to the content that most appeals to each of them. Images that represent titles on Netflix (what we at Netflix call “artwork”) have proven to be one of the most effective ways to help our members discover the content they love to watch. We thus need to have a rich and diverse set of artwork that is tailored for different parts of the Netflix experience (what we call product canvases). We also need to source multiple images for each title representing different themes so we can present an image that is relevant to each member’s taste.
Netflix继续投资于具有各种独特口味和兴趣的全球受众的内容。相应地，会员体验也必须不断发展，以将全球用户与最吸引他们的内容联系起来。代表Netflix标题的封面图（在Netflix我们称之为“艺术品”）已被证明是帮助我们的会员发现他们喜欢观看的内容的最有效方法之一。因此，我们需要拥有一组丰富多样的封面图，这些封面图是为Netflix不同部分（我们称为产品画布）的体验量身定制的。我们还需要为代表不同主题的每个标题获取多个封面图，以便我们呈现与每个成员的品味相关的图像。
Manual curation and review of these high quality images from scratch for a growing catalog of titles can be particularly challenging for our Product Creative Strategy Producers (referred to as producers in the rest of the article). Below, we discuss how we’ve built upon our previous work of harvesting static images directly from video source files and our computer vision algorithms to produce a set of artwork candidates that covers the major product canvases for the entire content catalog. The artwork generated by this pipeline is used to augment the artwork typically sourced from design agencies. We call this suite of assisted artwork “The Essential Suite”.
对于我们不断增长的标题目录，手动管理和查看这些高质量图像对于我们的产品创意策略生产者（在本文的其余部分中称为生产者）来说尤其具有挑战性。下面，我们讨论如何基于先前的工作直接从视频源文件中收获静态图像以及我们的计算机视觉算法，以生成一组涵盖整个内容目录主要“产品画布“的封面图候选。该方法将来自设计机构的图稿进行增强处理，生成封面图。我们称这套封面图辅助工具为“Essential Suite”。
Supplement, not replace 补充，而不是替代
Producers from our Creative Production team are the ultimate decision makers when it comes to the selection of artwork that gets published for each title. Our usage of computer vision to generate artwork candidates from video sources thus is focussed on alleviating the workload for our Creative Production team. The team would rather spend its time on creative and strategic tasks rather than sifting through thousands of frames of a show looking for the most compelling ones. With the “Essential Suite”, we are providing an additional tool in the producers toolkit. Through testing we have learned that with proper checks and human curation in place, assisted artwork candidates can perform on par with agency designed artwork.
我们创意制作团队的创意生产者是产品呈现最终的决策者，他们为每个影片标题选择合适的封面图候选。因此，我们使用计算机视觉从视频源生成封面图的重点是减轻我们的创意制作团队的工作量。团队宁愿将时间花在创造性和战略性任务上，而不是筛选数千帧图片以寻找最引人注目的那些图。通过“Essential Suite”，我们在生产者工具包中提供了额外的工具。通过测试，我们了解到，通过适当的检查和适当的人工策划，机器选出来的封面图可以和专业设计的封面图相媲美。
Design Agencies 设计机构
Netflix uses best-in-class design agencies to provide artwork that can be used to promote titles on and off the Netflix service. Netflix producers work closely with design agencies to request, review and approve artwork. All artwork is delivered through a web application made available to the design agencies.
Netflix使用一流的设计机构来提供可用于在Netflix服务内外推广的封面图。Netflix的创意生产者与设计机构紧密合作，对封面图进行请求，审查和批准。所有封面图均由设计机构通过Web应用程序提供。
The computer generated artwork can be considered as artwork provided by an “Internal agency”. The idea is to generate artwork candidates using video source files and “bubble it up” to the producers on the same artwork portal where they review all other artwork, ideally without knowing if it is an agency produced or internally curated artwork, thereby selecting what goes on product purely based on creative quality of the image.
可以将计算机生成的图稿视为“内部机构”提供的图稿。想法是使用视频源文件生成候选作品，然后将其“冒泡”，提供给创意生产者。创意生产者查看所有其他封面图，理想情况下，不知道该作品是由代理机构制作还是内部策划的封面图，从而选择处理方式完全基于图像的创意质量。
Assisted Artwork Generation Workflow 封面图辅助生成工作流程
The artwork generation process involves several steps, starting with the arrival of the video source files and culminating in generated artwork being made available to producers. We use an open source workflow engine Netflix Conductor to run the orchestration. The whole process can be divided into two parts.
封面图生成过程涉及几个步骤，从视频源文件的到达开始，最后达到将生成的封面图提供给创意生产者的目的。我们使用开源工作流引擎Netflix Conductor来运行业务流程。整个过程可以分为两部分
- Generation
- Review
1. Generation 生成
This article on AVA provides a good explanation on our technology to extract interesting images from video source files. The artwork generation workflow takes it a step further. For a given product canvas, it selects a handful of images from the hundreds of video stills most suitable for that particular product canvas. The workflow then crops and color-corrects the selected image, picks out the best spot to place the movie’s title based on negative space, selects and resizes the movie title and places it onto the image.
这篇关于AVA的文章很好地解释了我们从视频源文件中提取有趣图像的技术。封面图生成工作流程使它更进一步。对于给定的产品画布，它从数百个最适合该特定产品画布的视频静止图像中选择少量图像。然后，工作流程会裁剪并校正所选图像的颜色，选择最佳位置以根据负空间放置电影标题，选择并调整电影标题的大小并将其放置在图像上。
Here is an illustration of what it means if we had to do it manually
这说明了如果我们必须手动进行操作的流程
a. Image selection
b. Identify areas of interest
c. Cropped, color-corrected & title placed in the negative space
Image Selection / Analyze Image 图像选择/分析图像
Selection of the right still image is essential to generating good quality artwork. A lot of work has already been done in AVA to extract out a few hundreds of frames from hundreds of thousands of frames present in a typical video source. Broadly speaking, we use two methods to extract movie stills out of video source.
选择正确的静止图像对于生成高质量的艺术品至关重要。在AVA中已经完成了许多工作，以从典型视频源中存在的数十万个帧中提取几百个帧。广义上讲，我们使用两种方法从视频源中提取电影静止图像。
- AVA — Ava is primarily a character based algorithm. It picks up frames with a clear facial shot taking into account actors, facial expression and shot detection.
- Cinematics — Cinematics picks up aesthetically pleasing cinematic shots.
- AVA — Ava主要是基于演员的算法。它将演员，面部表情和镜头检测等因素考虑在内，以清晰的面部镜头拾取帧。
- Cinematics—Cinematics选择令人愉悦的镜头。
The combination of these two approaches produce a few hundred movie stills from a typical video source. For a season, this would be a few hundred shots for each episode. Our work here is to pick up the stills that best work for the desired canvas.
=Both of the above algorithms use a few heatmaps which define what kind of images have proven to be working best in different canvases.= The heatmaps are designed by internal artists who are experienced in designing promotional artwork/posters.
这两种方法的结合可以从一个典型的视频源中产生数百个电影静止图像。每一季，每个情节可以得到几百张截图。我们在这里的工作是挑选最适合所需画布的静止图像。
=以上两种算法都使用了一些热图，这些热图定义了已证明在不同画布上效果最佳的图像类型。= 热图由在设计促销性艺术品/海报方面经验丰富的内部艺术家设计
Heatmap for a BillboardWe make use of meta-information such as the size of desired canvas, the “unsafe regions” and the “regions of interest” to identify what image would serve best. “Unsafe regions” are areas in the image where badges such as Netflix logo, new episodes, etc are placed. “Regions of interest” are areas that are always displayed in multi-purpose canvases. These details are stored as metadata for each canvas type and passed to the algorithm by the workflow. Some of our canvases are cropped dynamically for different user interfaces. For such images, the “Regions of interest” will be the area that is always displayed in each crop.
我们利用诸如所需画布的大小，“不安全区域”和“感兴趣区域”之类的元信息来确定哪种图像最适合。“不安全区域”是图像中放置徽标（例如Netflix标志，新剧集等）的区域。“感兴趣的区域”是始终显示在多用途画布中的区域。这些详细信息存储为每种画布类型的元数据，并由工作流传递给算法。我们的某些画布针对不同的用户界面进行了动态裁剪。对于此类图像，“感兴趣区域”将是每种作物中始终显示的区域。
Unsafe regionsThis data-driven approach allows for fast turnaround for additional canvases. While selecting images, the algorithms also returns back suggested coordinates within each image for cropping and title placement. Finally, it associates a “score” with the selected image. This score is the “confidence” that the algorithm has on the selection of candidate image on how well it could perform on service, based on previously collected stats.
这种数据驱动的方法可以快速完成其他画布的周转。在选择图像时，算法还会返回每个图像内的建议坐标，以进行裁切和标题放置。最后，它将“分数”与所选图像相关联。该分数是算法基于先前收集的统计信息，在选择候选图像时对其在服务上的表现有多大的“信心”。
Image Creation 影像创作
The artwork generation workflow collates image selection results from each video source and picks up the top “n” images based on confidence score.
图稿生成工作流程会整理来自每个视频源的图像选择结果，并根据置信度得分挑选出前“ n”幅图像。
The selected image is then cropped and color-corrected based on coordinates passed by the algorithm. Some canvases also need the movie title to be placed on the image. The process makes use of the heatmap provided by our designers to perform cropping and title placement. As an example, the “Billboard” canvas shown on a movie’s landing page is right aligned, with the title and synopsis shown on the left.
然后根据算法传递的坐标对所选图像进行裁剪和颜色校正。某些画布还需要将电影标题放置在图像上。该过程利用了我们设计师提供的热图来进行裁剪和标题放置。例如，电影着陆页上显示的“广告牌”画布是右对齐的，标题和摘要显示在左侧。
Billboard Canvas
The workers to crop and color correct images are made available as separate titus jobs. The workflow invokes the jobs, storing each output in the artwork asset management system and passes it on for review.
裁剪和校正彩色图像的工人可作为单独的工作使用。工作流调用作业，将每个输出存储在图稿资产管理系统中，并将其传递以供检查。
2. Review 审核
For each artwork candidate generated by the workflow, we want to get as much feedback as possible from the Creative Production team because they have the most context about the title. However, getting producers to provide feedback on hundreds of generated images is not scalable. For this reason, we have split the review process in two rounds.
对于工作流生成的每个候选作品，我们希望从创意制作团队获得尽可能多的反馈，因为他们对标题的了解最多。但是，要让创意生产者提供有关数百个生成图像的反馈是不可扩展的。因此，我们将审核过程分为两轮。
Technical Quality Control (QC) 技术质量控制（QC）
This round of review enables filtering out images that look obviously wrong to a human eye. Images with features such as human actors with an open mouth, inappropriate facial expressions or an incorrect body position, etc are filtered out in this round.
通过这一轮审核，可以滤除人眼看上去明显错误的图像。在这一轮中，将滤除具有人类演员张口，不适当的面部表情或不正确的身体姿势等特征的图像。
For the purpose of reviewing these images, we use a video/image annotation application that provides a simple interface to add tags for a given list of videos or images. For our purposes, for each image, we ask the very basic question “Should this image be used for artwork?”
为了查看这些图像，我们使用视频/图像注释应用程序，该应用程序提供了一个简单的界面来为给定的视频或图像列表添加标签。为了我们的目的，对于每个图像，我们都会问一个非常基本的问题：“该图像应用于艺术品吗？”
The team reviewing these assets treat each image individually and only look for technical aspects of the image, regardless of the theme or genre of the title, or the quantity of images presented for a given title.
审查这些的团队将分别处理每个图像，并且仅从技术层面考虑图像，而不考虑标题的主题或流派，或为给定标题显示的图像数量。
When an image is rejected, a few follow up questions are asked to ascertain why the image is not suitable to be used as artwork.
当图像被拒绝时，会提出一些后续问题，以确定为什么该图像不适合用作艺术品。
All this review data is fed back to the image selection, cropping and color corrections algorithms to train and improve them.
所有这些评论数据都反馈给图像选择，裁剪和颜色校正算法，以对其进行训练和改进。
Editorial QC 编辑质量控制
Unlike technical QC, which is title agnostic, editorial QC is done by producers who are deeply familiar with the themes, storylines and characters in the title, to select artwork that will represent the title best on the Netflix service.
与技术质量控制（与标题无关）不同，编辑质量控制由对标题中的主题，故事情节和角色非常熟悉的制片人完成，以选择最能代表Netflix服务标题的艺术品。
The application used to review generated artwork is the same application that producers use to place and review artwork requests fulfilled by design agencies. A screenshot of how generated artwork is presented to producers is shown below.
用于审阅生成的图稿的应用程序与生产者用来放置和审阅设计机构完成的图稿请求的应用程序相同。下面显示了如何将生成的艺术品呈现给生产者的屏幕截图
Similar to technical QC, the option here for each artwork is whether to approve or reject the artwork. The producers are encouraged to provide reasons why they are rejecting an artwork.
与技术质量控制类似，每个艺术品的选项是批准还是拒绝艺术品。鼓励制作人提供拒绝艺术品的原因。
Approved artwork makes its way to the artwork’s asset management system, where it resides alongside other agency-fulfilled artwork. From here, producers have the ability to publish it to the Netflix service.
批准的图稿进入图稿的资产管理系统，并与其他代理完成的图稿一起存在。制作人可以从此处将其发布到Netflix服务。
Conclusion 结论
We have learned a lot from our work on generating artwork. Artwork that looks good might not be the best depiction of the title’s story, a very clear character image might be a content spoiler. All of these decisions are best made by humans and we intend to keep it that way.
However, assisted artwork generation has a place in supporting our creative team by providing them with another avenue to pick up their assets from, and with careful supervision will help in their challenge of sourcing artwork at scale.
我们从生成封面图的工作中学到了很多东西。看起来不错的封面图可能不是标题故事的最佳描绘，非常清晰的人物形象可能会破坏内容。所有这些决定都是人类最好的决定，我们打算保持这种状态。
但是，封面图辅助生成可以为我们的创意团队提供支持，为他们提供另一种获取资源的途径，并且经过认真的监督将有助于他们应对大规模生产封面图的挑战。