正如我们在之前的博客文章中所描述的那样，在Netflix，我们广泛使用个性化设置，并将每种情况都视为向我们的5700万以上会员中的每个人提供正确内容的机会。成员与我们的推荐进行互动的主要方式是通过主页，当他们在任何受支持的设备上登录Netflix时，就会看到该主页。主页的主要功能是帮助每个成员轻松找到自己喜欢的观看内容。我们面临的一个问题是，我们的目录包含的视频远远超过单个页面上显示的视频，并且每个成员都有自己独特的兴趣爱好。因此，一般的算法挑战就变成了如何最佳地调整每个成员的主页，使其相关，涵盖他们的兴趣和和观看意图，并且仍然允许他们探索其它内容。

As we’ve described in our previous blog posts, at Netflix we use personalization extensively and treat every situation as an opportunity to present the right content to each of our over 57 million members. The main way a member interacts with our recommendations is via the homepage, which they see when they log into Netflix on any supported device. The primary function of the homepage is to help each member easily find something to watch that they will enjoy. A problem we face is that our catalog contains many more videos than can be displayed on a single page and each member comes with their own unique set of interests. Thus, a general algorithmic challenge becomes how to best tailor each member’s homepage to make it relevant, cover their interests and intents, and still allow for exploration of our catalog.

这种类型的问题并不是Netflix独有的，新闻网站，搜索引擎和在线商店等其他组织也面临着这种问题。任何需要从大量可用的可能性中选择项目，然后以连贯且易于导航的方式展示它们的站点，都将面临相同的挑战。当然，优化Netflix主页的问题有其独特的方面，，比如界面限制，以及其电影和电视的消费方式与其他媒体相比都存在差异。

This type of problem is not unique to Netflix, it is faced by others such as news sites, search engines, and online stores. Any site that needs to choose items from a large number of available possibilities and then present them in a coherent and easy-to-navigate manner will face the same general challenges. Of course, the problem of optimizing Netflix homepages has its own unique aspects, ranging from interface constraints to differences with how movies and TV are consumed compared to other media.

An example of a personalized Netflix homepage on our website.Currently, the Netflix homepage on most devices is structured with videos (movies and TV shows) organized into thematically coherent rows presented in a two-dimensional layout. Members can scroll either horizontally on a row to see more videos in that row or vertically to see other rows. Thus, a key part of our personalization approach is how we choose rows to display on the homepage. This involves figuring out how to select the rows most relevant to each member, how to populate those rows with videos, and how to arrange them on the limited page area such that selecting a video to watch is intuitive. In the rest of this post, we will highlight what we think are the most relevant and interesting aspects of this problem and how we can go about solving some of them.

当前，大多数设备上的Netflix主页都是由视频（电影和电视节目）组成的，这些视频按二维布局呈现为主题一致的行。成员可以水平滚动查看该行中的更多视频，也可以垂直滚动查看其他行。因此，我们的个性化方法的关键部分是我们如何选择要显示在首页上的行。这涉及弄清楚如何选择与每个成员最相关的行，如何用视频填充这些行，以及如何将它们排列在有限的页面区域上，从而直观地选择要观看的视频。在本文的其余部分，我们将重点介绍我们认为与该问题最相关，最有趣的方面，以及我们如何才能解决其中的一些问题。

Evolution of our personalization approach.

Why Rows Anyway?

We organize our homepage into a series of rows to make it easy for members to navigate through a large portion of our catalog. By presenting coherent groups of videos in a row, providing a meaningful name for each row, and presenting rows in a useful order, members can quickly decide whether a whole set of videos in a row is likely to contain something that they are interested in watching. This allows members to either dive deeper and look for more videos in the theme or to skip them and look at another row. This would not be the case if, for example, the page contained a large, unorganized collection of relevant videos.

我们将首页组织成一系列的行，以使成员可以轻松浏览目录的很大一部分。通过连续显示一组连续的视频，为每行提供一个有意义的名称并按有用的顺序显示行，成员可以快速确定一行完整的视频集是否可能包含他们感兴趣的观看内容。这使成员可以更深入地研究主题中的更多视频，或者跳过它们并查看另一行。例如，如果该页面包含大量无组织的相关视频，则不是这种情况。

A possible row of titles that might be watched by one of our Netflix original characters.One natural way to group videos is by genre or sub-genre or other video metadata dimensions like release date. Of course, the relationship between videos in a row does not have to be due to metadata alone, but can also be formed from behavioral information (for example from collaborative filtering algorithms), videos we think a member is likely to watch, or even groups of videos watched by a friend. Thus, each row can offer a unique and personalized slice of the catalog for a member to navigate. Part of the challenge and fun of creating a personalized homepage is figuring out new ways to create useful groupings of videos, which we are constantly experimenting with (e.g., rows of titles that might be watched by one of our Netflix original characters shown above).

对视频进行分组的一种自然方法是按流派或子流派或其他视频元数据维度（例如发布日期）进行分类。当然，连续视频之间的关系不必仅取决于元数据，还可以由行为信息（例如，来自协作过滤算法），我们认为成员可能观看的视频甚至是分组来形成朋友观看的视频数量。因此，每一行都可以提供目录的唯一且个性化的切片，以供成员导航。创建个性化主页的挑战和乐趣的一部分是想出新方法来创建有用的视频分组，我们正在不断尝试（例如，上面显示的Netflix原始角色之一可能会观看的标题行）。

Process for creating and choosing rows.

Once we have a set of possible video groups to consider for a page, we can begin to assemble the homepage from them. To do this, we start by finding candidate groupings that are likely relevant for a member based on the information we know about them. This also involves coming up with the evidence (or explanations) to support the presentation of a row, for example the movies that the member has previously watched in a genre. Next, we filter each group to handle concerns like maturity rating or to remove some previously watched videos. After filtering, we rank the videos in each group according to a row-appropriate ranking algorithm, which produces an ordering of videos such that the most relevant videos for the member in a group are at the front of the row. From this set of row candidates we can then apply a row selection algorithm to assemble the full page. As the page is assembled, we do additional filtering like deduplication to remove repeat videos and format rows to the appropriate size for the device.

一旦我们考虑了一组可能的视频组用于一个页面，就可以开始从中组合主页。为此，我们首先根据我们了解的候选成员分组来查找可能与该成员相关的候选分组。这还涉及拿出证据（或解释）来支持一行的呈现，例如，该成员以前观看过的流派的电影。接下来，我们过滤每个组以处理诸如成熟度等级之类的问题，或删除一些以前观看的视频。过滤后，我们根据适合行的排名算法对每个组中的视频进行排名，该算法产生视频的排序，从而使与该组成员最相关的视频位于该行的最前面。然后，从这组候选行中，我们可以应用行选择算法来组装整个页面。在页面组装完成后，我们会进行重复数据删除等其他过滤操作，以删除重复的视频并将行格式设置为适合设备的大小。

Page-level algorithmic challenge 页面级算法挑战

To algorithmically create a good personalized homepage means assembling one page per member profile and device from thousands of videos that may be relevant for a member and from easily tens of thousands of potential rows, each with a variable number of videos. On top of that, we need to balance several factors that often compete for precious screen real estate. Our approach to personalization and recommendation largely focuses on helping our members find something new to watch, which we call discovery. However, we also want to make it easy for a member to watch the next episode of a show or re-watch something that they watched in the past, which normally falls outside the realm of recommendation. =We want our recommendations to be accurate in that they are relevant to the tastes of our members, but they also need to be diverse so that we can address the spectrum of a member’s interests versus only focusing on one. We want to be able to highlight the depth in the catalog we have in those interests and also the breadth we have across other areas to help our members explore and even find new interests. We want our recommendations to be fresh and responsive to the actions a member takes, such as watching a show, adding to their list, or rating; but we also want some stability so that people are familiar with their homepage and can easily find videos they’ve been recommended in the recent past.= Finally, we need to be able to place task-oriented rows, such as “My List,” in amongst the more discovery-oriented rows.

要通过算法创建良好的个性化首页，是指从成千上万的与会员相关的视频和成千上万的潜在行（每个视频的数目不固定）为每个会员个人资料和设备组合一页。最重要的是，我们需要权衡一些经常争夺珍贵屏幕的因素。我们的个性化和推荐方法主要集中在帮助我们的会员找到值得关注的新事物，我们称之为发现discovery。但是，我们还希望使成员能够轻松观看节目的下一集或重新观看他们过去观看的内容，而这通常超出了推荐范围。
我们希望我们的建议是准确的，因为它们与我们的会员的口味有关，但它们也必须是多种多样的，以便我们可以解决会员利益的范围而不是仅仅关注一个人的利益。我们希望能够突显我们对这些兴趣感兴趣的目录的深度，以及我们在其他领域所拥有的广度，以帮助我们的成员探索甚至发现新的兴趣。我们希望我们的建议是新鲜的，并能响应会员采取的行动，例如观看表演，添加到他们的列表或评分；但我们也希望保持稳定性，以便人们熟悉其主页并可以轻松找到他们最近推荐的视频。
最后，我们需要能够将面向任务的行（例如“我的列表”）放在更多面向发现的行中。

Each device has different hardware capabilities that can limit the number of videos or rows displayed at any one time and how big the whole page can be. As such, the page generation process must be aware of the constraints of the device for which it is creating the page, including the number of rows, the minimum and maximum length of a row, the size of the visible portion of the page, and whether or not certain rows are required or are not applicable for a certain device.

While there are many challenges to page generation, tackling recommendation problems at this level also opens up new solutions. As mentioned before, selecting a diverse set of items is important in a recommendation system. However, it can be challenging to navigate a diverse ranking since the relevant items may be blended with other items that do not match someone’s current intent. However, by presenting a two-dimensional navigation layout, a member can scroll vertically to easily skip over entire groups of content that may not match their current intent and then find a more relevant set, which they can then scroll horizontally to see more recommendations in that set. This allows for coherent, meaningful individual rows to be selected while maintaining the diversity of the videos shown on the whole page, and thus lets the member have both relevance and diversity.

每个设备具有不同的硬件功能，这些功能可以限制一次显示的视频或行的数量以及整个页面的大小。因此，页面生成过程必须知道为其创建页面的设备的约束，包括行数，行的最小和最大长度，页面可见部分的大小以及是否需要某些行或不适用于某些设备。

尽管页面生成面临许多挑战，但在此级别解决推荐问题也带来了新的解决方案。如前所述，在推荐系统中选择一组不同的项目很重要。但是，由于相关项目可能会与不符合某人当前意图的其他项目混合在一起，因此导航到不同的排名可能会带来挑战。但是，通过呈现二维导航布局，成员可以垂直滚动以轻松跳过可能不符合其当前意图的整个内容组，然后找到一个更相关的集合，然后他们可以水平滚动以查看更多建议。该集。这样一来，可以选择连贯且有意义的单个行，同时保持整个页面上显示的视频的多样性，

Building a page algorithmically 通过算法构建页面

There are several approaches for how we can build our homepage algorithmically. The most basic is a rule-based approach, which we used for a long time. Here a set of rules define a template that dictates for all members what types of rows can go in certain positions on the page. For example, the rules could specify that the first row would be Continue Watching (if any), then Top Picks (if any), then Popular on Netflix, then 5 personalized genre rows, and so on. The only personalization in this approach was from selecting candidate rows in a personalized way, such as including“Because you watched” rows for videos someone has watched in the past and genre rows based on known genre preferences. To choose specific rows within each type, simple heuristics and sampling were used. We evolved this template using A/B testing to understand where to place rows for all members.

有几种方法可用于以算法方式构建主页。最基本的是基于规则的方法，我们使用了很长时间。在这里，一组规则定义了一个模板，该模板为所有成员指示在页面上的某些位置可以进入哪种类型的行。例如，规则可以指定第一行是“继续观看”（如果有），然后是“热门精选”（如果有），然后是“在Netflix上受欢迎”，然后是5个个性化类型行，依此类推。这种方法唯一的个性化设置是通过以个性化方式选择候选行，例如包括某人过去观看过的视频的“因为您观看了”行，以及根据已知的流派首选项进行的流派。为了选择每种类型中的特定行，使用了简单的试探法和采样。

This approach served us well, but it ignored many aspects we consider important for the quality of the page, such as the quality of the videos in the row, the amount of diversity on the page, the affinity of members for specific kinds of rows, and the quality of the evidence we can surface for each video. It also made it hard to add new types of rows, because for a new row to succeed it would need to not only contain a relevant set of videos in a good order but also be placed appropriately in the template. Because of this, the rules for the template grew over time and became too complex to handle the variety of rows and how they should all be placed, which represented a local optimum for the member experience.

这种方法对我们很有帮助，但它忽略了许多我们认为对页面质量至关重要的方面，例如行中视频的质量，页面上的多样性，成员对特定类型行的亲和力，以及我们可以为每个视频展示的证据的质量。这也使添加新类型的行变得困难，因为要使新行成功，它不仅需要以良好的顺序包含一组相关的视频，而且还需要适当地放置在模板中。因此，模板的规则随着时间的增长而变得越来越复杂，以至于无法处理各种各样的行以及应如何放置所有行，这代表了成员体验的局部最优。

To address these issues, we can instead think of personalizing the ordering of rows on the homepage. The simplest approach for doing this is to treat rows as items in a ranking problem, which we call a row-ranking approach. For this approach, we could leverage a lot of existing recommendation or learning-to-rank approaches by developing a scoring function for rows, applying it to all the candidate rows independently, sorting by that function, and then picking the top ones to fill the page. Even though the space of rows may be relatively big, this type of approach could be relatively fast and may result in reasonable accuracy. However, doing this would lack any notion of diversity, so someone could easily get a page full of slight variations of their interests, such as many rows each with different variants of comedies: late-night, family, romantic, action, etc.

为了解决这些问题，我们可以考虑对主页上的行进行个性化设置。执行此操作的最简单方法是将行视为排序问题中的项目，我们称之为行排序方法。对于这种方法，我们可以通过为行开发一个评分函数，将其独立地应用于所有候选行，按该函数进行排序，然后选择最上面的行来填充行，从而利用很多现有的推荐或学习排名方法页。即使行的空间可能相对较大，这种方法也可能相对较快并可能导致合理的准确性。但是，这样做会缺乏多样性的概念，因此某人可以轻松获得一个页面，其中充满了他们的兴趣的细微变化，例如许多行，每行都有不同的喜剧变体：深夜，家庭，浪漫，

A simple way to add in diversity is to switch from a row-ranking approach to a stage-wise approach using a scoring function that considers both a row as well as its relationship to both the previous rows and the previous videos already chosen for the page. In this case, one can take a simple greedy approach and pick the row that maximizes this function as the next row to use and then re-score all the rows for the next position taking that selection into account. Depending on the diversity function, this greedy selection may not lead to an optimal page. Using a stage-wise approach with k-row lookahead could result in a more optimal page than greedy selection, but it comes with increased computational cost. Other approaches to greedily add diversity based on submodular function maximization can also be used.

添加多样性的一种简单方法是使用评分函数从行排名方法切换到逐级方法，该评分函数既考虑行又考虑其与先前行和先前已为页面选择的视频的关系。 在这种情况下，可以采用简单的贪心方法，选择最大化此函数的行作为要使用的下一行，然后将下一个位置的所有行重新评分。 这种贪心的选择可能无法产生最佳页面。 使用具有k行前瞻的分阶段方法可以产生比贪心选择更优化的页面，但是它带来了增加的计算成本。也可以使用其他基于子模块函数最大化贪婪地增加分集的方法。

However, even the stage-wise algorithm is not guaranteed to produce an optimal page because a fixed horizon may limit the ability to fill in better rows further down the page. Thus, if we can instead take a page-wise approach by defining a full-page scoring function, we can try to optimize it by choosing rows and videos appropriately to fill the page. Of course, the space of possible pages is huge, even larger than the space of possible rows. Since a page layout is defined in a discrete space, directly optimizing a function that defines the quality of the whole page is a computationally prohibitive integer programming problem.

然而，即使是阶段式算法也不能保证产生最佳页面，因为固定的时间范围可能会限制在页面下方填充更好的行的能力。 如果我们可以定义整页评分函数，我们可以尝试通过适当选择行和视频来填充页面来优化它。 当然，页面组合的搜索空间很大，因此直接优化定义整个页面质量的函数在算力上是很难实现的。

When solving a page optimization problem with any of these approaches, there are also various constraints that need to be taken into account that were mentioned before, like deduping, filtering, and device-specific constraints. Each of these constraints add to the complexity of the optimization problem.

当使用这些方法中的任何一种来解决页面优化问题时，还需要考虑前面提到的各种约束，例如重复数据删除，过滤和特定于设备的约束。这些约束中的每一个都增加了优化问题的复杂性。

Notional importance of navigation modeling.Members are more likely to scan vertically than horizontally, which means videos presented in the upper left are much more likely to be seen than those in the lower right.

导航建模的名义重要性。成员更有可能在垂直方向而非水平方向进行扫描，这意味着左上角的视频比右下角的视频更有可能被观看。

When forming the homepage it is also important to consider how members navigate the page, i.e., to consider which positions on the page they are likely to pay attention to and interact with in a session. Placing the most relevant videos in the positions that are most likely to be seen, which tends to be the upper-left corner, should reduce the time for a member to find something relevant to watch. However, modeling navigation on a two-dimensional page is difficult, especially taking into account that different people may navigate differently, people’s navigation patterns may change over time, there are differences in navigation across different device types based on the interaction design, and that navigation is clearly dependent on the relevance of the content shown. With an accurate navigation model, we can inform better placement of videos and rows and where on the page to focus on relevance as opposed to diversity.

在形成主页时，考虑成员如何浏览页面也是重要的，即，考虑他们可能在会话中注意页面上的哪些位置。 将最相关的视频放置在最有可能被看到的位置（通常是左上角），可以减少成员查找与观看相关的内容的时间。 然而，在二维页面上建模导航是困难的，特别是考虑到不同的人可能导航模式不同，人们的导航模式可能随时间而变化，基于交互设计在不同设备类型之间存在导航差异，并且导航 显然取决于所显示内容的相关性。 通过精确的导航模型，我们可以更好地了解视频和行的位置以及页面上的重点。

Machine Learning for page generation 机器学习进行页面生成

At the core of building a personalized page is a scoring function that can evaluate the quality of a row or a page. While we could use heuristics or intuition for building such a scoring function and tune it using A/B testing, we prefer to learn a good function from the data so that we can easily incorporate new data sources and balance the various different aspects of a homepage. To do this, we can use a machine learning approach to create the scoring function by training it using historical information of which homepages we have created for our members, what they actually see, how they interact, and what they play.

构建个性化页面的核心是评分功能，可以评估行或页面的质量。 虽然我们可以使用启发式或直觉来构建这样的评分函数并使用A / B测试进行调整，但我们更愿意从数据中学习一个好的函数，以便我们可以轻松地合并新的数据源并平衡主页的各个不同方面。 为此，我们可以使用我们为user创建不同的主页，他们实际看到的内容，他们的interactions以及他们的playback进行训练，从而用机器学习算法创建评分功能。

There is a large set of features that we could potentially use to represent a row for our learning algorithms. Since rows contain a set of videos, we can use any features of those videos in the row representation, either by aggregating across the row or indexing them by position. These features can be simple metadata or more useful model-based features that represent how good of a recommendation we believe a specific video is for a member. Of course, we have many different recommendation approaches, so we can include them as different features to learn an ensemble of them at the page level. We can also look at the quality of the evidence associated with the row, such as how much support there is for a member being interested in a specific genre. We can also look at past interactions with the row to see if that row or similar such rows have been consumed in the past by the member. We can also add simple descriptive features like how many videos are in a row, in what position a row is being placed on a page, or how often we’ve shown the row in the past. Diversity can also be additionally incorporated into the scoring model when considering the features of a row compared to the rest of the page by looking at how similar the row is to the rest of the rows or the videos in the row to the videos on the rest of the page.

我们可以使用大量features来represent rows。由于行包含一组视频，因此我们可以在行表示中聚合使用这些视频的features。这些feature可以是简单的元数据或更有用的基于model的features，它们表示给用户推荐特定视频的理想程度。当然，我们有许多不同的推荐方法，可以使用ensemble方法把它们聚合起来。我们还可以查看与行相关的一些feature，例如对于对特定genre感兴趣的user有多少，以及当前user过去是否已消耗该行或类似行。我们还可以添加简单的描述性feature，例如这一行连续多少个视频，该行放置在页面上的位置，以及我们在过去显示该行的频率。我们还可以通过查看行与其余行的相似程度或行中的视频与其余部分的视频相似程度，将多样性纳入评分。

While the space of potentially useful features is quite large, there are several challenges with training machine learning models for scoring rows. One challenge is dealing with presentation bias, where a member can only play from a row on the homepage that we’ve chosen to display, which can have a huge impact on the training data. To further complicate things, the position of a row on the page can greatly affect whether a member actually sees the row and then chooses to play from it. To handle these presentation and position biases, we need to be extremely careful about how we select training data for our algorithms. There is also a challenge around how attribution is allowed in the model; a video may have been played in a certain row in the past, but does that mean the member would have chosen that same video if it was placed in a different row but in the first position? Perhaps the title of a row being “Critically Acclaimed Documentaries” was responsible for play where it may not have been selected without that additional evidence, for example, in a “New Releases” row, even if it was in a better position. Learning over features to represent diversity can also be challenging because while the space of potential rows at different positions on the page is large, when the rest of the page (or the already chosen rows) is taken into account for diversity, the space of possible pages is even larger.

用于行评分的机器学习模型存在若干挑战。一个挑战是处理presentation bias，由于用户只能在我们显示的主页上选择视频播放，没被显示的训练数据就可能产生偏差。更复杂的是，页面上某行的位置会极大地影响user是否能实际看到该并选择从中进行播放。为了处理这些presentation和position bias，我们需要非常小心地选择训练数据。关于如何在模型中选择视频归属的行也存在挑战;视频可能在过去的某一行中播放过，但这是否意味着如果该视频被放置在不同的行的第一个位置，该user会选择相同的视频？多样性的引入也具有挑战性，因为页面上不同位置的潜在行的特征空间已经很大，但当页面的其余部分考虑到多样性时，可能的特征空间变得更大也更难以搜索。

Page-level metrics 页面级指标

To deal with these challenges, as with any algorithmic approach, choosing a good metric is important. Of fundamental importance in page generation is how to evaluate the quality of the pages produced by a specific algorithm during offline experimentation. While we ultimately will test any potential algorithmic improvement online in an A/B test, we would like to be able to focus our precious A/B testing resources on algorithms that we have evidence are likely to improve the quality of the pages. We also need to be able to tune the parameters of those algorithms before A/B testing. To do this, we can use historical data to generate hypothetical pages from new algorithmic approaches, provided we can choose a good metric for page quality.

为了应对这些挑战，就像使用任何算法方法一样，选择一个良好的指标很重要。在页面生成中最重要的是如何评估离线实验期间由特定算法生成的页面的质量。虽然我们最终将在A / B测试中在线测试任何潜在的算法改进，但我们希望能够将宝贵的A / B测试资源集中在我们有可能改善页面质量的算法上。我们还需要能够在A / B测试之前调整那些算法的参数。为此，我们可以使用历史数据从新的算法方法生成假设的页面，前提是我们可以为页面质量选择良好的指标。

Example of two-dimensional recall metrics. For each page variant, the fractions on the side represent the recall at 1-by-3, 2-by-3, and 3-by-3 metrics, respectively.

二维召回指标示例。对于每个页面变体，侧面的分数分别表示1×3、2×3和3×3指标的召回率。

To come up with page-level quality metrics, we took inspiration from ranking metrics that are common in information retrieval (many of which exist in the literature) for a one-dimensional list and created ones that work over a two-dimensional layout. For instance, consider a simple metric like Recall@n, which measures the number of relevant items in the top n divided by the total number of relevant items. We can extend it in two dimensions to be Recall@m-by-n, where now we count the number of relevant items in first m rows and n columns on the page divided by the total number of relevant items. Thus, Recall@3-by-4 may represent quality of videos displayed in the viewport on a device that initially can show 3 rows and 4 videos at a time. One nice property of recall defined this way is that it automatically can handle corner-cases like duplicate videos or short rows. We can also hold one of the values n (or m) fixed and sweep across the other to calculate, for instance, how the recall increases in the viewport as the member would scroll down the page.

为了提出页面级质量指标，我们从一维列表中的指标中获取灵感，并创建了在二维布局上工作的指标。例如，考虑一个简单的指标，如Recall @ n，我们可以将它扩展为两个维度为Recall @ m-by-n，现在我们计算页面前m行和n列中相关项目的数量除以相关项目的总数。因此，Recall @ 3-by-4可以表示设备上视窗中显示的视频质量，该设备最初一次可以显示3行和4个视频。以这种方式定义的召回的一个很好的属性是它可以自动处理像重复视频或短行这样的corner case。我们还可以将其中一个值n（或m）固定并扫过另一个来计算，例如，当user向下滚动页面，视窗中的视频增加时。

Comparison of four page algorithms in recall up to a fixed column position while sweeping the row position. The red line is the previous rule-based approach and the blue is a personalized layout.

四种页面算法的比较：在扫描行位置时，可调用到固定的列位置。红线是以前的基于规则的方法，蓝线是个性化的布局。

Of course, Recall is a basic metric and requires choosing values for m and n, but we can likewise extend metrics that assign a score or likelihood for a member seeing a position, like NDCG or MRR, to the two-dimensional case. We can also adapt navigation models like Expected Reciprocal Rank to incorporate two-dimensional navigation through the page and take into account the cascading aspect of browsing. With such page-level metrics defined, we can use them to evaluate changes in any of the algorithmic approaches used to generate the page, not just the algorithms for ordering the rows, but also the selection, filtering, and ranking algorithms, or any of the input data that they use.

当然，Recall是一个基本指标，需要选择m和n的值，但是我们同样可以扩展指标，为在二维情况下看到某个位置的成员（如NDCG或MRR）分配得分或可能性。我们还可以调整导航模型，例如“预期倒数排名”，以在页面中纳入二维导航，并考虑浏览的级联方面。定义了这样的页面级指标后，我们可以使用它们来评估用于生成页面的任何算法方法的变化，不仅是用于对行进行排序的算法，还可以是选择，过滤和排名算法，或者其中的任何一种他们使用的输入数据。

Other challenges 其他挑战

There is no shortage of challenging questions that come up in engineering the homepage. For example: When is it appropriate to take into account other context variables such as the time of the day or device, in how we populate the homepages? How do we find the appropriate trade-off between finding the optimal page and computational cost? How do we form the home pages during the critical first few sessions of a member, precisely at the time when we have the least information about them? We need to think about and weigh the importance of each of these questions every day in order to continually improve the Netflix homepages.

在设计主页时不乏富挑战性的问题。例如：在如何填充首页时，什么时候考虑其他上下文变量（例如一天中的时间或设备）是合适的？我们如何在找到最佳页面和计算成本之间找到适当的取舍？在成员的关键的前几届会议中，确切地说是在我们掌握的信息最少的时候，我们如何形成主页？我们需要每天思考和权衡每个问题的重要性，以便不断完善Netflix主页。

Conclusion 结论

While Netflix may be most famous in the recommendations community for the Netflix prize, we think of personalized page generation as the next step in the evolution of our personalization approach from rating prediction to video ranking to now page generation. We have taken the initial step of coming up with our first algorithm for personalized page generation that showed significantly better online performance than our existing template, and deployed it last year. However, personalized page generation is a challenging problem that involves balancing a multitude of factors, and we think that this is just the beginning. There is a lot of potential to improve the homepages for all of our members and help them easily find content they will love.

尽管因为 Netflix prize 的原因Netflix在推荐社区中可能是最著名的，但我们认为个性化页面生成是我们的个性化方法从评级预测到视频排名再到现在页面生成。我们已迈出了第一步，提出了第一个用于个性化页面生成的算法，该算法显示出比我们现有模板明显更好的在线性能，并于去年进行了部署。但是，个性化页面的生成是一个具有挑战性的问题，涉及多个因素之间的平衡，我们认为这仅仅是个开始。有很大的潜力可以改善我们所有成员的主页，并帮助他们轻松找到自己喜欢的内容。

We are always looking for talented researchers and engineers to join our team. So if you are interested in helping us solve these types of problems and increasing global happiness, please take a look at some of our open positions on the Netflix jobs page.

Originally published at techblog.netflix.com on April 9, 2015.

*by Chris Alvino and *Justin Basilico