Diagnosing latency issues

诊断延迟问题

文档翻译：源文地址：https://redis.io/docs/reference/optimization/latency/

Finding the causes of slow responses

寻找响应慢的原因。

This document will help you understand what the problem could be if you are experiencing latency problems with Redis.

这个文档将帮你理解Redis延迟问题的原因。

In this context latency is the maximum delay between the time a client issues a command and the time the reply to the command is received by the client. Usually Redis processing time is extremely low, in the sub microsecond range, but there are certain conditions leading to higher latency figures.

这篇文章里的延迟指的是客户端发送命令到客户端接收返回之间的最大延迟。通常Redis处理特别快，在微秒级别，但是有些条件会导致高延迟。

I’ve little time, give me the checklist

紧急情况检查清单

The following documentation is very important in order to run Redis in a low latency fashion. However I understand that we are busy people, so let’s start with a quick checklist. If you fail following these steps, please return here to read the full documentation.

下面的文档对于低延迟运行redis很重要，但是，我们都是大忙人，所以咱先来一个快速检查清单。如果你按以下步骤没成功，再来读整篇文档

1. Make sure you are not running slow commands that are blocking the server. Use the Redis Slow Log feature to check this.

   确保你没有运行阻塞服务的慢命令。使用SlowLog功能可以检查。

2. For EC2 users, make sure you use HVM based modern EC2 instances, like m3.medium. Otherwise fork() is too slow.

   对于EC2（Elastic Compute Cloud 亚马逊弹性计算云）用户，确保你使用基于现代EC2实例的HVM，如 m3.medium。否则fork()操作太慢了。

3. Transparent huge pages must be disabled from your kernel. Use to disable them, and restart your Redis process.

   THP必须被禁用，使用 禁用然后重启Redis进程。

4. If you are using a virtual machine, it is possible that you have an intrinsic latency that has nothing to do with Redis. Check the minimum latency you can expect from your runtime environment using . Note: you need to run this command in the server not in the client.

   如果用虚拟机，内部的延迟Redis没法处理。你可以使用命令从运行环境获取最低延迟信息。注意：你需要在服务器端运行这个命令，而不是客户端。

5. Enable and use the Latency monitor feature of Redis in order to get a human readable description of the latency events and causes in your Redis instance.

   启用Latency monitor功能来获取导致你Redis实例延迟的事件的描述。

In general, use the following table for durability VS latency/performance tradeoffs, ordered from stronger safety to better latency.

基本上，使用以下表格持久化延迟性能权衡策略，按安全性到低延迟进行排序

1. AOF + fsync always: this is very slow, you should use it only if you know what you are doing.
   

   AOF + fsync always: 非常慢

2. AOF + fsync every second: this is a good compromise.
   

   AOF + fsync every second 比较折中

3. AOF + fsync every second + no-appendfsync-on-rewrite option set to yes: this is as the above, but avoids to fsync during rewrites to lower the disk pressure.
   

   AOF + fsync every second + no-appendfsync-on-rewrite option set to yes: 与上面相同，但避免了重写时进行fsync，以降低磁盘压力。

4. AOF + fsync never. Fsyncing is up to the kernel in this setup, even less disk pressure and risk of latency spikes.
   

   AOF + fsync never:更低的磁盘负载和延迟毛刺风险。

5. RDB. Here you have a vast spectrum of tradeoffs depending on the save triggers you configure.
   

   RDB.基于配置的触发器性能差距很大

And now for people with 15 minutes to spend, the details…

接着说明要花15分钟来看详细说明。

Measuring latency

测量延迟

If you are experiencing latency problems, you probably know how to measure it in the context of your application, or maybe your latency problem is very evident even macroscopically. However redis-cli can be used to measure the latency of a Redis server in milliseconds, just try:

如果你遇到延迟问题，你也许知道如何在你的应用上下文中测量延迟，或者你的延迟问题非常明显甚至肉眼可见。可以被用来测量 Redis 服务的毫秒单位下的延迟。使用如下命令：

Using the internal Redis latency monitoring subsystem

使用内置的Redis延迟监控子系统

Since Redis 2.8.13, Redis provides latency monitoring capabilities that are able to sample different execution paths to understand where the server is blocking. This makes debugging of the problems illustrated in this documentation much simpler, so we suggest enabling latency monitoring ASAP. Please refer to the Latency monitor documentation.

While the latency monitoring sampling and reporting capabilities will make it simpler to understand the source of latency in your Redis system, it is still advised that you read this documentation extensively to better understand the topic of Redis and latency spikes.

Latency baseline

延迟基准线

There is a kind of latency that is inherently part of the environment where you run Redis, that is the latency provided by your operating system kernel and, if you are using virtualization, by the hypervisor you are using.

有一种延迟是Redis运行环境内生的延迟，是由你的操作系统内核产生的，并且如果你使用虚拟化，会由虚拟机管理程序产生。

While this latency can’t be removed it is important to study it because it is the baseline, or in other words, you won’t be able to achieve a Redis latency that is better than the latency that every process running in your environment will experience because of the kernel or hypervisor implementation or setup.

虽然无法消除此延迟，但研究它很重要，因为它是基线，换句话说，环境中运行的每个进程由于内核或虚拟机管理程序实现或设置产生的Redis延迟就已经到底了，你再怎么优化也不可能优化的比这个延迟更低。

We call this kind of latency intrinsic latency, and starting from Redis version 2.8.7 is able to measure it. This is an example run under Linux 3.11.0 running on an entry level server.

我们称这种延迟为内源性延迟，在Redis 2.8.7版本开始工具可以测试它。下面以Linux 3.11.0版本的入门级服务器上运行为例。

Note: the argument is the number of seconds the test will be executed. The more time we run the test, the more likely we’ll be able to spot latency spikes. 100 seconds is usually appropriate, however you may want to perform a few runs at different times. Please note that the test is CPU intensive and will likely saturate a single core in your system.

注意：参数100是测试执行将被执行的秒数。我们运行测试的时间越长，我们越能定位延迟峰值。100秒一般就很合适，但你可能希望不同的时间执行一些次测试运行。要注意这个测试会占用大量CPU资源，而且可能会使系统中的单个核饱和。

Note: redis-cli in this special case needs to run in the server where you run or plan to run Redis, not in the client. In this special mode redis-cli does not connect to a Redis server at all: it will just try to measure the largest time the kernel does not provide CPU time to run to the redis-cli process itself.

注意，在这个特殊场景中需要在你运行或计划运行Redis实例的服务器上运行，而不是在其他客户端服务器上运行。在这种特殊模式下，工具压根不会连接Redis服务：它只会尝试测量 内核从不提供CPU时间运行 到 切换到Redis-cli 进程本身的最长时间。

In the above example, the intrinsic latency of the system is just 0.115 milliseconds (or 115 microseconds), which is a good news, however keep in mind that the intrinsic latency may change over time depending on the load of the system.

在上面的例子中，系统内生延迟只有0.115毫秒（或115微秒），很好。然而要记住内生延迟会随着系统负载的变化而改变。

Virtualized environments will not show so good numbers, especially with high load or if there are noisy neighbors. The following is a run on a Linode 4096 instance running Redis and Apache:

虚拟化环境的数值不会这么好，特别是在高负载下，或者是同宿主的其他虚拟机负载高的情况。以下是Linode 4096实例上执行运行 Redis 和 Apache的测试：

Here we have an intrinsic latency of 9.7 milliseconds: this means that we can’t ask better than that to Redis. However other runs at different times in different virtualization environments with higher load or with noisy neighbors can easily show even worse values. We were able to measure up to 40 milliseconds in systems otherwise apparently running normally.

可以看到内生 延迟达到了9.7毫秒，这意味着我们请求Redis不会比这个更快了。然而其他的在不同时间在不同的高负载虚拟化环境或高负载集群的场景下，内生延迟的值会更加糟糕。我们能够在显然正常运行的系统中测量高达40毫秒的差距。

Latency induced by network and communication

络和通信产生的延迟

Clients connect to Redis using a TCP/IP connection or a Unix domain connection. The typical latency of a 1 Gbit/s network is about 200 us, while the latency with a Unix domain socket can be as low as 30 us. It actually depends on your network and system hardware. On top of the communication itself, the system adds some more latency (due to thread scheduling, CPU caches, NUMA placement, etc …). System induced latencies are significantly higher on a virtualized environment than on a physical machine.

客户端使用TCP/IP连接或Unix域连接来连接到Redis服务。千兆 络的常规延迟大概200微秒，使用Unix域套接字可以达到低到30微秒。延迟取决于你的 络和系统硬件。除了通信本身之外，系统还增加了一些延迟（由于线程调度，CPU缓存，NUMA放置等原因）。虚拟化环境中的系统引起的延迟明显高于物理机。

The consequence is even if Redis processes most commands in sub microsecond range, a client performing many roundtrips to the server will have to pay for these network and system related latencies.

结果就是，即使Redis大部分命令处理时间在亚毫秒级别范围内，客户端到服务器端执行往返也会因为 络和系统的消耗而造成延迟。

An efficient client will therefore try to limit the number of roundtrips by pipelining several commands together. This is fully supported by the servers and most clients. Aggregated commands like MSET/MGET can be also used for that purpose. Starting with Redis 2.4, a number of commands also support variadic parameters for all data types.

一个高效的客户端会因为使用管道指执行一些命令来限制到服务端的往返次数。这在服务端和多数客户端都是支持的。聚合命令如 MSET/MGET也可以用来达到这个目的。从Redis2.4开始，许多命令还支持所有数据类型的可变参数（即一次可以传多个参数）。

Here are some guidelines:

这有一些指导方针：

• If you can afford it, prefer a physical machine over a VM to host the server.
  

  有能力上物理机，别用虚拟机。

• Do not systematically connect/disconnect to the server (especially true for web based applications). Keep your connections as long lived as possible.
  

  不要系统地连接/断开与服务器的连接（特别是对于基于Web的应用程序）。尽可能使用长连接。

• If your client is on the same host than the server, use Unix domain sockets.
  

  如果您的客户端与服务器位于同一主机上，请使用 Unix 域套接字

• Prefer to use aggregated commands (MSET/MGET), or commands with variadic parameters (if possible) over pipelining.
  

  首选使用聚合命令 （MSET/MGET）或带有可变参数的命令（如果可能）而不是流水线。

• Prefer to use pipelining (if possible) over sequence of roundtrips.
  

  首选使用流水线（如果可能）而不是往返顺序。

• Redis supports Lua server-side scripting to cover cases that are not suitable for raw pipelining (for instance when the result of a command is an input for the following commands).
  

  Redis 支持 Lua 服务器端脚本，以涵盖不适合原始管道的情况（例如，当命令的结果是后续命令的输入参数时）。

On Linux, some people can achieve better latencies by playing with process placement (taskset), cgroups, real-time priorities (chrt), NUMA configuration (numactl), or by using a low-latency kernel. Please note vanilla Redis is not really suitable to be bound on a single CPU core. Redis can fork background tasks that can be extremely CPU consuming like or . These tasks must never run on the same core as the main event loop.

在 Linux 上，有些人可以通过使用进程放置（任务集）、cgroups、实时优先级 （chrt）、NUMA 配置（numactl）或使用低延迟内核来实现更好的延迟。请注意，普通的 Redis 并不适合绑定在 单个 CPU 内核上。Redis 可以fork后台任务，这些任务可能非常消耗 CPU，例如 []（https://redis.io/commands/bgsave）或 []（https://redis.io/commands/bgrewriteaof） 。这些任务决不能在与主事件循环相同的内核上运行。

In most situations, these kind of system level optimizations are not needed. Only do them if you require them, and if you are familiar with them.

在大多数场景 ，这些系统级别优化并不必要。只有确实需要并且了解它们的情况下才能做。

Single threaded nature of Redis

Redis单线程的本质

Redis uses a mostly single threaded design. This means that a single process serves all the client requests, using a technique called multiplexing. This means that Redis can serve a single request in every given moment, so all the requests are served sequentially. This is very similar to how Node.js works as well. However, both products are not often perceived as being slow. This is caused in part by the small amount of time to complete a single request, but primarily because these products are designed to not block on system calls, such as reading data from or writing data to a socket.

Redis 使用 大部分 单线程设计。这意味着单个进程使用一种称为多路复用的技术为所有客户端请求提供服务。这意味着 Redis 可以在每个给定时刻为单个请求提供服务，因此所有请求都是按顺序处理的。这与 Node.js的工作方式非常相似。然而，这两种产品通常都不被认为是缓慢的。这部分是由于完成单个请求的时间很短，但主要是因为这些产品被设计为不阻塞系统调用，例如从套接字读取数据或将数据写入套接字。

I said that Redis is mostly single threaded since actually from Redis 2.4 we use threads in Redis in order to perform some slow I/O operations in the background, mainly related to disk I/O, but this does not change the fact that Redis serves all the requests using a single thread.

我说过 Redis大部分是单线程的，因为实际上从Redis 2.4开始，我们在Redis中使用线程，以便在后台执行一些缓慢的I/O操作，主要与磁盘I/O有关，但这并不能改变Redis使用单个线程为所有请求提供服务的事实。

Latency generated by slow commands

由慢命令产生的延迟

A consequence of being single thread is that when a request is slow to serve all the other clients will wait for this request to be served. When executing normal commands, like or or this is not a problem at all since these commands are executed in constant (and very small) time. However there are commands operating on many elements, like , , and others. For instance taking the intersection of two big sets can take a considerable amount of time.

单线程的结果是，当一个请求处理速度慢了，所有其他客户端将等待此请求处理完成。当执行常规命令时，如或或，这根本不是问题，因为这些命令是在恒定(和非常小)的时间内执行的。但是，有许多命令对许多元素进行操作，例如，，等。例如，采用两个大集合的交集可能需要相当长的时间。

The algorithmic complexity of all commands is documented. A good practice is to systematically check it when using commands you are not familiar with.

所有的算法复杂度都有记录。一个好的实践是，当你使用一个不熟悉的命令时，先系统地检查一下它。

If you have latency concerns you should either not use slow commands against values composed of many elements, or you should run a replica using Redis replication where you run all your slow queries.

如果您有延迟问题，那你就不应对由许多元素组成的值使用慢速命令，或者你应该在Redis副本上执行慢查询。

It is possible to monitor slow commands using the Redis Slow Log feature.

使用 Slow Log工具可以监控慢命令。

Additionally, you can use your favorite per-process monitoring program (top, htop, prstat, etc …) to quickly check the CPU consumption of the main Redis process. If it is high while the traffic is not, it is usually a sign that slow commands are used.

此外，您可以使用自己喜欢的进程监控程序（，，等）来快速检查主Redis进程的CPU消耗。如果它很高而流量不高，则通常表示使用了慢速命令。

IMPORTANT NOTE: a VERY common source of latency generated by the execution of slow commands is the use of the command in production environments. , as documented in the Redis documentation, should only be used for debugging purposes. Since Redis 2.8 a new commands were introduced in order to iterate the key space and other large collections incrementally, please check the , , and commands for more information.

重点注意：一个非常常见的因慢命令执行而产生的延迟是在生产环境的使用 命令。 命令如同在Redis文档中记录的一样，应该只被用于调试目的。Redis2.8后新引入增量迭代key和大集群的命令，具体查看 , , and 命令的信息。

Latency generated by fork

因产生的延迟

In order to generate the RDB file in background, or to rewrite the Append Only File if AOF persistence is enabled, Redis has to fork background processes. The fork operation (running in the main thread) can induce latency by itself.

为了在后台生成RDB文件，或者重写AOF文件，Redis会fork后台进程。在主纯种中进行fork操作会产生延迟。

Forking is an expensive operation on most Unix-like systems, since it involves copying a good number of objects linked to the process. This is especially true for the page table associated to the virtual memory mechanism

fork操作在多数Unix系统中是很重的操作，因为会复制主进程中的大量对象。在虚拟机的页表中尤其明显。

For instance on a Linux/AMD64 system, the memory is divided in 4 kB pages. To convert virtual addresses to physical addresses, each process stores a page table (actually represented as a tree) containing at least a pointer per page of the address space of the process. So a large 24 GB Redis instance requires a page table of 24 GB / 4 kB * 8 = 48 MB.

例如在 Linux或AMD64系统，内存以4kb的页进行分隔。为了转换虚拟地址到物理地址，每个进程会存储一个页表（实际以树形式展现）包含至少一个进程地址空间的指针。所以一个24G的Redis实例，需要一个48M的页表。

When a background save is performed, this instance will have to be forked, which will involve allocating and copying 48 MB of memory. It takes time and CPU, especially on virtual machines where allocation and initialization of a large memory chunk can be expensive.

当一个后台保存执行时，这个实例将被form，同时将分配和复制48M内存。这将会消耗时间和CPU，特别在虚拟机上，分配和初始化一个大内存块消耗很大。

Fork time in different systems

不同系统的Fork时间

Modern hardware is pretty fast at copying the page table, but Xen is not. The problem with Xen is not virtualization-specific, but Xen-specific. For instance using VMware or Virtual Box does not result into slow fork time. The following is a table that compares fork time for different Redis instance size. Data is obtained performing a BGSAVE and looking at the filed in the command output.

现代硬件复制页表很快，但是Xen虚拟机不是。XEN虚拟机的问题不是虚拟化特性，而是Xen特性。例如使用VMware或者VirtualBox不会减慢fork时间。以下是不同实例数下fork时间对比的表格。数据是运行bgsave后观察命令输出的字段取得的。

However the good news is that new types of EC2 HVM based instances are much better with fork times, almost on par with physical servers, so for example using m3.medium (or better) instances will provide good results.

好消息是EC2 HVM新类型的实例fork时间很好，和物理机差不多了，所以使用如m3.medium(或更好的)实例可以提供更好的效果。

• Linux beefy VM on VMware 6.0GB RSS forked in 77 milliseconds (12.8 milliseconds per GB).
• Linux running on physical machine (Unknown HW) 6.1GB RSS forked in 80 milliseconds (13.1 milliseconds per GB)
• Linux running on physical machine (Xeon @ 2.27Ghz) 6.9GB RSS forked into 62 milliseconds (9 milliseconds per GB).
• Linux VM on 6sync (KVM) 360 MB RSS forked in 8.2 milliseconds (23.3 milliseconds per GB).
• Linux VM on EC2, old instance types (Xen) 6.1GB RSS forked in 1460 milliseconds (239.3 milliseconds per GB).
• Linux VM on EC2, new instance types (Xen) 1GB RSS forked in 10 milliseconds (10 milliseconds per GB).
• Linux VM on Linode (Xen) 0.9GBRSS forked into 382 milliseconds (424 milliseconds per GB).

As you can see certain VMs running on Xen have a performance hit that is between one order to two orders of magnitude. For EC2 users the suggestion is simple: use modern HVM based instances.

如您所见，在 Xen 上运行的某些 VM 的性能下降幅度在一个数量级到两个数量级之间。对于 EC2 用户，建议很简单：使用基于 HVM 的现代实例。

Latency induced by transparent huge pages

由 透明大页导致的延迟

Unfortunately when a Linux kernel has transparent huge pages enabled, Redis incurs to a big latency penalty after the call is used in order to persist on disk. Huge pages are the cause of the following issue:

当Linux内容开启了透明大页（transparent huge pages， THP），Redis在为了在硬盘持久化而执行后会引起很大的延迟损失。大页面是以下问题的原因：

1. Fork is called, two processes with shared huge pages are created.

调用，创建两个共享大页面的进程。

2. In a busy instance, a few event loops runs will cause commands to target a few thousand of pages, causing the copy on write of almost the whole process memory.

在繁忙的实例中，运行几个事件循环将导致命令以几千页为目标，从而导致在写入时复制几乎涉及到整个进程内存。

3. This will result in big latency and big memory usage.

这会导致巨大延迟和巨大的内存占用。

Make sure to disable transparent huge pages using the following command:

确保要禁用透明大页，使用如下命令

Latency induced by swapping (operating system paging)

由 swapping(操作系统分页)产生的延迟

Linux (and many other modern operating systems) is able to relocate memory pages from the memory to the disk, and vice versa, in