We recommend the following setup for deploying Longhorn in production.

Important: We recommend enabling only one data engine (V1 or V2) per cluster. When both data engines are enabled simultaneously, each node runs separate instance-manager pods for V1 and V2, each with its own guaranteed CPU reservation. This significantly increases CPU overhead per node (the V2 instance-manager pod alone consumes at least 1 dedicated CPU core, which is configurable, for the spdk_tgt process). If your workloads do not require both engines, disable the unused one to conserve resources. For more information about guaranteed CPU reservations and how CPU consumption differs between V1 and V2 upgrade flows, see Guaranteed Instance Manager CPU.

• Minimum Recommended Hardware
  • V1 Data Engine
  • V2 Data Engine
• Architecture
• Operating System
• Kubernetes
  • Kubernetes Version
  • CoreDNS Setup
• Nodes and Disk Setup
  • Use a Dedicated Disk
  • Minimal Available Storage and Over-provisioning
  • Disk Space Management
  • Setting up Extra Disks
  • V2 Data Engine: Block-Type Disks
• Configuring Default Disks Before and After Installation
• Volumes Performance Optimization
  • IO Performance
  • Space Efficiency
  • Disaster Recovery
• Deploying Workloads
• Volumes Maintenance
• Guaranteed Instance Manager CPU
  • V1 Data Engine
  • V2 Data Engine
• StorageClass
• Scheduling Settings
  • Replica Node Level Soft Anti-Affinity
  • Allow Volumes Creation with Degraded Availability
  • Replica Auto-Balance

Minimum Recommended Hardware

V1 Data Engine

• 3 nodes
• 4 vCPUs per node
• 4 GiB per node
• SSD/NVMe or similar performance block device on the node for storage (recommended)
• HDD/Spinning Disk or similar performance block device on the node for storage (verified)
  • 500/250 max IOPS per volume (1 MiB I/O)
  • 500/250 max throughput per volume (MiB/s)

Warning: While Longhorn can function with HDDs (spinning disks) as storage, it is important to understand that latency plays a much more important role in volume stability than IOPS or throughput. This is because HDDs are mechanical, relying on spinning platters and moving read/write heads to access data. This physical movement introduces inherent delays (seek time and rotational delay), leading to much higher latency compared to the SSDs or NVMe drives, which utilize flash memory and have no moving parts. This can directly cause instability, especially when multiple input-output intensive tasks are running, such as:

• Foreground IOs to the replicas
• Foreground IOs from the replicas
• Rebuilding volumes
• Backups or other workloads

The increased latency due to the use of HDDs, combined with other input-output workloads, can lead to volume instability. Therefore, we recommend SSD or NVMe drives for better performance and stability, especially for production workloads.

The mentioned IOPS and throughput (500/250 max IOPS per volume and 500/250 max throughput per volume) are intended as general references based on the test setup but should not be treated as hard requirements. Latency, not just throughput, is the most important factor in ensuring system stability.

V2 Data Engine

In addition to the V1 requirements above, nodes hosting V2 volumes have these additional requirements:

• 3 nodes
• Additional 1 CPU core per node dedicated to each V2 instance-manager pod (the spdk_tgt process uses intensive polling and consumes 100% of a dedicated CPU core)
• Additional 2 GiB memory per node reserved for huge pages (1024 × 2 MiB-sized pages)
• Local NVMe SSDs are strongly recommended for V2 volumes to achieve optimal storage performance
• Linux kernel 6.7 or later for NVMe/TCP support and better stability.
• Required kernel modules: vfio_pci, uio_pci_generic, nvme-tcp
• AMD64 CPUs require SSE4.2 instruction support

Note: The V2 Data Engine leverages the Storage Performance Development Kit (SPDK) with user space NVMe drivers that provide zero-copy, highly parallel, direct access to SSDs. Using local NVMe disks is strongly recommended for enabling V2 volumes to achieve optimal storage performance. For the full setup guide, see V2 Data Engine Requirements.

Architecture

Longhorn supports the following architectures:

1. AMD64
2. ARM64

Operating System

Note: CentOS Linux has been removed from the verified OS list below, as it has been discontinued in favor of CentOS Stream [ref], a rolling-release Linux distribution. Our focus for verifying RHEL-based downstream open source distributions will be enterprise-grade, such as Rocky and Oracle Linux.

The following Linux OS distributions and versions have been verified during the v1.12.0 release testing. However, this does not imply that Longhorn exclusively supports these distributions. Essentially, Longhorn should function well on any certified Kubernetes cluster running on Linux nodes with a wide range of general-purpose operating systems, as well as verified container-optimized operating systems like SLE Micro.

Longhorn relies heavily on kernel functionality and performs better on some kernel versions. The following activities, in particular, benefit from usage of specific kernel versions.

• Optimizing or improving the filesystem: Use a kernel with version v5.8 or later. See Issue #2507 for details.
• Enabling the Freeze Filesystem for Snapshot setting: Use a kernel with version 5.17 or later to ensure that a volume crash during a filesystem freeze cannot lock up a node.
• Enabling the V2 Data Engine: Use a kernel with version 5.19 or later to ensure NVMe/TCP support. Use a kernel with version 6.7 or later for improved stability (avoids potential memory corruption from SPDK upstream issue #3116).

The list below contains known broken kernel versions that users should avoid using:

Kubernetes

Kubernetes Version

Please ensure your Kubernetes cluster is at least v1.21 before upgrading to Longhorn v1.12.0 because this is the minimum version Longhorn v1.12.0 supports.

We recommend running your Kubernetes cluster on one of the following versions. These versions are the active supported versions prior to the Longhorn release, and have been tested with Longhorn v1.12.0.

Referenced to https://endoflife.date/kubernetes.

CoreDNS Setup

Ensure that CoreDNS runs with at least 2 replicas to maintain high availability. This setup minimizes interruptions in the DNS resolution if one CoreDNS pod experiences a temporary disruption.

Nodes and Disk Setup

We recommend the following setup for nodes and disks.

Use a Dedicated Disk

It’s recommended to dedicate a disk for Longhorn storage for production, instead of using the root disk.

Minimal Available Storage and Over-provisioning

If you need to use the root disk, use the default minimal available storage percentage setup which is 25%, and set overprovisioning percentage to 100% to minimize the chance of DiskPressure.

If you’re using a dedicated disk for Longhorn, you can lower the setting minimal available storage percentage to 10%.

For the Over-provisioning percentage, it depends on how much space your volume uses on average. For example, if your workload only uses half of the available volume size, you can set the Over-provisioning percentage to 200, which means Longhorn will consider the disk to have twice the schedulable size as its full size minus the reserved space.

Disk Space Management

Since Longhorn doesn’t currently support sharding between the different disks, we recommend using LVM to aggregate all the disks for Longhorn into a single partition, so it can be easily extended in the future.

Setting up Extra Disks

Any extra disks must be written in the /etc/fstab file to allow automatic mounting after the machine reboots.

Don’t use a symbolic link for the extra disks. Use mount --bind instead of ln -s and make sure it’s in the fstab file. For details, see the section about multiple disk support.

V2 Data Engine: Block-Type Disks

Unlike the V1 Data Engine which uses filesystem-type disks, the V2 Data Engine stores volume data on block-type disks. The following best practices apply to V2 block-type disk setup:

• Use local NVMe disks: SPDK is equipped with a user space NVMe driver that provides zero-copy, highly parallel, direct access to SSDs. Using local NVMe disks is strongly recommended for V2 volumes.

• Ensure disk is clean before adding: Starting with v1.11.0, Longhorn prevents adding block disks that contain an existing file system or partition table. Run wipefs -a /path/to/block/device before adding a disk.

• IOMMU group isolation: For SPDK to claim a disk via vfio-pci, the NVMe device must be in an isolatable IOMMU group. If the device shares an IOMMU group with a PCIe bridge, it cannot be used with the SPDK NVMe driver and must be used in AIO mode instead.

• Huge pages configuration: Ensure 2 GiB of 2 MiB-sized huge pages (1024 pages) are configured persistently on each V2 node via kernel boot parameters. See Enable HugePages for detailed instructions.

• Kernel modules: Ensure vfio_pci, uio_pci_generic, and nvme-tcp modules are loaded and configured to load automatically at boot. See Load Kernel Modules for detailed instructions.

For the complete V2 Data Engine setup guide, see V2 Data Engine Requirements.

Configuring Default Disks Before and After Installation

To use a directory other than the default /var/lib/longhorn for storage, the Default Data Path setting can be changed before installing the system. For details on changing pre-installation settings, refer to this section.

The Default node/disk configuration feature can be used to customize the default disk after installation. Customizing the default configurations for disks and nodes is useful for scaling the cluster because it eliminates the need to configure Longhorn manually for each new node if the node contains more than one disk, or if the disk configuration is different for new nodes. Remember to enable Create default disk only on labeled node if applicable.

Volumes Performance Optimization

Before configuring workloads, ensure that you have set up the following basic requirements for optimal volume performance.

• SATA/NVMe SSDs or disk drives with similar performance
• 10 Gbps network bandwidth between nodes
• Dedicated Priority Class for system-managed and user-deployed Longhorn components. By default, Longhorn installs the default Priority Class longhorn-critical.

The following sections outline other recommendations for production environments.

IO Performance

• Storage network: Use a dedicated storage network to improve IO performance and stability.

• Longhorn disk: Use a dedicated disk for Longhorn storage instead of using the root disk.

• Replica count: Set the default replica count to “2” to achieve data availability with better disk space usage or less impact to system performance. This practice is especially beneficial to data-intensive applications.

• Storage tag: Use storage tags to define storage tiering for data-intensive applications. For example, only high-performance disks can be used for storing performance-sensitive data.

• Data locality: Use best-effort as the default data locality of Longhorn StorageClasses.

  For applications that support data replication (for example, a distributed database), you can use the strict-local option to ensure that only one replica is created for each volume. This practice prevents the extra disk space usage and IO performance overhead associated with volume replication.

  For data-intensive applications, you can use pod scheduling functions such as node selector or taint toleration. These functions allow you to schedule the workload to a specific storage-tagged node together with one replica.

Space Efficiency

• Recurring snapshots: Periodically clean up system-generated snapshots and retain only the number of snapshots that makes sense for your implementation.

  For applications with replication capability, periodically delete all types of snapshots.

• Recurring filesystem trim: Periodically trim the filesystem inside volumes to reclaim disk space.

• Snapshot space management: Configure global and volume-specific settings to prevent unexpected disk space exhaustion.

Disaster Recovery

• Recurring backups: Create recurring backup jobs for mission-critical application volumes.

• System backup: Create periodic system backups.

Deploying Workloads

If you’re using ext4 as the filesystem of the volume, we recommend adding a liveness check to workloads to help automatically recover from a network-caused interruption, a node reboot, or a Docker restart. See this section for details.

Volumes Maintenance

Using Longhorn’s built-in backup feature is highly recommended. You can save backups to an object store such as S3 or to an NFS server. Saving to an object store is preferable because it generally offers better reliability. Another advantage is that you do not need to mount and unmount the target, which can complicate failover and upgrades.

For each volume, schedule at least one recurring backup. If you must run Longhorn in production without a backupstore, then schedule at least one recurring snapshot for each volume.

Longhorn system will create snapshots automatically when rebuilding a replica. Recurring snapshots or backups can also automatically clean up the system-generated snapshot.

Guaranteed Instance Manager CPU

We recommend setting the CPU request for Longhorn instance manager pods.

V1 Data Engine

The Guaranteed Instance Manager CPU setting allows you to reserve a percentage of the total allocatable CPU resources on each node for each instance manager pod when the V1 Data Engine is enabled. The default value is 12.

You can also set a specific milli CPU value for instance manager pods on a particular node by updating the node’s Instance Manager CPU Request field.

Note: This field will overwrite the above setting for the specified node.

Refer to Guaranteed Instance Manager CPU for more details.

V2 Data Engine

The Guaranteed Instance Manager CPU setting allows you to reserve a percentage of the total allocatable CPU resources on each node for each instance manager pod when the V2 Data Engine is enabled. This reservation applies to the entire V2 instance manager pod.

The primary CPU consumer in the pod is the Storage Performance Development Kit (SPDK) target daemon (spdk_tgt). By default, spdk_tgt typically uses 1 dedicated CPU core in polling mode. The Data Engine CPU Mask setting controls which CPU cores spdk_tgt runs on.

Reserving sufficient CPU is essential for maintaining engine and replica stability, especially during periods of high node workload. The default value of the Guaranteed Instance Manager CPU setting is 12%.

Note: When both V1 and V2 Data Engines are enabled on the same node, separate instance-manager pods are created for each engine. Plan CPU resources accordingly to ensure each instance-manager pod has sufficient CPU allocation.

StorageClass

We don’t recommend modifying the default StorageClass named longhorn, since the change of parameters might cause issues during an upgrade later. If you want to change the parameters set in the StorageClass, you can create a new StorageClass by referring to the StorageClass examples.

Scheduling Settings

Replica Node Level Soft Anti-Affinity

Recommend: false

This setting should be set to false in production environment to ensure the best availability of the volume. Otherwise, one node down event may bring down more than one replicas of a volume.

Allow Volumes Creation with Degraded Availability

Recommend: false

This setting should be set to false in production environment to ensure every volume have the best availability when created. Because with the setting set to true, the volume creation won’t error out even there is only enough room to schedule one replica. So there is a risk that the cluster is running out of the spaces but the user won’t be made aware immediately.

Replica Auto-Balance

Recommend: least-effort

For production environments, we recommend setting Replica Auto-Balance to least-effort. This setting ensures that at least one replica is placed on a different node in each zone, providing extra high availability (HA).

In certain edge cases, you might consider using the best-effort, which continuously attempts to evenly distribute replicas across nodes and zones. However, this setting can lead to frequent rebuilds if the cluster is unstable.

For most users, having multiple replicas without Replica Auto-Balance setting is sufficient to achieve basic HA, especially if you prefer to avoid excessive rebuilds and resource usage.