NVMe over TCP in enterprise storage: advantages and implementation practice

Today's storage systems (SSDs) must meet ever-increasing demands for performance and infrastructure flexibility. The advent of NVMe (Non-Volatile Memory Express) SSDs revolutionized the speed of data access, but to unlock their full potential, data transfer protocols had to evolve accordingly.

NVMe over TCP (NVMe/TCP) is a protocol designed for efficient use of NVMe drives in standard IP networks. It allows to transfer NVMe commands directly through the TCP/IP stack, providing minimum latency and maximum performance without the need for specialized infrastructure. This approach is especially relevant for organizations looking to modernize storage systems to meet today's performance, scalability, and cost-efficiency requirements.

Architecture and operating principles

NVMe over TCP realizes direct connection of processor to NVMe-storage devices via standard TCP/IP network stack and the most common Ethernet-switches.

The key features of the protocol are:

• Preservation of NVMe efficiency during network transmission due to the optimized command and data format
• Minimal overhead for protocol processing
• Built-in support for multipathing access (multipathing)
• Reliability mechanisms at the TCP level
• Ability to use standard network functions, including load balancing.

Technical advantages

NVMe over TCP provides end-to-end (end-to-end) access to NVMe drives. From a performance standpoint, NVMe/TCP provides minimal latency due to the high-speed NVMe communication protocol and efficient use of network bandwidth. At the same time, there is no need for special QoS mechanisms, which greatly simplifies configuration and maintenance.

Special attention is paid to the fault tolerance of the system. Protocol architecture supports asymmetric access through both controllers to NVMe Namespace. In case of failure of one of the controllers there is an automatic switch to the optimal path, and in case of power failures the guaranteed data preservation with minimal recovery time is provided.

Ease of deployment

NVMe over TCP eliminates the need for specialized hardware, allowing you to leverage your existing network infrastructure. This makes it easy and affordable to implement for a wide range of users.

1. The solution's affordability is due to the lack of specialized requirements:

• Operation on standard Ethernet switches;
• No need for Lossless Ethernet support;
• Ability to disable complex QoS mechanisms without performance loss.

2. Protocol implementation simplifies administration:

• Single network infrastructure for all traffic types;
• Familiar monitoring and management tools;
• Built-in support for modern automation tools.

3. Protocol has high scalability flexibility:

• Easy addition of new nodes;
• Linear performance growth with expansion;
• Support for different classes of storage within a single infrastructure.
  

Compatibility with existing infrastructure

Implementing NVMe over TCP does not require major infrastructure changes. Organizations can make a gradual transition from traditional protocols, while maintaining investments in existing network infrastructure and monitoring systems. The protocol integrates well with modern software-defined environments, supports work with container platforms and orchestration systems.

Compared to traditional data transfer protocols, NVMe over TCP demonstrates its advantages through the use of standard network infrastructure. NVMe over TCP runs on top of standard Ethernet networks, which greatly simplifies implementation and reduces total cost of ownership.

Practical implementation

Theoretical advantages of IP-protocols find their confirmation in real implementations. An illustrative example of successful implementation of the modern approach to data storage systems is a line of storage systems from the Russian manufacturer ITPOD, part of ITG corporation, completely built on the principles of work in IP-networks and supporting the provision of disk resources via NVMe over TCP protocol.

When developing All-flash data storage systems, ITPOD engineers were guided by the principles we discussed above: maximum use of the advantages of modern IP-networks, rejection of outdated technological limitations and focus on real business needs. The result is the Full Flash line, which demonstrates in practice all the advantages of the new architecture.

Technological foundation

In addition to standard block access via iSCSI, the storage system supports the NVMe over TCP protocol, which allows to maximize the use of modern NVMe SSD drives in standard IP-networks. The solution takes into account various usage scenarios: from high-performance systems based on TLC SSDs for mission-critical applications to cost-effective configurations with QLC SSDs for tasks with less stringent performance requirements.

Special attention is paid to optimizing performance on IP networks. Built-in load balancing and data flow control mechanisms ensure stable operation even under heavy mixed traffic conditions, which is especially important for unified infrastructures. At the same time, the system does not require complex configuration or special network configuration, fully realizing the principle of ease of implementation.

Practical results

The experience of operating systems based on NVMe over TCP in various organizations confirms the effectiveness of this technology. In particular, customers note

• Significant reduction of total cost of ownership due to the use of standard network infrastructure;
• Simplified administration due to a single network infrastructure;
• High and predictable performance in a variety of use cases;
• Flexibility to scale and modernize;
• Reduced time to deploy new services.
  

Import substitution in practice

Given the need for technological sovereignty, the fact that the storage system is managed by a Russian-developed operating system is of particular value. This provides guaranteed technical support and has its own advantages:

• Regular software updates;
• Prompt elimination of possible problems;
• Adaptation to specific customer requirements;
• Compatibility with other domestic solutions.

Conclusion

Introduction of NVMe over TCP opens a new chapter in the development of data storage systems. This protocol not only provides high performance and reliability, but also greatly simplifies the deployment and operation of storage systems. On the example of ITPOD solutions we see how theoretical advantages of the technology are translated into practical results: reduction of total cost of ownership, simplified administration and flexibility of scaling.

Of particular importance is the fact that the introduction of NVMe over TCP supports the course on technological independence. The use of open standards and broad support from Ethernet equipment manufacturers create a reliable basis for the development of domestic storage solutions.