High Performance Computing Fabrics

Intel® Omni-Path

Fabric 100 Series

Intel® Omni-Path

Host Fabric Interface

Intel® Omni-Path Edge

Switches 100 Series

Intel® Omni-Path Director

Class Switches 100 Series

Intel® Omni-Path Fabric

Software Components

The Next-Generation Fabric

 

Intel® Omni-Path Architecture (Intel® OPA) delivers the performance for tomorrow’s high performance computing (HPC) workloads and the ability to scale to tens of thousands of nodes—and eventually more—at a price competitive with today’s fabrics. The Intel® OPA 100 Series product line is an end-to-end solution of PCIe* adapters, silicon, switches, cables, and management software. As the successor to Intel® True Scale Fabric, this optimized HPC fabric is built upon a combination of enhanced IP and Intel® technology.

 

For software applications, Intel® OPA will maintain consistency and compatibility with existing Intel® True Scale Fabric and InfiniBand* APIs by working through the open source OpenFabrics Alliance (OFA) software stack on leading Linux* distribution releases. Intel® True Scale Fabric customers will be able to migrate to Intel® OPA through an upgrade program.

The Future of High Performance Fabrics

 

Current standards-based high performance fabrics, such as InfiniBand*, were not originally designed for HPC, resulting in performance and scaling weaknesses that are currently impeding the path to Exascale computing. Intel® Omni-Path Architecture is being designed specifically to address these issues and scale cost-effectively from entry level HPC clusters to larger clusters with 10,000 nodes or more. To improve on the InfiniBand specification and design, Intel® is using the industry’s best technologies including those acquired from QLogic and Cray alongside Intel® technologies.

 

While both Intel® OPA and InfiniBand Enhanced Data Rate (EDR) will run at 100Gbps, there are many differences. The enhancements of Intel® OPA will help enable the progression towards Exascale while cost-effectively supporting clusters of all sizes with optimization for HPC applications at both the host and fabric levels for benefits that are not possible with the standard InfiniBand-based designs.

 

Intel® OPA is designed to provide the:

 

  • Features and functionality at both the host and fabric levels to greatly raise levels of scaling
  • CPU and fabric integration necessary for the increased computing density, improved reliability, reduced power, and lower costs required by significantly larger HPC deployments
  • Fabric tools to readily install, verify, and manage fabrics at this level of complexity

 

Intel® Omni-Path Key Fabric Features and Innovations

 

Adaptive Routing

 

Adaptive Routing monitors the performance of the possible paths between fabric end-points and selects the least congested path to rebalance the packet load. While other technologies also support routing, the implementation is vital. Intel®’s implementation is based on cooperation between the Fabric Manager and the switch ASICs. The Fabric Manager—with a global view of the topology—initializes the switch ASICs with several egress options per destination, updating these options as the fundamental fabric changes when links are added or removed. Once the switch egress options are set, the Fabric Manager monitors the fabric state, and the switch ASICs dynamically monitor and react to the congestion sensed on individual links. This approach enables Adaptive Routing to scale as fabrics grow larger and more complex.

 

Dispersive Routing

 

One of the critical roles of fabric management is the initialization and configuration of routes through the fabric between pairs of nodes. Intel® Omni-Path Fabric supports a variety of routing methods, including defining alternate routes that disperse traffic flows for redundancy, performance, and load balancing. Instead of sending all packets from a source to a destination via a single path, Dispersive Routing distributes traffic across multiple paths. Once received, packets are reassembled in their proper order for rapid, efficient processing. By leveraging more of the fabric to deliver maximum communications performance for all jobs, Dispersive Routing promotes optimal fabric efficiency.

 

Traffic Flow Optimization

 

Traffic Flow Optimization optimizes the quality of service beyond selecting the priority—based on virtual lane or service level—of messages to be sent on an egress port. At the Intel® Omni-Path Architecture link level, variable length packets are broken up into fixed-sized containers that are in turn packaged into fixed-sized Link Transfer Packets (LTPs) for transmitting over the link. Since packets are broken up into smaller containers, a higher priority container can request a pause and be inserted into the ISL data stream before completing the previous data.

 

The key benefit is that Traffic Flow Optimization reduces the variation in latency seen through the network by high priority traffic in the presence of lower priority traffic. It addresses a traditional weakness of both Ethernet and InfiniBand* in which a packet must be transmitted to completion once the link starts even if higher priority packets become available.

 

Packet Integrity Protection

 

Packet Integrity Protection allows for rapid and transparent recovery of transmission errors between a sender and a receiver on an Intel® Omni-Path Architecture link. Given the very high Intel® OPA signaling rate (25.78125G per lane) and the goal of supporting large scale systems of a hundred thousand or more links, transient bit errors must be tolerated while ensuring that the performance impact is insignificant. Packet Integrity Protection enables recovery of transient errors whether it is between a host and switch or between switches. This eliminates the need for transport level timeouts and end-to-end retries. This is done without the heavy latency penalty associated with alternate error recovery approaches.

 

Dynamic Lane Scaling

 

Dynamic Lane Scaling allows an operation to continue even if one or more lanes of a 4x link fail, saving the need to restart or go to a previous checkpoint to keep the application running. The job can then run to completion before taking action to resolve the issue. Currently, InfiniBand* typically drops the whole 4x link if any of its lanes drops, costing time and productivity.

 

Ask How Intel® Omni-Path Architecture Can Meet Your HPC Needs

 

Intel® is clearing the path to Exoscale computing and addressing tomorrow’s HPC issues. Contact your Intel® representative or any authorized Intel® True Scale Fabric provider to discuss how Intel® Omni-Path Architecture can improve the performance of your future HPC workloads.

Intel® Omni-Path Host Fabric Interface (HFI)

 

Designed specifically for HPC, the Intel® Omni-Path Host Fabric Interface (Intel® OPA HFI) uses an advanced connectionless design that delivers performance that scales with high node and core counts, making it the ideal choice for the most demanding application environments.  The Intel® OPA HFI supports 100Gb/s per port, which means each HFI port can deliver up to 25GBps per port of bidirectional bandwidth. The same ASIC utilized in the HFI will also be integrated into future Intel® Xeon® processors, and used in third party products.

Intel® Omni-Path Host Fabric Interface (HFI) Optimizations

 

Much of the improved HPC application performance and low end-to-end latency at scale comes from the following enhancements:

 

Enhanced Performance Scaled Messaging (PSM).

 

The application view of the fabric is derived heavily from and is application-level software compatible with the demonstrated scalability of the Intel® True Scale Fabric architecture.  This has been accomplished by leveraging an enhanced next generation version of the Performance Scaled Messaging (PSM) library, which has already proven its scalability advantage through major deployments by the U.S. Government Department of Energy, among others.  PSM is specifically designed for the Message Passing Interface (MPI) and is very lightweight compared to using verbs – about 1/10th the user space code of verbs.  This leads to extremely high MPI and PGAS (Partitioned Global Address Space) message rates (i.e. short message efficiency) compared to using InfiniBand verbs.

 

“Connectionless” message routing.

 

The Intel® Omni-Path Architecture based on a connectionless design does not establish connection address information between nodes, cores, or processes, as opposed to a traditional implementation where this information is maintained in the cache of the adapter.  As a result, connectionless design delivery consistent latency independent of the scale or messaging partners.  This implementation offers greater potential to scale performance across a large node/core count cluster, while maintaining low end-to-end latency as the application is scaled across the cluster.

Overview

 

The next generation of High performance computing (HPC) fabrics use the Intel® Omni-Path Architecture to create fabrics that meet the needs of the most demanding set of applications. The Intel® Omni-Path Fabric Edge Switch consists 2 models, an entry-level 24 port switch for small clusters and a 48 port switch, both supporting 100Gb/s for all ports.

 

The larger switch, in addition to enabling a 48-port fabric in 1U, can be combined with other edge switches and directors to build much larger multi-tier fabrics. These Intel® Omni-Path Edge Switch are members of the Intel® Omni-Path Fabric 100 series of switches, host adapters and software delivering an exceptional set of high-speed networking features and functions.

Intel® Omni-Path Architecture Switch Fabric Optimizations

 

While similar to existing technology, the Intel® Omni-Path Architecture has been enhanced to overcome the scaling challenges of large-sized clusters.

 

These enhancements include:

 

High Message Rate Throughput. Intel® Omni-Path Architecture is designed to support high message rate traffic from each node through the fabric.  With ever-increasing processing power and core counts in Intel® Xeon® and Intel® Xeon® Phi processors, means the fabric has to support high bandwidth as well as message rate throughput.

 

48-port Switch ASIC. The Intel® Omni-Path Architecture switch 48 port design provides for improved fabric scalability, reduced latency, increased density and reduced cost & power.   In fact, the 48-port ASIC can enable 2-tier configurations up to 27,648 nodes, or over 2.3x what’s possible with current InfiniBand solutions.   Depending on fabric size, this can reduce fabric infrastructure requirements in a typical fat tree configuration by over 50%, since fewer switches, cables, racks, and power are needed as compared to today’s 36-port switch ASICs.[i]  Table 1 summarizes the Intel® Omni-Path Architecture advantages.

 

Deterministic Latency. Features in the Intel® Omni-Path Architecture will help minimize the negative performance impacts of large MTUs on small messages, thereby helping maintain consistent latency for InterProcess Communication (IPC) messages (e.g. MPI messages) when large messages (typically storage) are being simultaneously transmitted in the fabric.  This will allow the Intel® Omni-Path Architecture to bypass lower priority large packets to allow higher priority small packets creating a low and more predictable latency through the fabric.

 

Enhanced End-to-End Reliability.  The Intel® Omni-Path Architecture will also deliver efficient detection and error correction, which is expected to be much more efficient then Forward Error Correction (FEC) defined in the InfiniBand Standard.  Enhancements include zero load for detection, and if a correction is required, packets only need to being re-transmitted from the last link, and not all the way from the sending node, which enables near zero additional latency for a correction.

 

Intel® Omni-Path Edge Switch 100 Series

 

Overview

 

Intel® Omni-Path Director Class Switches (DCS) are constructed utilizing Intel®’s next generation 48 radix switch silicon which possesses many innovative features meant to provide optimum performance for both small and large fabrics. Both models are dense form factor designs capable of supporting up to 768 100 Gb/s ports in a low 20U footprint.

 

The Intel® Omni-Path Fabric Director Class Switches, 100 Series were designed to be modular such that customers can tailor their system configuration to meet present and future needs. Along with edge switches, host adapters and software.

Intel® Omni-Path Architecture Switch Fabric Optimizations

 

While similar to existing technology, the Intel® Omni-Path Architecture has been enhanced to overcome the scaling challenges of large-sized clusters.

 

These enhancements include:

 

High Message Rate Throughput.  Intel® Omni-Path Architecture is designed to support high message rate traffic from each node through the fabric.  With ever-increasing processing power and core counts in Intel® Xeon® and Intel® Xeon® Phi processors, means the fabric has to support high bandwidth as well as message rate throughput.

 

48-port Switch ASIC.  The Intel® Omni-Path Architecture switch 48 port design provides for improved fabric scalability, reduced latency, increased density and reduced cost & power.   In fact, the 48-port ASIC can enable 2-tier configurations up to 27,648 nodes, or over 2.3x what’s possible with current InfiniBand solutions.   Depending on fabric size, this can reduce fabric infrastructure requirements in a typical fat tree configuration by over 50%, since fewer switches, cables, racks, and power are needed as compared to today’s 36-port switch ASICs.   Table 1 summarizes the Intel® Omni-Path Architecture advantages.

 

Deterministic Latency.  Features in the Intel® Omni-Path Architecture will help minimize the negative performance impacts of large MTUs on small messages, thereby helping maintain consistent latency for InterProcess Communication (IPC) messages (e.g. MPI messages) when large messages (typically storage) are being simultaneously transmitted in the fabric.  This will allow the Intel® Omni-Path Architecture to bypass lower priority large packets to allow higher priority small packets creating a low and more predictable latency through the fabric.

 

Enhanced End-to-End Reliability.  The Intel® Omni-Path Architecture will also deliver efficient detection and error correction, which is expected to be much more efficient then Forward Error Correction (FEC) defined in the InfiniBand Standard.  Enhancements include zero load for detection, and if a correction is required, packets only need to being re-transmitted from the last link, and not all the way from the sending node, which enables near zero additional latency for a correction.

 

Intel® Omni-Path Director Class Switches 100 Series

Software Components

 

Intel® Omni-Path Architecture software comprises the Intel® OPA Host Software Stack and the Intel® Fabric Suite.

 

Intel® OPA Host Software

 

Intel®’s host software strategy is to utilize the existing OpenFabrics Alliance interfaces, thus ensuring that today’s application software written to those interfaces run with Intel® OPA with no code changes required.  This immediately enables an ecosystem of applications to “just work.”

 

All of the Intel® Omni-Path host software is being open sourced.

 

As with previous PSM generations PSM provides a fast data path with an HPC-optimized lightweight software (SW) driver layer.  In addition, standard I/O-focused protocols are supported via the standard verbs layer.

 

Intel® Fabric Suite:

 

Provides comprehensive control of administrative functions using a mature Subnet Manager. With advanced routing algorithms, powerful diagnostic tools and full subnet manager failover, the Fabric Manager simplifies subnet, fabric, and individual component management, easing the deployment and optimization of large fabrics.

 

Intel® Fabric Manager GUI

 

Provides an intuitive, scalable dashboard and analysis tools for viewing and monitoring fabric status and configuration.  The GUI may be run on a Linux* or Windows* desktop/laptop system with TCP/IP connectivity to the Fabric Manager.

Intel® Omni-Path Fabric Suite FastFabric Toolset

 

Guided by an intuitive interface, provides for rapid, error-free installation and configuration of Intel® OPA host software and management software tools, as well as simplified installation, configuration, validation, and optimization of HPC fabrics.

client reviews

find us

4921 Legends Dr

Lawrence, KS 66049

(785) 841-9559

sales@atipa.com

Q Security

Clearance

Engineers

Schedule #35F-0439P

Copyright© 2001-2017 Microtech Computers Inc.

All rights reserved.

Atipa Technologies, a division of Microtech Computers, Inc., is not responsible for typographical or photographic errors. Designated trademarks and brands are the property of their respective owners.