Tweet This: Tweet
Share on LinkedIn:
By Ross Peterson, Kovarus, Storage Engineer
I have met some confusion recently as it pertains to storage protocols and the physical media that corresponds to them. More specifically, iSCSI has risen in the data center as a worthy rival to good old fashioned Fibre Channel. The prior limitations of the 1Gbps speed of iSCSI in the past was inefficient for most high-performance SAN workflows and business critical applications today. Recently though, iSCSI speeds have improved to 10Gbps via optical cable or twinax, and is no longer restricted to just a copper RJ45 cable operating at 1Gbps.
So, what is iSCSI then?
In a nutshell, iSCSI is a SCSI frame encapsulated in an IP packet, and it can be transmitted via either 1Gbps or 10Gbps transmit speeds. iSCSI contains the same reliable command and response protocol that Fibre Channel (FC) has, only it can be transmitted across an existing IP network. The primary advantage to iSCSI is the fact that if your organization does not already have an FC environment set up, you can leverage your existing IP infrastructure without the need to purchase a separate FC SAN network. Obviously, this can be a huge advantage for reducing CAPEX when looking to implement a new storage system/infrastructure, but there are a few best practices that should be considered to make iSCSI a high-performing and robust replacement for FC.
- In an iSCSI environment, it is best to have a separate subnet and/or network dedicated for iSCSI traffic. This helps reduce performance risk to the existing IP infrastructure. If it is possible to use entirely different switches, then do so. If that is not possible, separate VLANS should be dedicated to iSCSI at a minimum.
- Redundancy is our best friend when it comes to any IT system, therefore, it is best practice to have two separate subnets, one per NIC/HBA on the host (initiator) side. It is also recommended to have each subnet/VLAN on a different physical switch if possible (see figure 1).
- If your organization has multiple, different types of storage systems, it is a good idea to create separate subnets for each of those systems. Although it varies by vendor, some storage systems have unique performance requirements over the IP network, and having a separate subnet per storage system enables maximum flexibility, performance, and QoS. For example, it has been noted that a VNX may benefit from jumbo frames when they exhibit a larger, more sequential workload, while the smaller and randomized I/O from an all-flash array such as an XtremIO may benefit from a non-jumbo frame environment.
- Although a major benefit to iSCSI is the ability to use existing IP hardware such as a regular NIC on the host-side, it is recommended to use what is called a TCP offload engine (TOE) if possible. These specialized HBAs contain software and firmware that enables them to process the TCP and iSCSI protocol overhead from the host CPU and OS, which may help with host performance. Most storage system vendors have specific I/O modules that are a TOE type, so be sure to check with your vendor.
- Be sure to enable CHAP for security. Without CHAP enabled, any host, client, server, etc. that has access to the iSCSI subnet and iSCSI initiator software will be able to access the storage unchallenged. CHAP is a security handshake protocol which enables authentication from initiator to target via a secret password. With CHAP enabled, any system that does not know the secret will not be able to establish an iSCSI session.
What Cable do I use?
Most storage vendors support iSCSI at two speeds: 1Gps and 10Gps. If using the 1Gps speed, a regular CAT5e/Cat6/802.3 RJ45 cable is used. If 10Gbps is used, most vendors support two options: optical LC cables (just like in Fibre Channel figure 2), and twinax copper. When using optical cables, be sure that you have the proper SFP+ transceivers installed at the switch-side, and that your storage system comes with the necessary SFP+ transceivers. If the storage system does not come with these transceivers, most vendors require a specific model and firmware version, so be sure to get the recommendation from the vendor when it comes to compatibility. When twinax cables are used, the cable is copper and is pre-connected to an SFP+ transceiver, so a separate SFP+ is not needed for either the switch-side or storage system. Be sure to check with your vendor on the supported twinax cable types such as if active or passive cables are needed. Some vendors such as EMC only support active twinax cables, and storage arrays such as the VNX require a specific SFP+ transceiver or it will fault the system. See figure 3 for a twinx cable, and figure 4 for an SFP+.
Figure 2. Traditional Optical Cable.
Figure 3. Twinax Cable.
Figure 4. SFP+ Transceiver.
While this post may be as clear as mud, hopefully it will answer the more common questions about how to design and implement iSCSI for a SAN environment. Since 10Gbps iSCSI can take either an optical cable or twinax form, it is important to determine which type is best to be used in the data center, and double check that the compatible technology of the cables/SFP+ have been ordered before the day of the install.