Understanding the FCOE Technology

I have recently started following the blogosphere re: FCOE technology and here is a handy references of a few things i found interesting.

Nigel Poulton http://blogs.rupturedmonkey.com/.

YouTube References

Intro to FCoE w/ EMC http://www.youtube.com/watch?v=EZWaOda8mVY – basic 101 of the technology

Deploying FCoE w/ EMC http://www.youtube.com/watch?v=y6IHMXEGRXs – the new equipment that you’ll need including cables, adapters and switches

FCoE Ecosystem http://www.youtube.com/watch?v=ExxilbDvZ1g

FCoE discussions w/ Kash Sheik of Cisco on the Storage (http://www.youtube.com/watch?v=GTz9S0cSdNo) and Network (http://www.youtube.com/watch?v=r1mI-eB8-iE) implications of fcoe

Presentations
EMC World 2009 presentation: Fibre Channel over Ethernet (FCoE), iSCSI and the Converged Data Center – http://www.slideshare.net/stuminiman/fibre-channel-over-ethernet-fcoe-iscsi-and-the-converged-data-center
Innovation Network Lecture – Journey Towards the Converged Data Center (archived webinar) https://community.emc.com/docs/DOC-4398

Papers
Introduction to Fibre Channel over Ethernet (FCoE) http://www.emc.com/collateral/hardware/white-papers/h5916-intro-to-fcoe-wp.pdf (Updated Nov ‘09)
Fibre Channel over Ethernet Techbook [from EMC E-Lab] http://www.emc.com/collateral/hardware/technical-documentation/h6290-fibre-channel-over-ethernet-techbook.pdf

Types of Virtualization / Types of Hypervisors

This blog post will explain the types of server virtualization that is prevalent today and the various types of hypervisors ( the virtualization layer) in scope today.

Types of Virtualization:
1) FULL VIRTUALIZATION
This is also known as native virtualization. This method consists of a virtualization layer or a hypervisor that intermediates between the guest operating system and the underlying hardware.
For Example: VMWARE ESX SERVER .
This runs on top of a native hardware and intermedites the guest OS and the hardware. The hypervisor monitors the instructions executed by the guest OS and traps certain privileged instructions that directly meddle with the underlying hardware since it is shared across the guest OSes. The guest OS can run unmodified in the case of Full / Native Virtualization.

2) PARA VIRTUALIZATION
This type of virtualization still consists of a virtualization layer called the hypervisor to co-ordinate access to the shared hardware by multiple guest Operating Systems, but the guest OSes are themselved tuned to be virtualization aware so that there is no need of any trapping and hence close to performance of a OS running on top of a real hardware.
Ex: XenServer
The guest OS need to be modified for the hypervisor on which it will run.

Type of Hypervisors:
"Hypervisor" - The "visor" who manages multiple "Super" visors. In early days , each Operating system itself was termed as the "Super"visor since it manages the hardware and software resources. Now, since the hypervisor abstracts or manages the resources across multiple guest Oses or Supervisors , it is called as "Hypervisor".

1) Type 1 Hypervisor: Hypervisors that run directly on top of physical hardware are called as Type 1 or bare-metal hypervisors.
Ex: ESX Server , Citrix XenServer, Oracle VM, Microsoft Hyper-V
The guest OSes run at the second-level on top of the Type 1 hypervisors.
Hardware -> Type 1 Hypervisor -> Guest OS

2) Type 2 Hypervisor: Hypervisors that run within an Operating System Environment are called as Type 2 Hypersiors or Hosted Hypervisors.
Ex: Vmware Workstation, Microsoft Virtual PC
The guest Oses run at the third-level on top of the Type 2 hypervisors.
Hardware -> Base-OS -> Type-2 Hypervisor -> Guest OS.

Fabric Discovery and Initialization

One of the most frequent questions that people ask is how does the FibreChannel initiators intialize and discover the targets and what happens before the initiator and the target start
communicating. Let me try to explain the same in a sequence of steps.

When a Node initializes in a Fibre Channel environment, the potential actions include the following
1. Link Speed Negotiation
2. Link Initialization and Determination of the port's operation mode(N_Port)
3. Fabric Login (FLOGI) with the Fabric Login Server ( 0xFF FF FE ). FLOGI is an ELS (Extended Link Service provided by FC)
4. Port Login (PLOGI) with the Name Server or Directory Server ( 0xFF FF FC) .This is infact the mechanism of the N_Port to register information with the Name Server.
5. Registering to receive State Change Notifications (SCR). This is typically done with the Fabric Controller ( 0X FF FF FD)
6. Retrieving Information from the Name Server. This is done using Name Server queries. This is the mechanism for address discovery of SCSI Storage Devices
(Ex: Some of the commonly used queries to get objects from the directory server are GA_NXT - Get al Next , GID_FT - Get FCID)
7. Once the address are retrieved, the initiator port does a logging in (PLOGI) with other node ports. Typically the SCSI-FCP initiators initiate the login and the targets wait for the login to occur.
8. The initiators then perform a Process Login (PRLI) to negotiate other information before they start sending commands to the target.
9. The initiator may then perform a TUR (Test Unit Ready) to check the readiness of the target unit. It will then do a REPORT LUNS to discover the LUNS supported by the target and then a SCSI Inquiry to obtain information about each LUN.

This sums up the typical initialization sequence and the stuff that happens.......

Hook up an analyzer between a Server and a Fabric Switch and capture the trace....

If you see something different from the sequence described above...let me know..:)-

ALUA

ALUA stands for Asymmetric Logical Unit Access. ALUA is also called as SCSI Target Port groups, TPGS(Target Port Group Support).

The ALUA pulled itself into effect into the SCSI standards to standardize the ways host multipathing software can know about the attributes of paths available to an asymmetric storage system having multi-port access.

Every asymmetric array vendor ( asymmetric arrays - multiported arrays which have differing performance characteristics when accessed through different ports) had a unique/ vendor specific method of identifying the characteristics of the path. They used a lot of vendor specific mechanisms to provide access to ( LUs - logical units) in case of path failures to trespass LUs.

This caused a lot of interoperability issues for the host multipathing vendors, they need to have a vendor specific mechanism for each and every asymmetric model out there and need to take of them in each of their support matrix. ALUA defines a set of SCSI command through which hosts can determine the path attributes by querying the target.

Say, for example if an Array 1 (Vendor X) has two paths P1 and S1 and Array 2 (Vendor Y) has two paths P1" and S1". Rather using 2 different vendor command sets to detect the primary and secondary path, the standard SCSI commands can be used to determine the same. This made the job of the multipathing software to handle things in a very clean fashion and handle the right multipathing failover polices/ load balancing policies in a universal fashion.

Access characteristics of a path to a logical unit can be determined using the Report Target Port Group (RTPG)scsi command. Access characteristics can be set using Set Target Port Group (STPG).

NPIV

NPIV stands for N-Port Virtualization. This is one of the port virtualization techniques that has come of late....mostly used by the host storage vendors ( adding to the confusion of the virtualization terminology...the most widely term in the industry..)

The concept fundamentally is to create a set of virtual N_Ports (Node Ports) associated with a physical adapter so that the link bandwidth can be shared across the many virtual N_ports and utilized to the full extent. This has particularly taken shape with the advent of the Virtualizedserver environments where there are many virtual machines running on a physical server sharing the physical resources.

The main reasoning behind the concept is to provide flexibility to the Guest OS'es in terms of access to the SAN resources. By providing many virtual N_ports with unique WWPN's from a single phyiscal adapter, the guest OS'es can be assigned to unique zones these and can be isolated from each other in terms of access to the SAN storage resources.

The main crux of providing the functionality lies with the host storage stack and with the switch vendors. Some array vendors do support the functionality thought it not absolutely needed except when you want to reduce your target port count.

An interesting link discussing the concept that i found useful can be found here.
http://www.soltechnology.com/articles/FC-Port-Channel-Storage-Virtualization.htm

Hope this helps in understanding the technology.......