FC HBA Speed

Fibre Channel 1GFC, 2GFC, 4GFC use an 8B/10B encoding scheme. Fibre Channel 10GFC, which uses a 64B/66B encoding scheme, is not compatible with 1GFC, 2GFC and 4GFC, and is used only to interconnect switches.

1, 2, 4, and 8 Gb Fibre Channel all use 8b/10b encoding.   Meaning, 8 bits of data gets encoded into 10 bits of transmitted information – the two bits are used for data integrity.   Well, if the link is 8Gb, how much do we actually get to use for data – given that 2 out of every 10 bits aren’t “user” data?   FC link speeds are somewhat of an anomaly, given that they’re actually faster than the stated link speed would suggest.   Original 1Gb FC is actually 1.0625Gb/s, and each generation has kept this standard and multiplied it.  8Gb FC would be 8×1.0625, or actual bandwidth of 8.5Gb/s.   8.5*.80 = 6.8.   6.8Gb of usable bandwidth on an 8Gb FC link.

10GE (and 10G FC, for that matter) uses 64b/66b encoding.   For every 64 bits of data, only 2 bits are used for integrity checks.   While theoretically this lowers the overall protection of the data, and increases the amount of data discarded in case of failure, that actual number of data units that are discarded due to failing serialization/deserialization is minuscule.    For a 10Gb link using 64b/66b encoding, that leaves 96.96% of the bandwidth for user data, or 9.7Gb/s.

So 8Gb FC = 6.8Gb usable, while 10Gb Ethernet = 9.7Gb usable.   Even if I was able to use all of the bandwidth available on an 8Gb FC port (which is very unlikely at the server access layer), with 10GE running FCoE, I’d still have room for 3 gigabit Ethernet-class “lanes”.   How’s that for consolidation?

10Gb FC has the same usable bandwidth, and without the overhead (albeit a small 2% or so) of FCoE, but you don’t get the consolidation benefits of using the same physical link for your storage and traditional Ethernet traffic.

http://www.unifiedcomputingblog.com/2010/04/08/8gb-fibre-channel-or-10gb-ethernet-w-fcoe/

FC Sites

http://www.cisco.com/en/US/docs/switches/datacenter/nexus5000/sw/configuration/guide/fm/tsfab.html#wp998439

http://www.cisco.com/en/US/docs/switches/datacenter/nexus5000/sw/troubleshooting/guide/n5K_ts_sans.html       // Troubleshooting SAN Switching Issues

FC Topology

http://www.docstoc.com/docs/104758623/Fibre-Channel-with-SCSI

FC - Basics

FC Basics -- All in a single image.

20 - this is a Node (WWNN)

     21 - this is the WWPN of Port 1

     22 - this is the WWPN of Port 2

FC_ID: 24 Bit Address Space

• N_Port ID is assigned by the fabric when a world wide name registers with it.

• For physical port, N_Port sends FLOGI (with WWPN of physical port) to address 0xFFFFFE to obtain a valid address (N_Port ID).

• For virtual port, N_Port sends FDISC (with WWPN of virtual  port) to address 0xFFFFFE to obtain an additional address (Virtual N_Port ID)

===============================================================

Fabric: term to describe a generic switching environment, it can consist of one or more interconnected switches (domains).

One Fibre Channel Switch = One Fabric Domain.
Maximum Domains = 239 in a single Fabric based on 24-bit addressing.
Principal Switch: Determine precedence between FC Swithced without adding a separate external. Fabric management software. it facilitates the initialization of the Fabric, and acts as controller of domains.

Switch Ports:
E_port – expansion port, connects 2 FC switches together to make a Fabric.
F_port – Fabric port where N_port attaches.
FL_port – Fabric loop port where NL_Port attaches.

Device Ports:
N_port – port designator for direct fabric attached devices.
NL_port – device that is attached to the loop (seldom used, support for old FC stuff).

FC_ID: 24 Bit Address Space: Dynamically assigned during login.
| Domain ID | Area ID | Node ID |
Domain ID = upper 8 bits, identifies the FC Switch
Area ID = middle 8 bits, identifies port number
Node ID = lower 8 bits, N_Port address

Unique Identification:
-Fixed 64bit value assigned by manufacturer.
-Used in mapping to upper layer protocols.
WWPN: uniquely identify a port (Port_Name).
WWNN: uniquely identify a node (Node_Name).

FC defines two types of login procedures:
Fabric Login (flogi) – After link established, N_Port sends flogi frames to the fabric and receive responses back.

N_Port Login (plogi) – Enable a N_port to communicate with another N_port. Once logged in, it will stay indefinitely.

FC transmits data in frames, defines variable length:
36 bytes of overhead
SOF – 4 bytes
Header – 24 bytes
CRC – 4 bytes
EOF – 4 bytes
Up to 2112 bytes of payload
Optional Header – up to 64 bytes
Data – up to 2048 bytes

FC Layers:
FC-4: Upper layer protocola interface – SCSI, IP, and so on roughly as transport layer in the OSI model.
FC-3: common Services
FC-2: Data delivery – Framing Protocol and Flow Control. Similer to layer 2 of OSI.
FC-1: Encoding – 8b/10b (1,2,4,9 Gb/s) and 64b/66b (10 Gb/s). Link Controls
FC-0: Physical Interface – Media Connectors and Cables.

FC starndards site: http://www.t11.org
FC-FS: Fibre Channel Framing and Signaling Interface
FC-LS: Fibre Channel Link Service
FC-GS: Fibre Channel General Services
FCP: Fibre Channel Protocol for SCSI

===============================================================

• Zones are partitions, similar to VLANs. Ports or WWNs can live in multiple zones at the same time.
• Soft zones enforce partitioning based on WWN, and they're difficult to manage if fiber moves to a new port.
• Hard zones are port-based: you can keep track of what node is attached where. Hybrid zones exist on some switches, for the paranoid: "WWN C must live on this port."

Soft Zones

Soft zoning means that the switch will place WWNs of devices in a zone, and it doesn't matter what port they're connected to. If WWN Q, for example, lives in the same soft zone as WWN Z, they will be able to talk to each other. Likewise, if Z and A are in a separate zone, they can see each other, but A cannot see Q. This is the complexity part; a feature that isn't widespread in Ethernet switches.

The benefit to using soft zones is that you can connect to any port on a switch, and know that you'll have access to the other nodes you're supposed to see.


Hard Zones

Hard zones are more like VLANs in the Ethernet world. You place the port into a zone, and anything connecting to that port is in the zone, or zones, which are configured for that port. Sure, it is less secure in the event of a physical attack where someone is able to move fiber connections.

FC Layers

Class of Service

The methodology with which the communication circuit is allocated and retained between the communicating N_Ports and by the level of delivery integrity required for an application.
Class of service (established during login)

Class 1: Dedicated Connection
               Full bandwidth allocated, in-order delivery guaranteed
Class 2: Multiplex 
               More like a LAN: connectionless transmission, dropped frames okay but get a notification.
Class 3: Datagram
                Similar to class 2 but no notification
Class 4: Fractional
               Establishes a bandwidth-limited Virtual Circuit (VC) path across the fabric.
Class 6: Multicast
               Multicast capabilities

Note : Classes 1, 2, and 3 may be supported with any of the three topologies. Classes 4 and 6 are only supported with the Fabric topology.

Class 3 : Datagram

• Provides connection-less service without any notification of non-delivery (BSY or RJT), delivery (ACK), or end-to-end flow control between two communicating Nx_Ports.
• Allows one Nx_Port to transmit consecutive frames to multiple destinations without establishing a dedicated connection with any specific Nx_Port.
• The Fabric, if present, and the destination Nx_Port are allowed to discard Class 3 frames without any notification to the transmitting Nx_Port.
• Allows one Nx_Port to receive consecutive frames from one or more Nx_Ports without having established dedicated connections.
• Class 3 service is requested by an Nx_Port on a frame by frame basis.
• The Fabric, if present, routes the frame to the destination Nx_Port.
• Nx_Port supporting Class 3 service is required to have logged in with the Fabric or the Nx_Ports.
• F_Port exercises buffer-to buffer flow control with the Nx_Port. 
• R_RDY is used for buffer-to-buffer flow control.
• If the Fabric is unable to deliver the frame to the destination Nx_Port, the frame is discarded and the source is not notified. 
• If the destination Nx_Port is unable to receive the frame, the frame is discarded and the source is not notified.
• End-to-end Credit is not used.

Flow Control

Flow control deals with the problem where a device receives frames faster than it can process them. The result is that the device is forced to drop some of the frames. 

Fibre Channel uses two types of flow control.

• Buffer to Buffer and 
• End to End

BB

Deals only with the link between an N_Port and an F_Port or between two N_Ports.

Ports on the link exchange values of how many frames can be received at a time from the other port. 

EE

End-to-End flow control is not concerned with individual links instead it deals with the source and destination N_Ports.

Two N_Ports log into each other, they report how many receive buffers are available for the other port.