Demystifying extended distance ficon, por Stephen R. Guendert

1. Demystifying Extended Distance FICON AKA Persistent IU Pacing Stephen R. Guendert, Ph.D. Brocade Communications Systems In 2008 IBM announced IU pacing enhancements…
of 10
All materials on our website are shared by users. If you have any questions about copyright issues, please report us to resolve them. We are always happy to assist you.
Related Documents
  • 1. Demystifying Extended Distance FICON AKA Persistent IU Pacing Stephen R. Guendert, Ph.D. Brocade Communications Systems In 2008 IBM announced IU pacing enhancements that allow the end user to deploy z/OS Global Mirror (zGM) over long distances without a significant impact to performance. This is more commonly known by the marketing term Extended Distance FICON. The more technically accurate term as defined in the FC-SB3/4 standards is persistent IU pacing. How this functionality works, and what it actually does for the end user has remained a mystery, thanks in large part to the marketing term used. This paper/presentation will demystify how the technology works, and how it can benefit the end user. .INTRODUCTIONAs part of the Feb. 26, 2008, IBM System z10 and DS8000 announcements, IBM announced Information Unit (IU)pacing enhancements that allow customers to deploy z/OS Global Mirror over long distances without a significantimpact on performance. This is more commonly known as Extended Distance FICON. The more technicallyaccurate term, as defined in the ANSI T11 FC-SB3/4 standards, is persistent IU pacing. The ANSI T11 FC-SB3standard was amended (FC-SB3/AM1) in January 2007 to incorporate the changes made with persistent IUpacing.Extended Distance FICON really is a marketing term created to add some “pizzazz” to the standards terminology.IBM coined the term in February 2008, but it really has nothing to do with increasing distance; rather, it’s aperformance enhancement for FICON data traffic traveling over a long distance. Enhanced Distance FICONwould be a more accurate marketing term. This paper will explain in detail exactly how it works.CHANNEL DISTANCE EXTENSION BACKGROUNDAt the time of its introduction in 1990, ESCON supported a maximum distance of 20 km. While it exploited therelatively new Fibre Channel technology, it employed circuit switching rather than the packet switching that istypical of today’s Fibre Channel implementations. This provided significantly improved physical connectivity, butretained the same one-at-a-time CCW challenge-response/permission seeking logical connectivity that had beenemployed by parallel channels. As a result, ESCON performance is significantly reduced at extended distancesby the multiple roundtrip delays required to fetch the channel programs, a performance deficiency commonlyreferred to as ESCON droop.By the late 1990s the shortcomings of ESCON were driving a new technical solution. FICON evolved in the late1990s to address the technical limitations of ESCON in bandwidth, channel/device addressing, and distance.Unlike parallel and ESCON channels, FICON channels rely on packet switching rather than circuit switching.FICON channels also rely on logical connectivity based on the notion of assumed completion rather than thepermission-seeking schema. These two changes allow a much higher percentage of the available bandwidth tobe employed for data transmission and for significantly better performance over extended distances.FICON PROTOCOL OVERVIEW AND PROTOCOL EFFICIENCYFICON is an Upper Level Protocol (ULP) of the Fibre Channel architecture/standard. FICON is also known by itsformal name of FC-SB2, which is how the formal Fibre Channel standards documentation refers to it. FICON andFC-SB2 are interchangeable terms. FC-SB2 defines both the physical and logical structure of the frames of dataused by FICON to transport data via channel programs. FICON offers far greater connectivity flexibility compared
  • 2. to ESCON. Unlike ESCON, FICON supports multiple concurrent I/Os, as well as full-duplex channel operations.ESCON processes one I/O at a time and is half-duplex. Since its introduction, FICON has evolved and newstandards have been introduced, including FC-SB3 for cascaded FICON and FC-SB4 for z High PerformanceFICON (zHPF). Table 1 compares ESCON and FICON key attributes.Numerous tests comparing FICON to ESCON have been done by IBM, other SAN equipment manufacturers andindependent consultants. The common theme among these tests is that they support the proposition that FICONis a much better performing protocol over a wide range of topologies when compared with traditional ESCONconfigurations. This performance advantage is very notable as the distance between the channel and control unitincreases.Figure 1 [BROC2010a] is a representation of what was referred to earlier as ESCON’s permission-seeking/challenge-response schema. ESCON channels only transmit a single CCW at a time to a storagesubsystem. Once the subsystem (such as a DASD array) has completed its work with the CCW, it notifies thechannel of this with a channel-end/device-end (CE/DE). The CE/DE presents the status of the CCW. If the statusof the CCW was normal, the channel then transmits the next CCW of the channel program. Assuming that we arediscussing DASD, this leads to approximately 75 percent of the reported connect time being representative oftime periods in which the channel is occupied, but data is not being transferred. This schema presents twodistance-related performance issues. First, each CCW experiences distance-related delays as it is transmitted.Second, the small size of the packet ESCON employs to transfer data, known as a Device Information Block(DIB) coupled with the very small data buffer employed by ESCON results in the phenomenon that is now knownas ESCON droop. ESCON droop is a substantial effective data rate drop experienced as the distance betweenthe host channel and the storage device increases. ESCON ESCON ESCON Channel CCW 1 CU Device CMD END CE/DE CCW 2 CMD END CE/DE CCW 3 CMD END CE/DE Figure 1. ESCON CCW Pattern FICON FICON FICON Channel CCW 1 CU Device CMD CCW 2 END CCW 3 CMD END CMD CE/DE END Figure 2. FICON CCW Pattern
  • 3. Capabilities ESCON ESCON to FICON Native FICON Native FICON bridge (FICON with FICON Express) Express 8 I/O operations at a Only one Any 8 32 open 64 open time exchanges exchanges Logically daisy- Any one I/O, Any 8 I/O Any number of Any number of chained control units take turns Concurrently I/Os I/Os to a single channel concurrently concurrently Utilization of a < 35% 50–70% 90% or more 90% or more channel on host Average I/Os per 2,000–2,500 2,500 6,000 20,000–52,000 channel (4k blocks) (CCW – zHPF) Unit addresses per 1,024 16,384+ 16,384+ 16,384+ channel Unit addresses per 1,024 4,096 16,384+ 16,384+ control unit Bandwidth Beyond 9 km Beyond 100km Beyond 100 km Beyond 100 km degradation Table 1. Channel protocol characteristicsFigure 2 [BROC2010a] represents the FICON CCW pattern, which is often referred to as assumed completion.The entire channel program is transmitted to the storage subsystem at the start of an I/O operation. If there aremore than 16 CCWs in the channel program, the remainder are transmitted in groups of 8 until the channelprogram is completed. The overwhelming majority of channel programs are 16 or fewer CCWs in length. After thestorage subsystem completes the entire channel program, it notifies the channel with a channel end/device end(CE/DE). The numerous turnarounds present with the ESCON protocol are eliminated. The FICON architecturehas been tested as supporting distances of up to 120 km without the problem of droop.Droop begins when the link distance reaches the point at which the time required for light to make one round tripon the link is equal to the time it takes to transmit a number of bytes that will fill the receiver’s buffer. The mainfactors are the speed of light, the physical link, the protocol itself, the physical link distance between the channeland control unit, the link data rate and the buffer capacity. With ESCON, droop begins at about 9 km; whereasFICON does not experience droop until at least 100 km.In a configuration where a channel is extended beyond traditional distances, the additional delay of the extensionsolution can add significantly to the droop effect of an ESCON or FICON channel. Generally, at 20 km ESCONdelivers less than half the maximum effective bandwidth that would be delivered at zero distance. FICON is notaffected until after 120 km.How do you overcome the effects of channel droop, maintain native (or near native) performance, and stillguarantee the integrity of the channel program and corresponding data and information? Until the introduction ofpersistent IU pacing/Extended Distance FICON, the solution was to implement a combination of frame-shuttle,channel, control unit, and device emulation technologies on channel extension hardware, either stand alonespecialized hardware, or a blade in a FICON director.
  • 4. SEQUENCES, EXCHANGES, AND INFORMATION UNITS (IUs)In early discussions about the design requirements for the extension of FICON channels, there was concernexpressed over the effect of a FICON flow control method called IU pacing and the effect that it might introducefor very long-distance WAN extension of channels. The FICON Upper Layer Protocol (ULP) employs threeresources: SB-3/SB-4 sequences, Open Exchanges (OEs), and IUs to manage the underlying physical credits(2K data buffers) used to transmit the actual data. A credit is a 2K physical data transmission buffer. They aremore commonly known as buffer-to-buffer credits, which are used for flow control between two optically adjacentFibre Channel ports. [ARTI2006]. Buffer credits are a crucial and often overlooked component in configurations in which frames will travel longgeographic distances between optically adjacent ports. System z FICON channel cards and storage host adapterports have a fixed number of buffer credits per port. Switching devices (including channel extension) typically canhave their buffer credit settings customized. This is important, as the long distances are typically between theswitching devices. Having the correct/optimal number of buffer credits ensures frames will “stream” over thedistance and the link will be fully utilized. An insufficient number will result in dormant periods on the link-stoppedtraffic. The optimal number of buffer credits depends on three variables: link speed, frame size, and distance[GUEN2007].When receiving frames from the link, the FC-PH standard provides a mechanism that assembles sub-blocks ofdata contained in the payloads of one or more frames into a single data block. This single data block is known asa sequence. Each FC-4 ULP defines the contents of the sequences used for ULP functions. When this content(and use of the sequence) is defined in this manner, it’s called an IU, which is a logical resource comprised of oneto four credits. FICON and FICON Express/Express 2/Express 4/Express 8 channels currently support amaximum of 16 concurrent IUs for each OE. SB-3 IUs contain commands, device-level controls, link controls, andother related functions. Common examples of SB-3 IUs are command IUs, solicited data IUs, unsolicited data IUs,solicited control IUs, and unsolicited control IUs.An Open Exchange (OE) is a logical resource representing an I/O being processed by a channel. FICON andFICON Express/Express 2 channels on zSeries servers can support 32 concurrent OEs. FICON Express 2/4/8channels on System z9 and later servers can support 64 concurrent OEs. IUs sent by a FICON channel duringthe execution of an SB-3 link-control function or device-level function are restricted to one exchange, known as anoutbound exchange. The IUs a FICON channel receives during an I/O operation are restricted to a differentexchange, known as an inbound exchange. Figure 3 shows the relationship between frames, sequences, andexchanges [KEMB2002].An inbound exchange and outbound exchange transfer IUs in a single direction. An existing pair occurs whenboth an inbound and outbound exchange simultaneously exist between a channel and control unit for execution ofthe same device-level or link-level function. The control unit is then said to be connected to the channel. Achannel program, executed in a single connection, uses only one exchange pair. If the connection is removed byclosing the exchanges during the channel program, a new exchange pair is required to complete the channelprogram. To avoid errors that can occur from lost IUs or due to the inability to send an IU, sufficient resourcesneed to be held in reserve. This reserve is necessary to support new exchange pairs for unexpected events. If anIU initiating a new exchange pair is received, sufficient resources must be available to properly receive the IU andto initiate a new exchange in response. Figure 3. Frame, sequence and exchange relationship
  • 5. IU PACING AND PERSISTENT IU PACINGIU pacing is an FC-SB-3 level 4 function that limits the number of Channel Command Words (CCWs), and,therefore, the number of IUs, that can either transmit (write) or solicit (read) without the need for additionalcontrol-unit-generated acknowledgements (called command responses). FC-SB-3 Rev 1.6 defines an IU pacingprotocol that controls the number of IUs that can be in flight from a channel to a control unit. The control unit mayincrease the pacing count (the number of IUs allowed to be in flight from channel to control unit) in the firstcommand response IU sent to the channel. The increased pacing count is valid only for the remainder of thecurrent outbound exchange. In certain applications, at higher link speeds and at long distances, a performancebenefit is obtained by the increase in the allowed pacing count.The IU pacing protocol has a limitation that the first burst of IUs from the channel to the control unit can be nolarger than a default value of 16. This causes a delay in the execution of channel programs with more than 16commands at long distances because a round trip to the control unit is required before the remainder of the IUscan be sent by the channel, upon the receipt of the first command response, as allowed by the increased pacingcount. Since flow control is adequately addressed by the FC-PH level buffer-to-buffer crediting function, IU pacingisn’t a flow control mechanism. Instead, IU pacing is a mechanism intended to prevent I/O operations that mightintroduce large data transfers from monopolizing access to Fibre Channel facilities by other concurrent I/Ooperations.In essence, IU pacing provides a load-sharing or fair-access mechanism for multiple, competing channelprograms. While this facility yields desirable results, ensuring more predicable I/O response times on heavilyloaded channels, it produces less optimal results for long-distance deployments. In these cases, increased linklatencies can introduce dormant periods on the channel and its Wide Area Network (WAN) link. Dormant periodsoccur when delays waiting for anticipated command responses increase to the point where the pacing windowprohibits the timely execution of CCWs that might otherwise be executed to ensure optimal performance.The nominal IU pacing window for 1, 2, 4, and 8Gbit/sec FICON implementations permits no more than 16 IUs toremain uncredited. Pacing credits can be adjusted dynamically from these values by control unit requests forspecific protocol sequences; however, the channel isn’t bound to honor control unit requests for larger IU pacingwindows.Persistent IU pacing is a method for allowing FICON channels to retain a pacing count that may be used at thestart of execution of a channel program. This may improve the performance of long I/O programs at higher linkspeeds and long distances by allowing the channel to send more IUs to the control unit, thereby eliminating thedelay of waiting for the first command response. The channel retains the pacing count value, presented by thecontrol unit in accordance with the standard, and uses that pacing count value as its new default pacing count forany new channel programs issued on the same logical path.Persistent IU pacing is described by modifications to the following functions specified in FC-SB-3. The changesrelevant to persistent IU pacing are summarized below:• Establish Logical Path, Logical Path Established, and Status-Parameter Field: The Establish Logical Path (ELP) function was modified with definitions for optional features for bits in the link-control information field. In particular, bit one(1) was modified for persistent IU pacing. When bit one of the link-control information field is set to zero in the ELP IU, persistent IU pacing is not supported. When bit one of the link control information field is set to one in the ELP IU, persistent IU pacing is supported. This change is also reflected in the Logical Path Established (LPE) function. Bytes 2 and 3 of word 0 of the status header (the status-parameter field) was also modified for persistent IU pacing. This is a 16-bit field that may contain a residual count or IU pacing parameter. If the conditions are such that neither the residual count nor an IU pacing parameter is to be presented, the control unit shall set the status-parameter field to zero, and the channel receiving the status DIB shall ignore the status-parameter field. The modifications made deal with the residual count , and the IU pacing parameter.• Residual Count: The residual count is a 16-bit unsigned binary number that represents the difference between the CCW count for a command and the quantity of data actually transferred either from or to
  • 6. the device for that command. For write commands, the residual count shall be equal to the difference between the CCW count of the current CCW for the command and the number of bytes actually written to the device. For read commands, the residual count shall be the difference between the CCW count for the current CCW for the command and the number of bytes actually read from the device and transferred to the channel. The residual count is meaningful only when the residual-count- valid (RV) bit is one.• Persistent IU Pacing Parameter Valid: When persistent IU pacing is enabled and the residual-count- valid (RV) bit is set to zero, the persistent IU pacing parameter valid bit is provided in the status parameter field bit 0. When set to one, this bit indicates that the IU pacing parameter is valid. When set to zero, this bit indicates that the IU pacing parameter is not valid, is ignored by the channel and shall not change the persistent pacing credit at the channel for the logical path. This bit is only meaningful when persistent pacing is enabled.• IU Pacing Parameter: The IU pacing parameter is an eight bit value that is carried in the least- significant byte of the status-parameter field. The use of the IU pacing parameter in the status parameter field depends on whether or not persistent IU pacing is enabled. When persistent IU pacing is not enabled, the IU pacing parameter is provided in the status-parameter field when status is presented for the first command of a channel program that is executed as an immediate operation or when presenting device end in order to reconnect when the chaining condition is set. When persistent IU pacing is enabled, the residual-count-valid (RV) bit is set to zero and the persistent IU pacing parameter valid bit is set to one, the IU pacing parameter shall be provided in the status parameter field. The IU pacing parameter shall be sent by the control unit to indicate to the channel the maximum number of IUs a channel may send on a given outbound exchange until it receives a command response IU that was sent because the CRR bit was set to one, on the existing inbound exchange. An IU pacing parameter of zero indicates that the control unit wishes to leave the default value of the IU pacing credit unchanged.FUNCTIONALITY AND OPERATION OF IU AND PERSISTENT IU PACINGAt the time of the February 2008 IBM announcement, the Extended Distance FICON capability was available onlyon the System z10 coupled with the DS8000 running Licensed Machine Code (LMC) level 5.3.1 or later.. Today,it’s supported on the System z196 and the System z10 processors running Driver 73 with MCL F85898.003, orDri
  • We Need Your Support
    Thank you for visiting our website and your interest in our free products and services. We are nonprofit website to share and download documents. To the running of this website, we need your help to support us.

    Thanks to everyone for your continued support.

    No, Thanks