Subject: FCIP: review comments-1

From: "Mallikarjun C." <[email-masked]>
Date: Thu, 14 Mar 2002 19:05:37 -0800

Some comments on rev9, excuse me if some/all these were earlier
discussed by the design team.
--
Mallikarjun

Mallikarjun Chadalapaka
Networked Storage Architecture
Network Storage Solutions Organization
Hewlett-Packard MS 5668 
Roseville CA 95747
[email-masked]

> 2. Purpose, Motivation and Objectives
.....
>    Network. Since Fibre Channel and IP Networking technologies are
>    compatible, 

I am not sure what's implied by this sentence....

Generally, I would think that the motivation to extend an FC SAN using 
IP networks is based on the ubiquity of the IP networks.

>    The fundamental assumption made in this specification is that the
>    Fibre Channel traffic is carried over the IP Network in such a
>    manner that the Fibre Channel Fabric and all Fibre Channel devices
>    on the Fabric are unaware of the presence of the IP Network. 

Can someone please comment on the protocol interactions that result in 
the failure to set up an FCIP Link if this latency expectation is not met?

>    2)  apply the mechanism described in 1) to an FC Fabric using an IP
>        network as an interconnect for two or more islands in an FC
>        Fabric.

S/b "an" w/ "the"; S/b "for" w/ "between"

> 3. Relationship to Fibre Channel Standards
.....
>    FC is standardized under American National Standard for Information
>    Systems of the National Committee for Information Technology
>    Standards (ANSI-NCITS) in its T11 technical committee.

I believe ANSI stands for Americal National Standards Institute.  Also, 
NCITS had been changed to INCITS last year.

 >    FC-BB and FC-BB-2 describe the relationship between an FC Fabric and
>    interconnect technologies not defined in by Fibre Channel standards
>    (e.g., ATM and SONET). FC-BB-2 is the natural Fibre Channel home for
>    describing relationships to TCP/IP and FCIP.

This is the first instance of FCIP's usage as a *protocol*.  It would be best 
preceded by a definition of what FCIP means as a protocol.  The previous text 
only describes the objectives of the document, but not the protocol.

> 3.2 This Specification and Fibre Channel Standards
.....
>    No attempt is being made to define a specific API between an FCIP
>    Entity and an FC Entity at this time because doing so risks
>    compromising the performance and efficacy of the resulting products.

I am somewhat concerned that the names are implying an incorrect distribution 
of functionality between "FC Entity" and "FCIP Entity"....  

From the name, I had assumed that FC Entity is simply a standard FC fabric 
element with no FCIP-isms.  But further reading of the document made me realize 
that it in fact knows about the TCP connections and even actively participates in 
QoS and authentication decisions.

As a first time reader, it appears to me that retaining only the FC E_Port functionality 
perhaps additionally providing timestamp and flow control services to the FCIP Entity
(and dropping everything else into the FCIP Entity) may be a lot cleaner distribution 
of functionality.  At any rate, I would like to understand the current distribution rationale.

BTW, can someone please clarify if the expected "role" of the FC Entity on
the FC side is indeed an E_Port?  It appears so from the requirement that FCIP
be totally transparent to the FC fabric, but I don't see it called out in Annex G.

>....fully functional and compliant
>    products can be built provided they contain both an FCIP Entity and
>    an FC Entity. The only products that cannot be built are those that
>    contain only one or the other.

The last sentence seems to stress the obvious at first glance, but I would 
think that products with just the FC Entity should be able to be built (not
that one would...) to act as regular fabric elements with no FCIP features?

Also, I take it that products with two FC Entities and one FCIP Entity 
for ex. are disallowed - but the last sentence seems to imply otherwise.

> 4. Terminology
....

>    FCIP Entity - The principal FCIP interface point to the IP Network
>    (see section 6.4).

This doesn't sound right to me....this definition is more appropriate for FCIP_LEP.
This can perhaps be described as "the entity responsible for the FCIP protocol
exchanges on the IP Network and which encompasses FCIP_LEP(s) and FCIP Control 
& Services module".

> 5. Protocol Summary
...
>    2)  Viewed from the IP Network perspective, FCIP Entities are peers
>        and communicate using TCP/IP. Each FCIP Entity is a TCP endpoint
>        in the IP-based network.

The second sentence seems a little context-sensitive, since each FCIP Entity
can be the TCP endpoint for several TCP connections. 

> 
>    3)  Viewed from the FC Fabric perspective, pairs of FCIP Entities,
>        in combination with their associated FC Entities, serve as an FC
>        Frame transmission component of the FC Fabric. The FC End Nodes
>        are unaware of the existence of the FCIP Link.

Can a "FC Frame transmission component" be something other than an E_Port?

>    6)  An FCIP Entity MAY contain multiple FCIP Link Endpoints, 

I would add "each of which MAY service multiple TCP connections, " here...

>but
>        each FCIP Link Endpoint (FCIP_LEP) communicates with exactly one
>        other FCIP_LEP.
> 
>    7)  When multiple FCIP_LEPs with multiple FCIP_DEs are in use,
>        selection of which FCIP_DE to use for encapsulating and
>        transmitting a given FC Frame is outside the scope of this
>        document. FCIP Entities do not actively participate in FC Frame
>        routing.

Couple of comments on this bullet (7) - 

Let's consider the case of multiple FCIP_DEs in one FCIP_LEP.
This draft does not specify how each inbound FC frame from the 
FC fabric is distributed onto one of these FCIP_DEs (TCP connections).   
Where is it specified in wrt routing on TCP connections?  I take it that 
the regular FC fabric routing rules aren't quite applicable in this case.

To stress the obvious, I think we should have some standard covering
this case - else we will end up with frames destined to the same D_ID
being striped on multiple TCP connections, and thus ending up OOO.

>    13) On a given TCP Connection, this specification relies on TCP/IP
>        to deliver a byte stream in the same order that it was sent.

Perhaps we should add - ", but allows confirmation of the same for each 
frame by checking the FCIP and FC CRCs."....

> 6.3 FC Entity
....
>    the combination shown in figure 3. As another example, the
>    combination cannot be used to replace cable connections in a Fibre
>    Channel Arbitrated Loop because loop primitive signals cannot be
>    encapsulated for transmission over TCP.

I am not sure the last sentence is needed since figure 3 is explicit 
about the usage of fabrics.

> 6.4 FCIP Entity
.....

>    The FCIP Entity is the connection interface point for the IP Network

As commented earlier, the "connection interface point" doesn't sound totally
correct - I would suggest "interfacing element" instead.

>    and is the owner of the IP Address and Well Known Port used to form
>    TCP Connections. An FC Fabric to IP Network interface product SHALL
>    provide each FC Entity FCIP Entity pair contained in the product

May I suggest a new term "FC-FCIP Pair" in place of "FC Entity FCIP Entity pair ".
I think it improves the general readability.

>    FC Entity with an interface to key IP Network features. The
>    interfaces to the IP Network features is implementation specific,
>    however, to maintain interoperability, the notable TCP/IP mechanisms
>    used are specified in this document as follows:

I think the last sentence is incorrectly implying interoperability issues
around the (private) FC Entity-FCIP Entity interface.

I would suggest a rewrite for the last sentence as something like -

"The interfaces to the IP Network features are implementation-specific.
  To aid interoperability, this document specifies the notable TCP/IP Network 
  features that are typically used ."

> 6.5 FCIP Link Endpoint (FCIP_LEP)
....

>An FCIP_LEP
>    MAY communicate with its FC Entity counterpart to coordinate flow
>    control.

Suggest adding the phrase "across the domains".

> 6.6 FCIP Data Engine (FCIP_DE)
.....
>    7)  In the absence of errors, the de-encapsulated FC Frame and time
>        stamp SHALL be passed to the FC Transmitter Portal for delivery
>        to the FC Entity.

It is nice to add a sentence about the handling in the presence of errors.
At a minimum, this should provide a cross-reference to the error detection
section.

> 6.6.1 FCIP Encapsulation of FC Frames
....
>    The FCIP Special Frame SHALL only be sent as the first bytes
>    transmitted in each direction on a newly formed TCP Connection and
>    only one FCIP Special Frame SHALL be transmitted in each direction
>    at that time (see section 9.1). After that all FCIP Frames SHALL
>    have the SF bit set to 0.

This para seems slightly out of context here, and perhaps should be moved
to section 9.1.  This usage semantics should ideally be preceded by what *is* 
a Special Frame and its purpose - though I could gather that from the 
usage and format descriptions in the entire document, I don't really find a
place where its purpose is defined...

> 
>    Table 1 summarizes the usage of the pFlags SF and Ch bits.

- It may be useful to add a protocol-specific bit to distinguish originated vs
echoed SF, so the parsing and validation can be self-contained.

- Also, I think a sentence should be added which says that the -pFlags field 
SHALL contain the ones-complement of the pFlags field.

---

From: "Mallikarjun C." <[email-masked]>
Date: Fri, 15 Mar 2002 19:42:26 -0800

Some more comments on rev9, excuse me if some/all these were earlier
discussed by the design team.
--
Mallikarjun

Mallikarjun Chadalapaka

....
>    The CRCV (CRC Valid) Flag SHALL be set to 0.
> 
>    The CRC field SHALL be set to 0.

I am surprised that the FCIP encapsulated header is never protected
by a CRC.  The error detection section clearly acknowledges the
possibility that TCP checksum could be inadequate for certain 
deployments - and even suggests checking the FC frame CRC (sort
of like a data digest) on reception at the Encapsulated Frame Receiver 
Portal.

My recommendation is to require an FCIP Entity to employ the header
CRC if the SF that it receives asks for CRC - IOW, similar to iSCSI's 
approach.  So, I guess I am also advocating a new bit in the pFlags field 
to announce this.  When this CRC expectation is announced, the FC CRC
checking test in 6.6.2.2 should also be a mandatory test - from the 
"semi-optional" list it is currently in.

> 6.6.2 FCIP Data Engine Error Detection and Recover

The last word in the above should be "Recovery".
 
> 6.6.2.1 TCP Assistance With Error Detection and Recovery
.....
>Thus, the byte stream passed from TCP to
>    the FCIP_LEP will be in order and free of errors detectable by the
>    TCP checksum. If TCP did not perform these functions, the FCIP_LEP
>    would have to.

Suggest rewording the last sentence (TCP always performs these functions
to *its* satisfaction, the question is if FCIP feels the same; secondly, FCIP_LEP's
error mgmt behavior is not dynamically contingent upon TCP's behavior as this 
sentence implies) -
  "To address the possibility that TCP did not perform these functions adequately 
    in a given FCIP deployment context, the FCIP_LEP verifies if these expectations
    are met."

> 
> 6.6.2.2 Errors in FCIP Headers and Discarding FCIP Frames
.....

>    Further, some errors in the encapsulation will result in the FCIP_DE
>    losing synchronization with the FCIP frames in the byte stream

I don't see "frames" here with the uppercase "F" used everywhere else.

Also, one observation is that FCEncap document uses "frames" consistently 
throughout, whereas this document chooses to use uppercase "F" (mostly).

>    1)  Length field validation -- 15 < Length < 545;

I assume "Frame Length" is meant by "Length" above.

> 6.6.2.3 Synchronization Failures
.....
> 
>    If an FCIP_DE determines that it cannot find the next FCIP Frame
>    header in the byte stream entering through the Encapsulated Frame
>    Receiver Portal, the FCIP_DE SHALL either:

S/b "either" w/ "do one of the following".

>    b)  recover synchronization by searching the bytes delivered by the
>        Encapsulated Frame Receiver Portal for a valid FCIP Frame header
>        having the correct properties

Useful to refer to 6.6.2.2 here for "correct properties". 

> 7. Checking FC Frame Transit Times in the IP Network
....
>    requirement in the FC Entity is based on a desire to include the

S/b "in"  w/  "on"

>    Frame. If no synchronized time stamp value is available to accompany
>    the entering FC Frame a value of zero SHALL be supplied.

From this, it isn't clear to me if FCIP *requires* only Synchronized operation.
If so, the draft should also specify how implementations are expected to 
deal with "benign" transitions into and out of the Unsynchronized state (i.e.
transitions happening when no Encapsulated Frames are being received).

>    The FC Entity SHALL
>    verify that the FC Entity it is communicating with on an FCIP Link
>    is using the same synchronized time source as it is, either Fibre
>    Channel services or SNTP server.

I see a chicken-and-egg problem for some implementations:  Since Fibre 
Channel time services are not available until the FCIP Link is successfully 
set up and since timestamp is to be carried on every FC Frame (including 
those Fibre Channel time service exchanges) once the Link is set up, the only 
decent option for an implementation (assuming it suppots only Synchronized
operation) is to rely on SNTP server.  Is this correct?  

If Unsynchronized operation is intended to be disallowed (my earlier 
question), then it seems to me that SNTP is the only option.

> 8. The FCIP Special Frame
.....

>    only one FCIP Special Frame SHALL be transmitted in each direction.

A general comment on this wording (and others similar to this).   I would
suggest rewording (to be much stronger) to:

The FCIP Special Frame SHALL be the first application data exchanged on 
each newly formed TCP connection, and only one FCIP Special Frame SHALL 
be transmitted in each direction.

>    Note: The combination of the Source FC Entity World Wide Name and
>    Source FCIP Entity Identifier fields uniquely identifies every FC
>    Entity FCIP Entity pair in the IP Network.

- S/b "Source FCIP Entity Identifier" w/ "Source FC/FCIP Entity Identifier"

- Also I take it that the "FC/FCIP Entity Identifier" is unique only within the
  scope of the given FC Entity's WWN.  So, does the model allow multiple
  FCIP Entities to be associated with the FC Fabric Entity WWN?

- From this statement, it implies to me that the Destination FC/FCIP Entity
  Identifier must be present in the special frame to ensure that the TCP connection
  is indeed established to the right "FC/FCIP Entity" under the scope of the 
  given Destination FC Fabric Entity WWN.  What am I missing?

>    The Destination FC Fabric Entity World Wide Name field MAY contain

Why isn't the above requirement a SHALL?

> 
> 9.1.2.2 Dynamic Creation of New TCP Connections
...

>     - The expected Destination FC Fabric Entity World Wide Name of the
>       FC Entity FCIP Entity pair to which the TCP Connection is being
>       made
>     - TCP Connection Parameters (see section 9.3)
>     - Quality of Service Information (see section 11)
> 
>    Based on this information, the FCIP Entity SHALL generate a TCP
>    connect request [8] to the FCIP Well-Known Port of 3225 (or other
>    configuration specific port number) at the IP Address specified by
>    the service advertisement. If the TCP connect request is rejected,
>    act to limit unnecessary repetition of attempts to establish similar
>    connections. If the TCP connect request is accepted, the FCIP Entity
>    SHALL follow the steps described in section 9.1.2.3 to complete the
>    establishment of a new FCIP_DE.
> 
>    It is recommended that an FCIP Entity not initiate TCP connect
>    requests to another FCIP Entity if incoming TCP connect requests
>    from that FCIP Entity have already been accepted.

This entire text is duplicated from previous section 9.1.2.1.  Seems like 
we could do with one instance (possibly in a new subsection).

> 
> 9.1.2.3 Connection Setup After a Successful TCP Connect Request
>
......
>All fields in the FCIP Special
>    Frame SHALL be filled in as described in section 8, particularly:

While the sentence above is unequivocally clear, I don't understand the need
for all the text that follows "particularly"....  It is confusingly repetitive since as far 
as I can tell, all these are covered in more or less the same language in section 8.

>    After the FCIP Special Frame bytes are sent on the newly formed
>    connection, the FCIP Entity SHALL wait for the FCIP Special Frame to
>    be echoed as the first bytes received on the newly formed connection.

- S/b "bytes are" w/ "is"
- S/b "first bytes" w/ "first TCP segment data"
- I take it that the onus is on the FCIP Entity that did the TCP active open 
  to send the SF.  That leads me to: What if there's a TCP simultaneous open?
  Would not each end end up sending the SF and waiting for the echo?  Also,
  would not the earlier rule on only one frame being transferred in each
  direction be violated then?
 
> 
>    If the echoed FCIP Special Frame bytes do not exactly match the FCIP
>    Special Frame bytes sent (words 7 through 17 inclusive), the FCIP
>    Entity SHALL close the TCP Connection and notify the FC Entity with
>    the reason for the closure.

Seems like all the 11 words are required to be compared. If so, what is the
Ch bit being used for?  IOW, why SHALL it be set by the responder?

> 9.1.3 Processing Incoming TCP Connect Requests
....
> 
>    The FCIP Entity SHALL listen for new TCP Connection requests [8] on
>    the FCIP Well-Known Port (3225). An FCIP Entity MAY also accept and
>    establish TCP Connections to a TCP port number other than the FCIP
>    Well-Known Port, as configured by the network administrator.

It may be useful to add that this usage is outside the scope of this document.

> 
>    The FCIP Entity SHALL determine the following information about the
>    requested connection:
> 
>     - Whether the requested connection is allowed

I take it that by means not specified in this spec?  If so, it's useful to call 
it out.

>    abort the TCP connect request [8]. If the requested connection is

I was told that "abort" isn't available on all implementations - perhaps
"abort/close" would be better....

>    The FCIP Entity MAY apply a timeout of not less than 90 seconds to
>    the waiting for the FCIP Special Frame bytes and if the timeout
>    expires the FCIP Entity SHALL close the TCP Connection and notify
>    the FC Entity with the reason for the closure.

I am not clear why this notification is required, since the local FC Entity did
not motivate the TCP connection establishment.  If it is intended for logging,
perhaps notifying the Control & Services module would be more appropriate.

>    If the Connection Nonce field contains a value identical to the most
>    recently received Connection Nonce from the same IP Address, the
>    FCIP Entity SHALL close the TCP Connection and notify the FC Entity
>    with the reason for the closure.

Same comment.

>    1)  Leave the Destination FC Fabric Entity World Wide Name field and
>        Ch bit both 0;

So allow the FCIP Link to be established?  It is unclear to me how implementations
adopting this option would be able to prevent unintended FCIP Link formation.

>    2)  Change the Destination FC Fabric Entity World Wide Name field to
>        match FC Fabric Entity World Wide Name associated with the FCIP
>        Entity that received the TCP connect request and change the Ch
>        bit to 1; or

In effect, this is being indirectly allowed as a legal discovery process.  Is
there a DoS concern here?  Also, would revealing the FC WWN be acceptable
in confidentiality terms?

>    3)  Close the TCP Connection without sending any response.

I like this option best, :-)

> 10.2 FC Fabric and IP Network Deployment Models
.....
>        Entities in an equal manner. This varies from a true Client-

S/b "varies" w/ "differs".