Subject: [ipfix] [issue] wording of transport protocol support

From: "MEYER,JEFFREY D (HP-Cupertino,ex1)" <[email-masked]>
Date: Thu, 20 Nov 2003 11:42:54 -0800

OK,

  Probably no point in arguing over what "simple" is, if there is some rough
consensus on wording:

     UDP MAY be used in deployments where exporters and collectors can
communicate over dedicated links which are not susceptible to congestion
issues.
     UDP SHOULD NOT be used in deployments where exporters and collectors
are communicating over links which are susceptible to congestion issues.
     SCTP SHOULD be used in deployments where exporters and collectors are
communicating over links which are susceptible to congestion issues.
     TCP MAY be used in deployments where exporters and collectors
communicate over links which are suscepible to congestion issues, but SCTP
is preferred, due to its ability to limit back pressure on exporters
(especially SCTP-PR) and its message vs. stream orientation.

  Comments on this specific point?  As Randy points out it may have IESG
issues, but no point in asking if we still don't agree on wording.

  Original e-mail:  http://ipfix.doit.wisc.edu/archive/2174.html

Regards,

  Jeff Meyer

-----Original Message-----
From: Randall Stewart (cisco) [mailto:[email-masked]]
Sent: Thursday, November 20, 2003 4:01 AM
To: MEYER,JEFFREY D (HP-Cupertino,ex1)
Cc: ipfix wg
Subject: Re: [ipfix] Forming consensus on IPFIX default protocol

MEYER,JEFFREY D (HP-Cupertino,ex1) wrote:

>Alex,
>
>  SCTP SHOULD be a kernel space protocol.  Obviously since you can bind to
>raw IP sockets you COULD and there is code out there to put it in user
>space, with all the associated penalties.
>
>  The only user space implementation I could actually find working links to
>is at:  http://www.sctp.de/sctp-download.html
>
>  Note that sigtran.org, links to sctp.org claiming a user space
"reference"
>implementation there.  Could only find kernel implementations on the
>download page :-(
>  
>
Jeff:

Yeah, thats true... its there but you have to know the name of the file :-0

The reason mainly because:

a) the user space implementation had fallen out of current standards (its
    not up to the Implementors-Guide v 10 like the kernel one).

b) It needed signifigant work to bring in a couple of the extension

c) We thought our efforts best focused on a good solid kernel 
implementation.

Now you can get the 4.5 release of the user space version on a cd when you
buy the book on SCTP.. but the copy in the book does not properly support
PR-SCTP or the other extension ADD-IP.

As to your other post.. if you can get that by the IESG I am fine with it.

>  Buried in SCTP.de's documentation is the following statement:
>
>    ... Currently this is implemented via function callbacks within 
>    one program (i.e. the ULP has functions registered at the very \
>    beginning of the program, that are to be called if some notification 
>    is due).  Right now, there is only one ULP instance. In the 
>    future the communication between SCTP and the ULP shall be done
>    with a locally bound UDP socket, that will be used to register a ULP
>    instance with the SCTP instance.
>
>     Thus it will be possible to asynchronously call functions of the 
>    SCTP protocol engine from a process that sends data on a local 
>    UDP socket, and is passed the results and other notification 
>    messages back over that socket.
>
>  So, at least this implementation (the only user space I could find),
>says they only support one process.  They say they might support 
>multiple processes in the future, but by having a single SCTP instance 
>(i.e. ONE user space process) turn around and dispatch local UDP 
>(i.e. more data copies) to additional processes.
>
>  
>

If you do go get the book (SCTP a reference guide) you would find a base 
SCTP
implementation that supports MULTIPLE processes. It does this by trading off
having only one process (a daemon) handle the I/O to the kernel. The 
consequence of
this is that data goes in and out of the kernel twice... It does work.. 
but there is a cost...

>  This seems like a disadvantage to me which is addressed by having
>this layer 4 protocol in the kernel, so that dispatching can be
>done directly to processes based on SCTP bound port, as opposed
>to all IP traffic bound to the SCTP IP protocol ID going up one 
>raw socket to one consumer.  Thus saving an unnecessary data copy
>for each transmission and receipt.
>  
>

exactly.. its a trade off.. but if you bind multiple raw sockets its 
even worse since
then every listener sees every SCTP packet... thats bad too :-0

>  Maybe I'm missing something, but your arguments, that it "just
>isn't so" ring rather hollow when I try to find concrete vs. anectdotal
>evidence.  Give me a URL and some actual description of how these 
>user space issues are avoided and maybe I can be convinced....
>If your answer is "we've addressed it" but the source is not 
>available, then I rest my case.
>
>
>  So, can we move past the bogus arguments that SCTP is somehow "just as
>easy" to use today as TCP, then maybe we can focus on how to word the
>protocol spec so that SCTP is encouraged, but TCP and UDP are allowed.
>
>  
>

If you have a kernel space implementation (which HP/Sun/Linux and BSD 
do) and it fully
supports the socket api (which HP's is soon to  I understand) then you can
easily move things from TCP or UDP to SCTP. Examples of this are:

1) I moved a 1.1 version of mozilla to use only SCTP with changing two lines
    of code. Considering that Mozilla's compressed source image is about 
50+ Megabytes
    not to bad.

2) I moved a version of apache 2.x to use BOTH SCTP and TCP with about 
100 lines of
    code. Again with such a large set of source.. not bad for making the 
engine support
   both TCP and SCTP.

3) I have also moved ftp/ftpd and inetd all to support SCTP with very 
little changes (we are
    talking a few lines of code).

4) Peter Lei moved the complete ssh suite over SCTP with a few lines of 
change.

So I do disagree with you that moving to SCTP is tough.. it just is not...

But I do, as I said above, agree, your idea for transport consensus 
looks like a
decent compromise to me ... but I have doubts as to if the IESG will 
agree :-0

R

>-- Jeff
>
>  
>
>>-----Original Message-----
>>From: Alex Audu [mailto:[email-masked]]
>>Sent: Wednesday, November 19, 2003 2:05 PM
>>To: MEYER,JEFFREY D (HP-Cupertino,ex1)
>>Cc: 'Danny McPherson'; ipfix wg
>>Subject: Re: [ipfix] Forming consensus on IPFIX default protocol
>>
>>
>>I know I am thousands of e-mails behind. I appologize for that. But I
>>wish people would stop making this lame argument that "SCTP is kernel
>>space protocol".  It is just totally bogus! And to say there is
>>only one user space implementation but it is not 
>>multiprocessing is just
>>incredibly disingenuous as well.
>>
>>I voted for SCTP because I know it is a better protocol. That is from
>>experience. I have articulated my reasons (all technical) on 
>>this list. And
>>if we can get past the emotional argument that CISCO may be pushing
>>SCTP, you'll agree that SCTP is a better choice than TCP for IPFIX.
>>It should be pointed out that SCTP's design was a group 
>>affort involving
>>people from Motorola, Cisco, Siemens, Nortel, Ericsson, Telocrdia and
>>other organizations.
>>
>>I hope we can get this behind us and move on.
>>
>>Regards,
>>Alex.
>>
>>
>>"MEYER,JEFFREY D (HP-Cupertino,ex1)" wrote:
>>
>>    
>>
>>>Danny,
>>>
>>>  Thanks for the additional information.  Sorry I couldn't 
>>>      
>>>
>>personally attend
>>    
>>
>>>the IETF, but dollars for travel are tight in our neck of the woods.
>>>
>>>  I certainly agree with your sentiment around UDP.  
>>>      
>>>
>>Regardless of IETF
>>    
>>
>>>policy, I suspect this will dominate for some time.
>>>
>>>  The one concern is the "implement what you like aspect".  
>>>      
>>>
>>The decision
>>    
>>
>>>dictates the mandatory support of SCTP, which is fine if 
>>>      
>>>
>>you're running on
>>    
>>
>>>Linux, but is much more problematic on other platforms.  
>>>      
>>>
>>There are key
>>    
>>
>>>differences between deploying kernel space and user space 
>>>      
>>>
>>implementations.
>>    
>>
>>>And although a user space implementation of SCTP is readily 
>>>      
>>>
>>available it
>>    
>>
>>>effectively allows only one process to use SCTP, because 
>>>      
>>>
>>all IP traffic
>>    
>>
>>>which is targeted at the SCTP IP protocol id is targeted to 
>>>      
>>>
>>a single user
>>    
>>
>>>space process.
>>>
>>>  As a product developer who needs to address customer's 
>>>      
>>>
>>desires to run on
>>    
>>
>>>Linux, HP-UX, Solaris and Windows, this mandatory aspect 
>>>      
>>>
>>raises serious
>>    
>>
>>>concerns, this is doubled when we are talking about Java 
>>>      
>>>
>>based collectors.
>>    
>>
>>>  In an ideal world all platforms would have SCTP out of 
>>>      
>>>
>>the box and Java
>>    
>>
>>>would have a nice object oriented class of sockets which 
>>>      
>>>
>>supports it.  This
>>    
>>
>>>is not the world today, nor do I expect it to be the world 
>>>      
>>>
>>for some time.
>>    
>>
>>>  If we're all willing to wink and agree that we're really 
>>>      
>>>
>>just going to use
>>    
>>
>>>UDP for the forseeable future, and that the choice of SCTP is really
>>>targeted at some distant nirvana-esque future, then I guess 
>>>      
>>>
>>it doesn't
>>    
>>
>>>matter what is chosen.  This seems to be the status of 
>>>      
>>>
>>Diameter today
>>    
>>
>>>(although the wink is around TCP vs. SCTP).
>>>
>>>  If however we want to deal with the brutal reality of the 
>>>      
>>>
>>playing field
>>    
>>
>>>today and define a protocol which can be readily realized 
>>>      
>>>
>>on any number of
>>    
>>
>>>platforms, then the only choice (given IETF constraints) is 
>>>      
>>>
>>TCP.  As Stuart
>>    
>>
>>>Smally says, "Denial ain't just a river in Egypt."
>>>
>>>Regards,
>>>
>>>  Jeff Meyer
>>>
>>>-----Original Message-----
>>>From: majordomo listserver 
>>>      
>>>
>[mailto:[email-masked]]On Behalf
>  
>
>>Of Danny McPherson
>>Sent: Wednesday, November 12, 2003 9:34 PM
>>To: ipfix wg
>>Subject: Re: [ipfix] Forming consensus on IPFIX default protocol
>>
>>I saw lots of folks raise their hand when the "How many
>>folks would like to see SCTP be the default" (~twice as many
>>as opposed to TCP) question was asked and a great number
>>of those were NOT Cisco employees -- not that it's even a
>>relevant argument here as we're all individuals!
>>
>>I've never liked the hand-raising thing much anyways, and
>>I'd prefer "humming" or nothing at all in the meeting, but
>>nonetheless...
>>
>>As a large consumer of flow information in my day job, I
>>suspect UDP will be all that matters in the near term (for
>>a number of presumably obvious reasons), and when folks are
>>ready to make a change SCTP does have some appealing
>>attributes.
>>
>>And of course, you're always welcome to implement anything
>>you'd like...
>>
>>-danny
>>
>>On Nov 12, 2003, at 6:09 PM, MEYER,JEFFREY D (HP-Cupertino,ex1) wrote:
>>
>>    
>>
>>>Nevil,
>>>
>>>  I guess I haven't heard anyone other than Cisco strongly supporting
>>>SCTP,
>>>but
>>>then again that was my impression for NFv9 vs. the other candidates.  I
>>>guess
>>>will just sit back and let John Chambers do the driving...
>>>      
>>>
>>--
>>Help        mailto:[email-masked] and say "help" in message
>>body
>>Unsubscribe mailto:[email-masked] and say
>>"unsubscribe ipfix" in message body
>>Archive     http://ipfix.doit.wisc.edu/archive/
>>
>>--
>>Help        mailto:[email-masked] and say "help" in message
>>    
>>
>body
>  
>
>>Unsubscribe mailto:[email-masked] and say
>>"unsubscribe ipfix" in message body
>>Archive     http://ipfix.doit.wisc.edu/archive/
>>    
>>
>
>--
>Help        mailto:[email-masked] and say "help" in message
body
>Unsubscribe mailto:[email-masked] and say
>"unsubscribe ipfix" in message body
>Archive     http://ipfix.doit.wisc.edu/archive/
>
>  
>

-- 
Randall R. Stewart
ITD
Cisco Systems Inc.
[email-masked] 815-342-5222(cell) or 815-477-2127(office)

--
Help        mailto:[email-masked] and say "help" in message body
Unsubscribe mailto:[email-masked] and say
"unsubscribe ipfix" in message body
Archive     http://ipfix.doit.wisc.edu/archive/

---

From: Nevil Brownlee <[email-masked]>
Date: Fri, 21 Nov 2003 09:06:17 +1300

IPFIXers:

>      UDP ...

Please guys, stop wasting your energy on discussing UDP, it's definitely
a rathole.

 - The IPFIX charter says "must run over a congestion-aware transport"
 - As WG members/participants we agreed to that charter
 - Now lets get the Information Model and Protocol details worked out
   so that IPFIX can run over TCP, SCTP and whatever else may be suitable
   in whatever situation

Of course any vendor is free to implement other transports, including
UDP.  But the IETF isn't going to publish a Standards-track RFC for a
new application-layer protocol which mentions UDP.

CHeers, Nevil

-----------------------------------------------------------------------
   Nevil Brownlee                   Director, Technology Development
   Phone: +64 9 373 7599 x88941     ITSS, The University of Auckland
   FAX: +64 9 373 7021      Private Bag 92019, Auckland, New Zealand

-------------------------------------------------
This mail sent through University of Auckland
http://www.auckland.ac.nz/

---

From: Mark Fullmer <[email-masked]>
Date: Thu, 20 Nov 2003 15:16:59 -0500

How many times does this discussion need to die?

We've been told no UDP.  Not MAY, not SHOULD, NEVER.

If there is vendor interest in running IPFIX over UDP then this
needs to happen off the IETF list.

Can we Please just let the transport issue die already?  IPFIX
will be specified to run over TCP and SCTP.  IPFIX will run over UDP
because vendors are going to implement it this way regardless of the
RFC/ID/IETF process.

mark

---

From: Peter Lei <[email-masked]>
Date: Thu, 20 Nov 2003 14:59:43 -0600

So... despite being very doubtful UDP can appear in the text
at all (we've been told that many times already), this seems
like reasonable text to me.

regards,
--peter

---

From: Peter Lei <[email-masked]>
Date: Thu, 20 Nov 2003 15:03:52 -0600

> will be specified to run over TCP and SCTP.  IPFIX will run over UDP
   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

yes... but we need to have text indicating this...  IMO the
proposed text is an attempt to get that consensus. i.e. SHOULD
use SCTP, MAY use TCP.

regards,
--peter

---

From: Robert Lowe <[email-masked]>
Date: Thu, 20 Nov 2003 15:04:17 -0600

> (especially SCTP-PR) and its message vs. stream orientation.

      Other non-congestion aware protocols MAY be used in deployments where
exporters and collectors can communicate over dedicated links which are not
susceptible to congestion issues.

No mention of UDP, or...

-Robert

---

From: "Randall Stewart (cisco)" <[email-masked]>
Date: Thu, 20 Nov 2003 18:32:27 -0600

Rob:

I think you may have hit upon the right combination here.. This seems
like it might be able to pass the test for the IESG.... not sure 
though.. let
me see if I can get an answer from where the objections will come.

I will report back in a bit :->

R

---

From: Mark Fullmer <[email-masked]>
Date: Thu, 20 Nov 2003 21:53:47 -0500

My personal view of reality here:

   1) UDP will remain in use to transport NetFlow and probably IPFIX
      as the transport of choice for at least the short term, where short
      term is years.

   2) SCTP may eventually take over, largely because Cisco is
      putting the development effort to get products to market
      in the short term with NFv9 over SCTP and hopefully in the
      long term with IPFIX over SCTP.

   3) TCP may end up playing a roll it may not.

So it doesn't matter where the SHOULD, MAY, or MUST go.  The market
will choose, not IETF.  A "MUST" for TCP is very unlikely going to
influence vendors who believe the "MUST" should be for SCTP or
visa versa.

At the last WG meeting SCTP was chosen as the default transport.  My
memory is that someone was going to post this to the mailing list
for a last call.

If I post a "last call" for the IPFIX transport with TCP and SCTP
as the options will everyone finally accept the vote as the final
decision?

ps, my memory about UDP not being an option was confirmed privately.

mark

---

From: Sebastian Zander <[email-masked]>
Date: Fri, 21 Nov 2003 11:17:38 +0100

> influence vendors who believe the "MUST" should be for SCTP or

I believe that if one protocol has serious advantages over the other
it should be given preference by the spec. The market may decide against
the "better" solution but we are engineers...

Similar if one protocol is more suitable under certain circumstances
the spec should give the reader advice when to use which protocol.

> as the options will everyone finally accept the vote as the final

Yes, a lastcall on the draft wording please! I think Jeff's text is an
excellent starting point. Personally I'm not against mentioning non
congestion-aware transport but if IESG won't accept it remove UDP
and congestion issues from the text (-> IPFIX is congestion-aware -
in the spec). Make the "why SCTP is preferred" a separate sentence/
paragraph for better readability.

My 0.02 cents,

Sebastian

-- 
Sebastian Zander                         E-mail: [email-masked]
Fraunhofer FOKUS / METEOR                Tel: +49-30-3463-7287
Kaiserin-Augusta-Allee 31                Fax: +49-30-3463-8287
D-10589 Berlin, Germany                  www.fokus.fraunhofer.de/usr/sebastian.zander

---

From: Stewart Bryant <[email-masked]>
Date: Fri, 21 Nov 2003 12:12:54 +0000

> ps, my memory about UDP not being an option was confirmed privately.

Although after the meeting Jon Peterson (Transport AD) told me that
he would be amenable to UDP, *PROVIDED* we wrote a suitable
applicability statement governing its use.

A new AD will be appointed to supervise IPFIX, so perhaps we could
park the UDP issue until they are on board?

Stewart

---

From: "Randall Stewart (cisco)" <[email-masked]>
Date: Fri, 21 Nov 2003 06:54:54 -0600

Dear All:

After an off-line discussion with Allision and Jon, with a few
minor tweaks (included below) the transport AD's  are ok
with the following:

"
     SCTP SHOULD be used in deployments where exporters and collectors are
     communicating over links which are susceptible to congestion.
    TCP MAY be used in deployments where exporters and collectors
     communicate over links which are suscepible to congestion, but SCTP
     is preferred, due to its ability to limit back pressure on exporters
     (especially when using PR-SCTP) and its message vs. stream 
orientation.

    Other non-congestion aware protocols MAY be used in deployments where
    exporters and collectors always communicate over dedicated links 
which are not
    susceptible to congestion.
"

I think this allows the reality and provides for a future migration to 
congestion
aware protocols...

Any objections or comments???

R

---

From: Robert Lowe <[email-masked]>
Date: Fri, 21 Nov 2003 09:57:38 -0600

>     communicating over links which are susceptible to congestion.

Slight grammatical tweak: change "are communicating" to "communicate".  Also,
would it be wise to spell out *exactly* what "susceptible to congestion"
means, i.e. flat-out state "any public communication links/Internet"???  Or
do notes like that belong in the applicability document?

>     (especially when using PR-SCTP) and its message vs. stream orientation.
> 

Even if the offer were there to mention UDP specifically, why not just
leave the door open for others too?  That way you avoid even the slightest
hint of an endorsed recommendation, and everyone is happy.

-Robert