Subject: [IPFIX] recent ipfix drafts and argus

From: Carter Bullard <[email-masked]>
Date: Mon, 27 Feb 2012 14:37:18 -0500

Gentle people,
I'm generally pretty quiet when it comes to IPFIX and its efforts.  But as the first
person to develop IP flow records in the 1980's, first to present the idea to the
community in 1992, the first to provide open source flow technology in 1995,
and the author of the longest lived open source flow system, argus; I feel that
I have to say something about the recent wave of IPFIX drafts.

The drafts on flow aggregation describe functionality that the Argus project started 
over 20 years ago.  The ideas of key modification, conversion of non-key attributes
to key members, aggregation operators, interval distribution and the architecture for it,
were all developed in argus a long long time ago.  draft-ietf-ipfix-a9n is basically
describing the functionality of argus's racluster(), rasplit(), and rabins() programs,
and every example given in the text of draft-ietf-ipfix-a9n can be generated using
argus's rabins(), with only a few gyrations of its command-line, today.

I personally would expect that if the IETF was going to describe something that is
"Standards Track", that there would be dozen's of implementations of this kind of
technology available, and that the WG is condensing years of experience to
arrive at a "Standards Track", but, this is not the case.  There is only one current
implementation of the complete capabilities of the features of draft-ietf-ipfix-a9n
that I am aware of, and that is in argus.

Taking just one of the technical descriptions in the draft, "interval distribution", I
am not aware of any description of this issue, or implementation of this type
of technology in the literature, outside of argus.  No Google search results for "flow
interval distribution".   In Argus we call it flow splitting.  The first line from a
Google search for "argus flow splitting" return:

Scholarly articles for argus flow splitting
… and prediction of flow statistics from sampled packet … - Duffield - Cited by 217

I'm not saying that Nick knows much about argus's support for flow splitting, but
its still pretty scary that the first hit is from a paper that is used in IPFIX documents.
One would have to assume that the IPFIX community should be aware.

My problem is that most of  draft-ietf-ipfix-a9n is prior work that is not widely
implemented, some of the features are still unique to argus.   While IETF support
of technology is a good thing, descriptions of technology without reference
is a difficult thing to interpret.  Is the IPFIX WG describing what they think is new
technology? Does the IPFIX WG think that many companies have implemented
this type of technology, and now its time to standardize it ?  Well, I'm not aware
of any implementation, open or closed, that does the complete set of what the
draft is recommending, other than argus.  So I don't think its new, nor widely
implemented.  I would say its a form of technology plagiarism.

IPFIX is considering adding non-IP flows to their definitions.  Argus is the only available
flow technology that has significant non-IP flow data models and support.  argus-1.2 had
flow generation, transport, analytics and storage of non-IP flows 20 years ago, with its
support for bi-directional ethernet, apple-talk and ARP transaction tracking and reporting.
In the last 10 years, argus has added MPLS, VLAN, ISO addresses, and Infiniband flow
models.  Not attributes, but true flow key elements.   This work is non-trivial.

The concept that the WG would consider dropping the IP from IPFIX and think that is
all that is needed, is really so completely wrong, that its laughable, and a dis-service
to those that have done the hard work to bring situational awareness and analytics
to non-IP traffic.   The same applies to bi-directional flows, but that is another story.

I would love to think that IPFIX could focus back on flow information exchange.
Multicast, non-template based connectionless transport strategies, say over UDT
as an example, rather than getting into areas for which the WG is unprepared to
do even a reasonable job, without resorting to dubious techniques.

Just a few comments, I hope that anyone finds it useful.

Carter

Carter Bullard
CEO/President
QoSient, LLC
150 E. 57th Street Suite 12D
New York, New York 10022

+1 212 588-9133 Phone
+1 212 588-9134 Fax

---

From: Benoit Claise <[email-masked]>
Date: Mon, 27 Feb 2012 23:52:56 +0100

Hi Carter,

After trying to abstract the style of your email, which I don't 
appreciate, I'm not too sure how to read your email.
Is this an IP claim? Or just "I've been doing this for years, so I know 
better"?

In all cases, that's a nice advertisement for your company... Maybe it 
was the point...

On my side, I certainly don't get my ideas from your products!
The last time I looked up your web site was at the time of RFC3955.
In total in my live, I don't think I spend more than 1/2 h on your web site.

And I don't feel like replying to the details of this email, or even 
playing the little game of comparing features of your company/my company.

Regards, Benoit.
>       Scholarly articles for *argus flow splitting*
>       <http://scholar.google.com/scholar?q=argus+flow+splitting&hl=en&as_sdt=0&as_vis=1&oi=scholart&sa=X&ei=-8NLT_6lKcnb0QHVs6z7DQ&ved=0CBoQgQMwAA>
>
> ... and prediction of *flow *statistics from sampled packet ... 
> <http://www.google.com/url?url=http://scholar.google.com/scholar_url%3Fhl%3Den%26q%3Dhttp://dl.acm.org/citation.cfm%253Fid%253D637225%26sa%3DX%26scisig%3DAAGBfm1Qq9_hOFJINho1051rzZ6qOD5wuA%26oi%3Dscholarr&rct=j&sa=X&ei=-8NLT_6lKcnb0QHVs6z7DQ&ved=0CBsQgAMoADAA&q=argus+flow+splitting&usg=AFQjCNFuMuC_b45uErbgoPHPab61egoZ3g> - 
> Duffield - Cited by 217
>
>
> _______________________________________________
> IPFIX mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ipfix

---

From: Carter Bullard <[email-masked]>
Date: Mon, 27 Feb 2012 18:49:56 -0500

Hey Benoit,
Well you should pay some attention.  You should know if there is prior art before you start presenting descriptions of technology, and you should give credit to that prior art.  Problem's come up when you present work, and your descriptions look like the prior art's source code, and your examples look like that prior arts program output.   Maybe its just a coincidence.

Argus is free and open source software, and the concepts that you are presenting in your draft were implemented in argus over 19 years ago, so I don't think I'm too worried about IP.  Its just the arrogance of it all that is a little bit of a concern.  IPFIX isn't doing the community any favors if its only authors don't have a clue.

Carter

Carter Bullard
CEO/President
QoSient, LLC

+1 212 588-9133 Phone
+1 212 588-9134 Fax

On Feb 27, 2012, at 5:52 PM, Benoit Claise wrote:

> Hi Carter,
> 
> After trying to abstract the style of your email, which I don't appreciate, I'm not too sure how to read your email. 
> Is this an IP claim? Or just "I've been doing this for years, so I know better"?
> 
> In all cases, that's a nice advertisement for your company... Maybe it was the point...
> 
> And I don't feel like replying to the details of this email, or even playing the little game of comparing features of your company/my company.
> 
> Regards, Benoit.
>> Gentle people,

---

From: Juergen Quittek <[email-masked]>
Date: Tue, 28 Feb 2012 09:25:33 +0000

Hi Carter,

You know the IPFIX WG for long and our process has always been open.
Aggregation (also referred to as "concentration") was always among
the requirements to take care of by the IPFIX WG, see, for example,
RFC 3917 from 2004.

The open IETF process allows anybody to support technical documents
with solutions, such as ones on IPFIX aggregation.  So far, no such
contribution has been made by argus developers or users.  This is a
pity, but nothing you can blame any active members of the IPFIX WG
for.  If you want to blame someone, then rather blame the argus
community including yourself.  You aware of the IPFIX WG and
contributions from you or argus users would have been highly
appreciated.

The IPFIX WG is and will always open for contributions from the
argus community.  If you think that a technical contributions from
argus or a reference to argus would improve any of our current
documents, then please contribute to them, as you had already done
in the past.  Please particularly do so if you think the way argus
solved technical problems is better than what has so far been
contributed to the IPFIX WG.

While your technical contributions will always be welcome, I ask
you in my role as WG co-chair to stop making harsh statements on
the mailing list, even though the main effect that you achieve
with these statements is discrediting yourself.  In particular
I refer to two statements you made:

1. You accuse authors of draft draft-ietf-ipfix-a9n of plagiarism.
Your main reason for this was that you stated "I had the same idea
before" and it's publicly available.  I haven't checked if the
ideas are exactly the same, but this is not the point.  It does
happen that two people independently have the same idea.
For accusing someone of plagiarism, more evidence is necessary,
at least if you want to be taken serious.

2. You claim that authors of draft-ietf-ipfix-a9n are lacking
expertise.  Indeed, there are leading experts among the authors
with high reputation and high expertise in the area of IP
flow monitoring.  Claiming they "don't have a clue" because they
don't know argus by heart is a statement that may be valid in
an argus-centric world, but not in the world of the IP flow
monitoring community.

I look forward to your technical contributions for improving the
Standards we are developing in the IPFIX WG.

    Juergen
    WG co-chair

On 28.02.12 00:49, "Carter Bullard" <[email-masked]> wrote:

>Hey Benoit,Well you should pay some attention.  You should know if there
>is prior art before you start presenting descriptions of technology, and
>you should give credit to that prior art.  Problem's come up when you
>present work, and your descriptions look like the prior art's source
>code, and your examples look like that prior arts program output.   Maybe
>its just a coincidence.
>
>Argus is free and open source software, and the concepts that you are
>presenting in your draft were implemented in argus over 19 years ago, so
>I don't think I'm too worried about IP.  Its just the arrogance of it all
>that is a little bit of a concern.  IPFIX isn't doing the community any
>favors if its only authors don't have a clue.
>
>Carter
>
>Carter Bullard
>CEO/President
>QoSient, LLC
>      to key members, aggregation operators, interval
>        distribution and the architecture for it,
>      describing the functionality of
>        argus's racluster(), rasplit(), and rabins() programs,
><http://scholar.google.com/scholar?q=argus+flow+splitting&hl=en&as_sdt=0&a
>s_vis=1&oi=scholart&sa=X&ei=-8NLT_6lKcnb0QHVs6z7DQ&ved=0CBoQgQMwAA>
>                
>              
>              
>                
>                  Š
>                      and prediction of flow statistics from
>                      sampled packet Š
><http://www.google.com/url?url=http://scholar.google.com/scholar_url%3Fhl%
>3Den%26q%3Dhttp://dl.acm.org/citation.cfm%253Fid%253D637225%26sa%3DX%26sci
>sig%3DAAGBfm1Qq9_hOFJINho1051rzZ6qOD5wuA%26oi%3Dscholarr&rct=j&sa=X&ei=-8N
>LT_6lKcnb0QHVs6z7DQ&ved=0CBsQgAMoADAA&q=argus+flow+splitting&usg=AFQjCNFuM
>uC_b45uErbgoPHPab61egoZ3g> - Duffield -
>                      Cited by 217
>                
>              
>            
>          
>        
>        
>        
>      
>[email-masked]://www.ietf.org/mailman/listinfo/ipfix
>    
>
>    
>  
>
>
>
>
>
>
>

---

From: Carter Bullard <[email-masked]>
Date: Tue, 28 Feb 2012 10:13:04 -0500

Hey Juergen,
It is not the responsibility of groups outside of the IETF to police IETF working groups
to ensure that they are doing the right thing. The IETF itself, has a responsibility to do
the right thing when it comes to publishing technical documents.

I am providing input to the WG that draft-ietf-ipfix-a9n-3 is not original work, the concepts
have been worked in the communications industry for a very long time, and there are open
public implementations of every concept in the draft.   I am also providing input to the WG
that draft-ietf-ipfix-a9n-3 appears to be using concepts, and examples that are curiously
specific to actual implementation and publications developed by Argus.

Based on the response from the authors to the mailing list, its clear that the authors
have not done any due diligence to ensure that their draft conforms to even the simplest of
ethics in technical publication.  Plagiarism is a very serious problem, and the authors
should know how to protect themselves from such a claim.

I am providing input to the WG that I believe the authors cannot defend themselves
from such a claim with their current draft, and that the WG should respond accordingly.

I would hope that the WG will use this input to improve their processes and products,
something that would benefit the entire flow community.

Carter

Carter Bullard
CEO/President
QoSient, LLC

+1 212 588-9133 Phone
+1 212 588-9134 Fax

---

From: Juergen Quittek <[email-masked]>
Date: Tue, 28 Feb 2012 16:32:22 +0000

Hi Carter,

It looks like there is a misconception on your side concerning IETF
publications.  You are confusing them with scientific conference and
journal publications.  The target of the IETF is not serving the community
with new unpublished ideas, but with standards specifications.  Whether
the technologies used for standards have been published before is not of
high relevance.  But of high relevance is if they are suited as a
standard.  The IETF has even a mechanism where people can claim IPR
related to IETF documents.  This is completely independent from the
authorship.

According to the rules of the IETF, authors of WG documents are assigned
by the WG chairs.  Typically these are the same authors that worked on an
individual draft on the same topic before, but not necessarily.  And there
is no rule at all at the IETF stating that a person can only be an author,
it he or she had the original idea described in the text.  If an RFC is
revised then often the original authors are removed and the ones making
the revision are added.  There is not concern about 'originality'.

All your complaints in this direction are pointless.

However, what the WG may consider is adding a reference to argus to
draft-ietf-ipfix-a9n.  You are welcome to provide text on which concepts
and methods are already well understood and tested from experiences with
argus.

    Juergen

On 28.02.12 16:13, "Carter Bullard" <[email-masked]> wrote:

>Hey Juergen,It is not the responsibility of groups outside of the IETF to
>police IETF working groups
>   Is this an IP claim? Or just "I've been doing this for years, so I
>
>   know better"?
>
>
>
>   In all cases, that's a nice advertisement for your company... Maybe
>
>   it was the point...
>
>
>
>   In total in my live, I don't think I spend more than 1/2 h on your
>
>   web site.
>
>
>
>   playing the little game of comparing features of your company/my
>
>   company.
>
>
>
>   Regards, Benoit.
>
>
>
>Gentle people,
>
>     I'm generally pretty quiet when it comes to IPFIX and its
>
>       efforts.  But as the first
>
>     person to develop IP flow records in the 1980's, first to
>
>       present the idea to the
>
>     community in 1992, the first to provide open source flow
>
>       technology in 1995,
>
>     and the author of the longest lived open source flow system,
>
>       argus; I feel that
>
>     I have to say something about the recent wave of IPFIX
>
>       drafts.
>
>
>
>
>
>     The drafts on flow aggregation describe functionality that
>
>       the Argus project started
>
>     over 20 years ago.  The ideas of key modification, conversion
>
>       of non-key attributes
>
>     were all developed in argus a long long time ago.
>
>        draft-ietf-ipfix-a9n is basically
>
>     and every example given in the text of draft-ietf-ipfix-a9n
>
>       can be generated using
>
>     argus's rabins(), with only a few gyrations of its
>
>       command-line, today.
>
>
>
>
>
>     I personally would expect that if the IETF was going to
>
>       describe something that is
>
>     "Standards Track", that there would be dozen's of
>
>       implementations of this kind of
>
>     technology available, and that the WG is condensing years of
>
>       experience to
>
>     arrive at a "Standards Track", but, this is not the case.
>
>        There is only one current
>
>     implementation of the complete capabilities of the features
>
>       of draft-ietf-ipfix-a9n
>
>     Taking just one of the technical descriptions in the
>
>       draft, "interval distribution", I
>
>     am not aware of any description of this issue,
>
>       or implementation of this type
>
>     of technology in the literature, outside of argus.  No Google
>
>       search results for "flow
>
>     interval distribution".   In Argus we call it flow splitting.
>
>        The first line from a
>
>                 Scholarly
>                     articles for argus flow splitting
>
>     I'm not saying that Nick knows much about argus's support for
>
>       flow splitting, but
>
>     its still pretty scary that the first hit is from a paper
>
>       that is used in IPFIX documents.
>
>     One would have to assume that the IPFIX community should be
>
>       aware.
>
>
>
>
>
>     My problem is that most of  draft-ietf-ipfix-a9n is prior
>
>       work that is not widely
>
>     implemented, some of the features are still unique to argus.
>
>         While IETF support
>
>     of technology is a good thing, descriptions of technology
>
>       without reference
>
>     is a difficult thing to interpret.  Is the IPFIX WG
>
>       describing what they think is new
>
>     technology? Does the IPFIX WG think that many companies have
>
>       implemented
>
>     this type of technology, and now its time to standardize it ?
>
>        Well, I'm not aware
>
>     of any implementation, open or closed, that does the complete
>
>       set of what the
>
>     draft is recommending, other than argus.  So I don't think
>
>       its new, nor widely
>
>     implemented.  I would say its a form of technology
>
>       plagiarism.
>
>
>
>
>
>     IPFIX is considering adding non-IP flows to their
>
>       definitions.  Argus is the only available
>
>     flow technology that has significant non-IP flow data models
>
>       and support.  argus-1.2 had
>
>     flow generation, transport, analytics and storage of non-IP
>
>       flows 20 years ago, with its
>
>     support for bi-directional ethernet, apple-talk and ARP
>
>       transaction tracking and reporting.
>
>     In the last 10 years, argus has added MPLS, VLAN, ISO
>
>       addresses, and Infiniband flow
>
>     models.  Not attributes, but true flow key elements.   This
>
>       work is non-trivial.
>
>
>
>
>
>     The concept that the WG would consider dropping the IP from
>
>       IPFIX and think that is
>
>     all that is needed, is really so completely wrong, that its
>
>       laughable, and a dis-service
>
>     to those that have done the hard work to bring
>
>       situational awareness and analytics
>
>     to non-IP traffic.   The same applies to bi-directional
>
>       flows, but that is another story.
>
>
>
>
>
>     I would love to think that IPFIX could focus back on flow
>
>       information exchange.
>
>     Multicast, non-template based connectionless transport
>
>       strategies, say over UDT
>
>     as an example, rather than getting into areas for which the
>
>       WG is unprepared to
>
>     do even a reasonable job, without resorting to dubious
>
>       techniques.
>
>
>
>
>

---

From: Carter Bullard <[email-masked]>
Date: Tue, 28 Feb 2012 12:03:08 -0500

Hey Juergen,
I know how the IETF works.  As you remember, I was a principal contributor
to the meetings that lead to the formation of IPFIX.   But all IETF documents
have an Acknowledgements and References sections for a reason.
When your authors use examples or technology that are well developed in
other projects, you should give those projects credit for the work they have
done.

Carter

Carter Bullard
CEO/President
QoSient, LLC

+1 212 588-9133 Phone
+1 212 588-9134 Fax

---

From: Nevil Brownlee <[email-masked]>
Date: Tue, 28 Feb 2012 23:48:59 -0800

Hi Carter:

As the other IPFIX co-chair I feel bound to respond to your comments.

Your idea that an Internet Standard should document something
that has "dozens of implementations" is weird, to say the least.
In many cases there are different groups of people proposing
different ideas, so to begin with there isn't even a single
implementation.

Your comment about 'a Google search for "Argus flow splitting"' is
disingenuous.  When I spent some time with Google looking for
information about Argus and its implementation, I simply wasn't able
to find anything.  Nick Duffield's papers about sampling, i.e.
   "Charging from sampled network usage" (2001),
   "Properties and Prediction of Flow Statistics from Sampled
     Packet Streams, " (2002),
   "Learn more, sample less: control of volume and variance in
     network measurement," (2005),
all cite the Quosient web page for Argus' way of defining flows,
but there's no text in any of these papers about Argus itself!

The IPFIX WG provides a forum for interested people to contribute
ideas about information reporting.  If you have something you'd like to
see in a WG draft, you need to contribute some text about it on the
IPFIX list. In the case of the Aggregation draft, you simply haven't
done that.

In short:
  - If there is published material out there about Argus,
      please tell us about it.
  - Kindly withdraw your mischievous accusations of plagiarism, these
      are unfounded.
  - Constructive technical discussion to the IPFIX list is, of course
      always welcome.

Cheers, Nevil

-- 
---------------------------------------------------------------------
  Nevil Brownlee                    Computer Science Department | ITS
  Phone: +64 9 373 7599 x88941             The University of Auckland
  FAX: +64 9 373 7453   Private Bag 92019, Auckland 1142, New Zealand

---

From: Carter Bullard <[email-masked]>
Date: Wed, 29 Feb 2012 11:38:43 -0500

Nevil,
I wasn't being disingenuous about Nick or the use of Google in any way.
Nick put in references to argus in his papers, and he didn't even really
say anything about it.  He even spelled the company's name correctly.

What I am saying is the aggregation draft is dealing with a topic that has
been around for a very long time, and there are lots of implementations of
all of the concepts needed in data aggregation around.  Billing packages,
statistics packages,  graphing software all have to deal with these issues.
I can provide you lots of text books that are good references on the topics of
data aggregation that the flow community needs, if you are interested.

With such an unbounded topic to talk about, with so much opportunity,
why do all of the examples in the draft look like documented argus examples
and program outputs?

I stated this in the original email.  I am stating it now.  If you are going to use
examples and technology from other efforts, reference those other efforts.

But each example in the draft is either a specific aggregation where argus has
dedicated programs to generate the output, or they are well documented
examples from the argus web site or mailing list.  

For god's sake, just change the specific objects, the time intervals
and the order and format of the output fields, and you would at least
not look like you took the examples from argus program output.

If you want to look like argus program output, fine. Mention argus.
If you want to look like SiLK, then use SiLK, and mention it.  If you don't
want to look like anybody's tools, then just change the format so it doesn't
look like anybody's tools.

The IPFIX WG is just a working group in the IETF.  It's mailing list is the
only avenue to present issues regarding IPFIX WG business.  I am
saying again, and hopefully for the last time, if you are going to use 
the work of other forums and groups in the flow community, simply give
those efforts credit.  If you don't want to do that, modify your work so
that it doesn't look like you just lifted it, without concern.

Carter

Carter Bullard
CEO/President
QoSient, LLC

+1 212 588-9133 Phone
+1 212 588-9134 Fax

---

From: Juergen Quittek <[email-masked]>
Date: Wed, 29 Feb 2012 21:41:23 +0000

Hi Carter,

If you think it would be beneficial for the draft to have a reference
to argus, then please make a concrete proposal which text or reference
should be added where to the draft.  Improvements are always welcome.

And yes, the IPFIX WG does give credits to contributions and sources
of ideas.  If we missed anything we accept comments and add credits.
And we welcome hints in this direction if they are concrete.

You claim in your message below that examples in draft-ietf-ipfix-a9n
have been taken from the argus web page.  So far you have not pointed
at anything concrete.  Your statements are too general such as "each
example in the draft" has taken somehow from argus.  Please be specific.
Which example from the IETF draft has been taken from which example on
the argus web site?

    Juergen 

On 29.02.12 17:38, "Carter Bullard" <[email-masked]> wrote:

>Nevil,I wasn't being disingenuous about Nick or the use of Google in any
>way.
>Š and prediction of flow statistics from sampled packet Š - Duffield -
>Cited by 217
>
>
>