Subject: Re: [ippm] Fwd: New Version Notification for draft-morton-ippm-twamp-rate-02.txt

From: Christofer Flinta <[email-masked]>
Date: Fri, 1 Feb 2013 14:56:58 +0000

Hi Al,

I get the impression that the use case for the draft is to monitor the subscriber's capacity between the CPE and some point higher up in the operator network in both directions.  If this is the case, I think there should be a problem statement in the introduction part of the draft, describing this use case. Also, since modems and home gateways have scarce resources when it comes to CPU and memory, I think that it should be stated in the draft that there are some requirements on the measurement protocol, such as it must be simple and light weight. With a description of this use case and its associated requirements I find it more acceptable to have a protocol which only sends bursts, since these are the simplest packet ensembles to send, and we know it's a challenge to create packet spacing - especially by a low cost device. 

Would it then be possible to add another use case in the draft, where the reflector is not a low-cost device, and thus less restricted by CPU and memory? This use case could be met in the protocol by another TWAMP mode or parameter that enables trains with different send rates. With two modes/parameters a reflector device could declare its ensemble-sending capabilities in the control protocol, e.g. a CPE modem would signal that it can only send back-to-back bursts, while a more powerful network node may signal that it can send ensembles with different rates within an interval.

I also think there should be a section in the draft around the interpretation of measurements using bursts. There are cases when delivery rate = APC, but there are other cases when they are not the same. We made some simple tests in the lab to illustrate the difference. Trains with 100 packets of test traffic were sent from device A to B via two routers. The bottleneck link between the routers is a 100 Mbps Ethernet link, which then offers 97 Mbps capacity at the IP layer. We defined four scenarios with different cross-traffic rates (0, 25, 50, 75 Mbps -  iperf UDP CBR)  over the bottleneck link. Then we sent trains with different send rate for each scenario. Here is a small table showing the results, where we define APC = 97 Mbps - cross traffic. As we can see, the delivery rates may deviate quite a lot from APC, depending on cross traffic and send rate:

Cross traffic          0     25    50    75
APC                        97     72    47    22
Delivery rate      97     77    65    55    Send rate =  100 Mbps
Delivery rate      97     83    72    63    Send rate =  150 Mbps
Delivery rate      97     94    93    88    Send rate = 1000 Mbps

Looking at a possible down-link measurement scenario in an operator network, the send rate of the test burst will probably be much larger than the link capacity of the subscriber link, which means that the delivery rate will show a value close to the subscriber link capacity almost regardless of the subscriber traffic (see Send rate = 1000 Mbps). Any large deviation from this value can then be interpreted either as congestion higher up in the aggregation part of the network or as reduced link capacity of the subscriber link, e.g. a distorted ADSL link. It's not clear how to differentiate between those interpretations. Perhaps the burst measurements must be complemented with CPE detection of the link state?

In an up-link scenario, the interpretation is even more unclear, since most probably the train send-rate = subscriber link capacity. In this case the subscriber up-link traffic will highly affect the measurements (see Send rate = 100 Mbps), and it will be hard to tell if a low delivery rate depends on either user traffic, congestion in the aggregation part or reduced link capacity. Regarding user traffic, I think it is not safe to assume that there are times when the user traffic is zero. People may play games, use Skype or watch Netflix late at night and file sharing applications are often active any time of the day. Should the CPE also monitor user traffic in order to give more input for the measurements?

Christofer

-----Original Message-----
From: MORTON JR., ALFRED (AL) [mailto:[email-masked]] 
Sent: den 25 januari 2013 00:27
To: Christofer Flinta; [email-masked]
Subject: RE: [ippm] Fwd: New Version Notification for draft-morton-ippm-twamp-rate-02.txt

Hi Christofer,

I think it's worth considering the addition of inter-ensemble packet spacing. It's clear that many past capacity measurement experiments have used this technique. But it's probably worth hearing from folks about their current experiments, too.  For example:

You mentioned that bursts can be used to estimate the peak throughput, and Available Path Capacity is also needed, and unless there is no user traffic, peak > available. I think there are cases where burst and fixed rate ensembles measure the same quantity, but they have different methodological constraints (you can't send bursts greater than the bottleneck buffer capacity, and while you can send very long ensembles of fixed rate packets you would be averaging the rate measurement over a long time scale, too, and you may end up measuring different values because of the different time-scales). 

Most of the Internet access rate studies conducted in the last few years have been designed to avoid times when the customers who host the measurement device are not sending competing traffic. Thus the emphasis is to assess the apparent peak rate (for a typical subscriber's segment of the path), further limited by any other bottlenecks with traffic from other subscribers, sort of an available capacity for the rest of the path. Strictly speaking, the rate being measured doesn't have a parallel in RFC 5136 (for this and other reasons), and that may mean we have more RFCs to write.

It's probably worth examining the requirements to "correctly" generate fixed rate packet ensembles, when the hardware doing the generation and/or measurement was designed for low-cost-large-scale deployment for a purpose other than measurement. Most of us are thinking "LMAP" these days.

In any case, the current TWAMP draft is compliant with the rate-problem draft requirement as stated, because "one or more" spacings includes zero spacing.

I hope other knowledgeable folks will chime in, too.

Al

> -----Original Message-----
> From: Christofer Flinta [mailto:[email-masked]]
> Sent: Thursday, January 24, 2013 5:12 AM
> To: MORTON JR., ALFRED (AL); [email-masked]
> Subject: RE: [ippm] Fwd: New Version Notification for 
> draft-morton-ippm- twamp-rate-02.txt
> 
> Hi Al, IPPM group,
> 
> I have some comments on draft-morton-ippm-twamp-rate-02.
> 
> As I understand it, the proposed protocol enables sending bursts of 
> test packets back-to-back in the forward or reverse direction. From 
> the collected time stamps in each receiving node the packet delivery 
> rate for each burst can be calculated, and this delivery rate may be 
> used as an estimate of the peak throughput for the path. However, 
> while peak throughput is an important measure, I think a general rate 
> measurement protocol should also enable measuring of Available Path 
> Capacity (APC) as defined in RFC 5136. In case of a network with user 
> traffic present (In- Service testing), peak throughput > APC, which 
> means that the delivery rate of a burst of back-to-back packets cannot 
> directly be used as an estimate of APC. Only if the network path is 
> empty of cross traffic (Out- of-Service testing) peak throughput = 
> APC, which means that delivery rate can be used as an APC estimate.
> 
> In order to estimate APC in a general case (both for In-Service 
> testing and Out-of-Service testing) each measurement node must be able 
> to send trains with different rates during a session. APC can then be 
> estimated by comparing the receive rates with the send rates, according to some scheme.
> I think this requirement is in line with the proposed requirement of a 
> test protocol in draft-ietf-ippm-rate-problem-01: "The test protocol 
> SHALL support test packet ensemble generation (category 3) ...", where  
> Category
> 3 is describes as: " One or more packet ensembles in a test stream, 
> using a fixed ensemble size in packets and one or more fixed 
> intra-ensemble packet spacings (including zero spacing)."
> 
> To meet this requirement, I propose two small changes to draft-morton-
> ippm-twamp-rate-02:
> 
> 1) Measuring APC in forward direction.
> The only change needed is to relax the requirement on the sending of 
> packets in 5.1.1 to allow packets to be sent with any packet interval 
> in a burst.
> 
> 2) Measuring APC in reverse direction.
> The Session-Sender needs to inform the Session-Reflector of the 
> desired send rate for each generated burst. The simplest way to do 
> this is to convert the desired send rate to a desired packet interval 
> in NTP format, valid for all packets in the burst, and send that value 
> in the Burst Initiation Packet. The Timestamp field is not used by the 
> reflector and not reflected back in Burst Generation mode and can 
> therefore be reused as a new field, e.g. called Desired Packet 
> Interval. This packet interval should be used by the reflector as a 
> waiting time for each packet in the generated burst, except for the 
> first packet in the burst, which should be sent as quickly as possible.
> 
> I believe these changes may strengthen the draft.
> 
> Best regards
> Christofer Flinta
> Ericsson Research
> 
> 
> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf 
> Of Al Morton
> Sent: den 3 september 2012 15:57
> To: [email-masked]
> Subject: [ippm] Fwd: New Version Notification for 
> draft-morton-ippm-twamp- rate-02.txt
> 
> IPPM,
> 
> We've updated this draft to coordinate with the problem statement as 
> it now stands.
> 
> Al and Len
> 
> >A new version of I-D, draft-morton-ippm-twamp-rate-02.txt
> >has been successfully submitted and posted to the IETF repository.
> >
> >Filename:       draft-morton-ippm-twamp-rate
> >Revision:       02
> >Title:          TWAMP Burst Rate Measurement Features
> >Creation date:  2012-09-03
> >WG ID:          Individual Submission
> >Number of pages: 19
> >URL:
> >http://www.ietf.org/internet-drafts/draft-morton-ippm-twamp-rate-02.txt
> >Status:          http://datatracker.ietf.org/doc/draft-morton-ippm-twamp-
> rate
> >Htmlized:        http://tools.ietf.org/html/draft-morton-ippm-twamp-rate-
> 02
> >Diff:
> >http://www.ietf.org/rfcdiff?url2=draft-morton-ippm-twamp-rate-02
> >
> >Abstract:
> >    This memo describes two rate-measurement features for the core
> >    specification of TWAMP - the Two-Way Active Measurement Protocol: an
> >    optional capability where the reflector host responds with a
> >    controlled burst of test-session packets (instead of a single
> >    packet), and an optional test mode that requires the responder to
> >    measure a burst of test packets and communicate the results in
> >    truncated packet(s).  Both features add the ability to control packet
> >    size in the tested direction, enabling asymmetrical packet size
> >    testing.  There is an open question on using TCP transport instead of
> >    UDP.
> >
> >
> >
> >
> >
> >
> >The IETF Secretariat
> 
> _______________________________________________
> ippm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ippm

---

From: Steve Baillargeon <[email-masked]>
Date: Fri, 1 Feb 2013 16:04:14 +0000

Hi Al
I suggest to add Christofer Flinta as a co-author of the draft-morton-ippm-twamp-rate. 
I believe his inputs are relevant and he can help out jump start those changes on the draft.

What do you think?

-Steve

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Christofer Flinta
Sent: February-01-13 9:57 AM
To: MORTON JR., ALFRED (AL); [email-masked]
Subject: Re: [ippm] Fwd: New Version Notification for draft-morton-ippm-twamp-rate-02.txt

Hi Al,

Cross traffic          0     25    50    75
APC                        97     72    47    22

Christofer

-----Original Message-----
From: MORTON JR., ALFRED (AL) [mailto:[email-masked]]
Sent: den 25 januari 2013 00:27
To: Christofer Flinta; [email-masked]
Subject: RE: [ippm] Fwd: New Version Notification for draft-morton-ippm-twamp-rate-02.txt

Hi Christofer,

Al

---

From: Greg Mirsky <[email-masked]>
Date: Fri, 1 Feb 2013 09:37:23 -0800

Dear Steve,
protocol's flexibility is important and valuable quality but we should
remember that performance measurements in connectionless packet switched
network can not reliably characterize performance, e.g. available
bandwidth, of a path between any pair of end-points. PM in connectionless
PSN characterizes network, not the path that being traversed by customer
traffic at given time. I'm not sure that 24/7 PM in IP PSN is useful tool
for network operators. It would be great to hear operators before we
proceed to adding more complexity to the protocol.

Regards,
Greg

On Sun, Jan 27, 2013 at 10:32 AM, sab <[email-masked]>wrote:

> Hi Al
> I agree with the changes proposed by Christofer. They will improve the
> protocol and its usability.
>
> Measuring the available path capacity may not be important to end users but
> it is definitively an important metric for the operators especially when it
> is measured 24/7.
>
> As I indicated in the Quebec meeting, the protocol should be flexible
> enough
> to accomodate any spacing between test packets in a burst in both forward
> and reverse directions. We should not worry about cost here. It is up to
> the
> implementer to decide the appropriate spacing and the balance between
> benefits and costs.
>
> I don't think we need to worry about LMAP as well. LMAP has it is own set
> of
> objectives.  As far as I know, we started the rate discussion much before
> LMAP and the TWAMP protocol should not ignore this use case.
>
> Regards
> Steve Baillargeon
>
>
>
> --
> View this message in context:
> http://ietf.10.n7.nabble.com/Fwd-New-Version-Notification-for-draft-morton-ippm-twamp-rate-02-txt-tp250601p354625.html
> Sent from the IETF - ippm mailing list archive at Nabble.com.

---

From: sab <[email-masked]>
Date: Fri, 1 Feb 2013 10:56:45 -0800 (PST)

Hi Greg
I am glad you agree on the need for protocol flexibility. This is really
important.
It actually reduces the total complexity in the long run e.g. avoid the need
to add another mode Y because mode X cannot do it.

I don’t know about you but we have reliably characterize path performance
for specific pair of endpoints.
Many operators have adopted on-going path performance monitoring especially
with hub-and-spoke topologies where the number of paths are limited or more
or less well-understood.

-Steve

--
View this message in context: http://ietf.10.n7.nabble.com/Fwd-New-Version-Notification-for-draft-morton-ippm-twamp-rate-02-txt-tp250601p355144.html
Sent from the IETF - ippm mailing list archive at Nabble.com.

---

From: <[email-masked]>
Date: Tue, 5 Feb 2013 09:38:55 +0100

Thanks, Greg.

I'm not sure I understand the use case of measuring path capacity by an active system as a provider.
This is relevant for which kind of network and which path bandwidth's? IP core networks today are
based on 10, 40 or 100 Gbit/s links. Pathes in these networks aren't always known.

Regards,

Rüdiger

________________________________
From: [email-masked] [mailto:[email-masked]] On Behalf Of Greg Mirsky
Sent: Friday, February 01, 2013 6:37 PM
To: sab
Cc: [email-masked]
Subject: Re: [ippm] Fwd: New Version Notification for draft-morton-ippm-twamp-rate-02.txt

Dear Steve,
protocol's flexibility is important and valuable quality but we should remember that performance measurements in connectionless packet switched network can not reliably characterize performance, e.g. available bandwidth, of a path between any pair of end-points. PM in connectionless PSN characterizes network, not the path that being traversed by customer traffic at given time. I'm not sure that 24/7 PM in IP PSN is useful tool for network operators. It would be great to hear operators before we proceed to adding more complexity to the protocol.

Regards,
Greg

On Sun, Jan 27, 2013 at 10:32 AM, sab <[email-masked]<mailto:[email-masked]>> wrote:
Hi Al
I agree with the changes proposed by Christofer. They will improve the
protocol and its usability.

As I indicated in the Quebec meeting, the protocol should be flexible enough
to accomodate any spacing between test packets in a burst in both forward
and reverse directions. We should not worry about cost here. It is up to the
implementer to decide the appropriate spacing and the balance between
benefits and costs.

I don't think we need to worry about LMAP as well. LMAP has it is own set of
objectives.  As far as I know, we started the rate discussion much before
LMAP and the TWAMP protocol should not ignore this use case.

Regards
Steve Baillargeon

--
View this message in context: http://ietf.10.n7.nabble.com/Fwd-New-Version-Notification-for-draft-morton-ippm-twamp-rate-02-txt-tp250601p354625.html
Sent from the IETF - ippm mailing list archive at Nabble.com.
_______________________________________________
ippm mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/ippm

---

From: Steve Baillargeon <[email-masked]>
Date: Thu, 28 Feb 2013 18:00:01 +0000

Hi Al
I have 3 comments about draft-morton-ippm-twamp-rate-02.txt.

1- In Section 4 where you are listing the MUST for the test protocol, you are missing variable intra-packet spacing in the pair, ensemble or train

2- In section 5, I would like to see a requirement on the test protocol for the two-way architecture where it can measure the rate in a specific transmission direction using a single Type-P test session where the test session has its own test data stream and also measure rates in both forward and reverse transmission directions using a single Type-P test session where the forward direction data stream is also used to measure the reverse direction rate. 

3-  In section 3, you indicate the two-way measurement architecture is favored and I agree. You also indicate that each direction of transmission must be evaluated separately and I agree. However this paragraph does not clarify the meaning of test streams vs test sessions. If you imply that each test stream requires its own test session, then I don't agree this is a good architectural decision since in many access networks following a hub-and-spoke topology, I expect the spoke to be the low complexity device (as you have indicated) and the hub to be the controller where the test sessions are configured, activated/deactivated and where data is probably collected. With In-Service testing including 24/7 monitoring, one challenge is the number of concurrent test sessions supported at the hub/controller. It is often the limiting factor for scaling the measurement system. Therefore it is desirable to include the case where a single test session can measure both the forward and reverse rates for a given Type-P packet. If you don't, you could end-up with twice as many test sessions than actually needed. 

Regards
Steve B

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Al Morton
Sent: den 3 september 2012 15:57
To: [email-masked]
Subject: [ippm] Fwd: New Version Notification for draft-morton-ippm-twamp-rate-02.txt

IPPM,

We've updated this draft to coordinate with the problem statement as it now stands.

Al and Len

>Status:          http://datatracker.ietf.org/doc/draft-morton-ippm-twamp-rate
>Htmlized:        http://tools.ietf.org/html/draft-morton-ippm-twamp-rate-02
>Diff: