Subject: [lmap] Scope, Use Cases and Users, and Which are Mandatory vs. Nice to Have

From: "STARK, BARBARA H" <[email-masked]>
Date: Thu, 8 Nov 2012 18:41:10 +0000

Following are my suggestions for problem scope, listing of use cases that have been mentioned to date, and my thoughts on which we need to focus on solving.

Proposed Scope: Performance measurements for mass market broadband connectivity services.

Use Cases 
1. Regulators want performance data for "mass market" broadband access network connectivity services. Regulators are the primary user in this use case. Regulators may have business requirements that involve access by others to the data collected for this use case (e.g., researchers are to be allowed access to data, or broadband customers should be allowed to access data for their own service), but these others are not considered primary users in this case.

2. Researchers want to be able to run and collect data from a variety of tests performed against broadband connections (end-to-end, segment by segment, maybe or maybe not limited to mass market or the access network). Researchers are the primary user.

3. Broadband "mass market" customers want to run tests against their broadband connections and see results of tests performed against their broadband connection. Regulators, researchers, and customers are the 3 sets of "users" that are covered by these 3 use cases. Broadband customers are the primary user.

4. ISPs want tests to assist in troubleshooting, and network architecting and engineering. [Note that I'm just listing this as a use case that has been mentioned, and not as something that is being requested by all ISPs -- no value judgment is implied by listing it here.] ISPs are the primary user.

Which Use Cases Are Mandatory
Based on last night's discussion, I believe that only the first use case is mandatory. The others are fine to consider, if they don't get in the way of solving use case #1.

If we can agree on these points, then I think the next step is to identify the Business Requirements of regulators. Ideally, we would get input from regulators around the world. We certainly need to try to solicit their input. However, I think it's a good idea to start by documenting the business requirements of the one regulator who we do have. Getting started may incite others to join in. Other "users" (or proxies of other users) are also welcome to provide their business requirements -- but they need to understand that satisfying their requirements is considered "nice to have" and not mandatory.

I'm going to work next on putting together a start for Business Requirements.
Barbara

---

From: Steve Baillargeon <[email-masked]>
Date: Thu, 8 Nov 2012 19:58:49 +0000

Hi 
I believe use case #4 is critical, at least for mobile broadband operators.
I would like to see an architecture that can accommodate both cases as much as possible including operator-initiated active measurements.
In theory, the metrics measured and reported by regulators/customers should be "consistent" with the metrics measured and collected by operators.

-Steve

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of STARK, BARBARA H
Sent: November-08-12 1:41 PM
To: [email-masked]
Subject: [lmap] Scope, Use Cases and Users, and Which are Mandatory vs. Nice to Have

Use Cases

I'm going to work next on putting together a start for Business Requirements.
Barbara
_______________________________________________
lmap mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/lmap

---

From: Marc Linsner <[email-masked]>
Date: Thu, 08 Nov 2012 15:38:57 -0500

Barbara,

Thanks for documenting the use cases below. I would like to submit use
case #5.

5. OTT providers of services.  OTT providers are interested in the QOE of
their customers and may instanciate a measurement client/controller/server
of their own accord.

-Marc-

-----Original Message-----
From: "STARK, BARBARA H" <[email-masked]>
Date: Thursday, November 8, 2012 1:41 PM
To: "[email-masked]" <[email-masked]>
Subject: [lmap] Scope, Use Cases and Users, and Which are Mandatory vs.
Nice to Have

>Following are my suggestions for problem scope, listing of use cases that
>have been mentioned to date, and my thoughts on which we need to focus on
>solving.
>
>1. Regulators want performance data for "mass market" broadband access
>network connectivity services. Regulators are the primary user in this
>use case. Regulators may have business requirements that involve access
>by others to the data collected for this use case (e.g., researchers are
>to be allowed access to data, or broadband customers should be allowed to
>access data for their own service), but these others are not considered
>primary users in this case.
>
>2. Researchers want to be able to run and collect data from a variety of
>tests performed against broadband connections (end-to-end, segment by
>segment, maybe or maybe not limited to mass market or the access
>network). Researchers are the primary user.
>
>3. Broadband "mass market" customers want to run tests against their
>broadband connections and see results of tests performed against their
>broadband connection. Regulators, researchers, and customers are the 3
>sets of "users" that are covered by these 3 use cases. Broadband
>customers are the primary user.
>
>4. ISPs want tests to assist in troubleshooting, and network architecting
>and engineering. [Note that I'm just listing this as a use case that has
>been mentioned, and not as something that is being requested by all ISPs
>-- no value judgment is implied by listing it here.] ISPs are the primary
>user.
>
>Based on last night's discussion, I believe that only the first use case
>is mandatory. The others are fine to consider, if they don't get in the
>way of solving use case #1.
>
>If we can agree on these points, then I think the next step is to
>identify the Business Requirements of regulators. Ideally, we would get
>input from regulators around the world. We certainly need to try to
>solicit their input. However, I think it's a good idea to start by
>documenting the business requirements of the one regulator who we do
>have. Getting started may incite others to join in. Other "users" (or
>proxies of other users) are also welcome to provide their business
>requirements -- but they need to understand that satisfying their
>requirements is considered "nice to have" and not mandatory.
>

---

From: <[email-masked]>
Date: Thu, 8 Nov 2012 21:01:15 +0000

similar to #1 
Operators want performance data for "mass market" broadband access network connectivity services.

One difference between the use cases is the timescale of interest. For example regulators are probably more interested in monthly averages over large numbers of users whereas in #4 ISPs are probably more interested in finer grain information.

Barbara,
I'm interested in why you want to narrow the focus to #1 (is it that your sceptical about the importance of the other use cases, or that they don't need standardisation, or something else?)

thanks
phil

________________________________________
From: [email-masked] [[email-masked]] On Behalf Of Marc Linsner [[email-masked]]
Sent: 08 November 2012 20:38
To: STARK, BARBARA H; [email-masked]
Subject: Re: [lmap] Scope, Use Cases and Users, and Which are Mandatory vs. Nice to Have

Barbara,

-Marc-

-----Original Message-----
From: "STARK, BARBARA H" <[email-masked]>
Date: Thursday, November 8, 2012 1:41 PM
To: "[email-masked]" <[email-masked]>
Subject: [lmap] Scope, Use Cases and Users, and Which are Mandatory vs.
Nice to Have

---

From: "STARK, BARBARA H" <[email-masked]>
Date: Thu, 8 Nov 2012 21:15:41 +0000

> I would like to see an architecture that can accommodate both cases as much
> as possible including operator-initiated active measurements.
> In theory, the metrics measured and reported by regulators/customers
> should be "consistent" with the metrics measured and collected by
> operators.
> 
> -Steve

<bhs> Hmm. While I agree that operators consider collecting such metrics to be important, I disagree that this should be a mandatory-to-consider driver for this "lmap" architecture.

On our wireline side, we already have an architecture for managing devices, running tests, collecting metrics, and measurement data collection. We don't need lmap to define something different for us.
I'm not an expert on our wireless architecture, but I haven't heard any requests from that side, for lmap to define an architecture for them. I do know that we do extensive mobile device management using OMA-DM. And I know that OMA-DM data models include elements for tests and some performance metric data collection. I know that if we need OM-DM extended to cover other data elements, that we have people capable of making this happen.

I'm aware of separate (from lmap) standards work that some operators are pursuing to improve their performance measurement capabilities. I completely support such efforts. And I believe that they are distinct from lmap and that their requirements to not, in any way, impose requirements on lmap.

I'm aware that there are operators who aren't already collecting metrics and do no device management. For these -- if they find the regulator-required metrics to be useful, that's great. If they want more than what the regulators want, then I consider trying to make sure that lmap meets their needs to be optional. 

But though I think it's optional, I fully support bringing the requirements in for us to consider. I think it's great to have people provide business requirements for all of the use cases, and not just the regulator use case. If you're able to provide business requirements for the operator use case that fit in well with the regulator requirements, then I may well support their inclusion in lmap. Optional doesn't mean that they can't be included. But if they don't fit in well, I, for one, will consider them to be unnecessary for lmap.
Barbara

> have been mentioned to date, and my thoughts on which we need to focus
> on solving.
> 
> network connectivity services. Regulators are the primary user in this use
> case. Regulators may have business requirements that involve access by
> others to the data collected for this use case (e.g., researchers are to be
> allowed access to data, or broadband customers should be allowed to access
> data for their own service), but these others are not considered primary
> users in this case.
> 
> 2. Researchers want to be able to run and collect data from a variety of tests
> performed against broadband connections (end-to-end, segment by
> sets of "users" that are covered by these 3 use cases. Broadband customers
> are the primary user.
> 
> 4. ISPs want tests to assist in troubleshooting, and network architecting and
> engineering. [Note that I'm just listing this as a use case that has been
> mentioned, and not as something that is being requested by all ISPs -- no
> value judgment is implied by listing it here.] ISPs are the primary user.
> 
> Based on last night's discussion, I believe that only the first use case is
> mandatory. The others are fine to consider, if they don't get in the way of
> solving use case #1.
> 
> If we can agree on these points, then I think the next step is to identify the
> Business Requirements of regulators. Ideally, we would get input from
> regulators around the world. We certainly need to try to solicit their input.
> However, I think it's a good idea to start by documenting the business
> requirements of the one regulator who we do have. Getting started may
> incite others to join in. Other "users" (or proxies of other users) are also
> welcome to provide their business requirements -- but they need to
> understand that satisfying their requirements is considered "nice to have"
> and not mandatory.
>

---

From: "STARK, BARBARA H" <[email-masked]>
Date: Thu, 8 Nov 2012 22:00:48 +0000

To be painfully blunt -- 
Measurements for regulatory purposes are a cost. They generate no revenue.
We heard regulators telling us yesterday that they want to mandate operators provide them with performance measurements.
I need to know what the minimum set of requirements is that an operator would need to comply with, in order to meet the regulatory requirements. The goal is to minimize the cost of the regulation. 

If the goals of other parties can be met without adding cost to the regulatorily-mandated architecture, then that's wonderful, and I'm happy to accommodate them. If the needs of others add cost to the mandated architecture, then that's very, very bad. To me, this means that all other use cases are optional. 

As I mentioned in my other email, we do extensive measurements today. We do this to increase customer satisfaction (which decreases churn and keeps customers loyal) and to decrease help desk and other operational costs, by having effective test mechanisms and data that allows us to do proactive network maintenance and management. But we don't need lmap to help us with this. If others do, ok, but it better not add to the cost of the solution that the regulators are going to make operators do to meet regulatory mandates.
Barbara

> From: [email-masked] [mailto:[email-masked]]
> 
> similar to #1
> One difference between the use cases is the timescale of interest. For
> example regulators are probably more interested in monthly averages over
> large numbers of users whereas in #4 ISPs are probably more interested in
> finer grain information.
> 
> Barbara,
> I'm interested in why you want to narrow the focus to #1 (is it that your
> sceptical about the importance of the other use cases, or that they don't
> need standardisation, or something else?)
> 
> thanks
> phil
> 
> Thanks for documenting the use cases below. I would like to submit use case
> #5.
> 
> 5. OTT providers of services.  OTT providers are interested in the QOE of their
> customers and may instanciate a measurement client/controller/server of
> their own accord.
> 
> -Marc-
> 
> >Following are my suggestions for problem scope, listing of use cases
> >that have been mentioned to date, and my thoughts on which we need to
> >focus on solving.
> >
> >by others to the data collected for this use case (e.g., researchers
> >are to be allowed access to data, or broadband customers should be
> >allowed to access data for their own service), but these others are not
> >considered primary users in this case.
> >
> >2. Researchers want to be able to run and collect data from a variety
> >of tests performed against broadband connections (end-to-end, segment
> >by segment, maybe or maybe not limited to mass market or the access
> >4. ISPs want tests to assist in troubleshooting, and network
> >architecting and engineering. [Note that I'm just listing this as a use
> >case that has been mentioned, and not as something that is being
> >requested by all ISPs
> >-- no value judgment is implied by listing it here.] ISPs are the
> >primary user.
> >
> >Based on last night's discussion, I believe that only the first use
> >case is mandatory. The others are fine to consider, if they don't get
> >in the way of solving use case #1.
> >

---

From: <[email-masked]>
Date: Fri, 9 Nov 2012 09:02:29 +0000

To give a different perspective, I think that #4 (including both fine per line measurement and overall sampling of products for aggregate network performance and benchmarking against other ISP/products) is a critical use case.

Some ISPs will benefit, others may not. I's not that all regulators in use case #1 will want to (or be able to) mandate that this measurement has to take place either.

While there are many currently deployed measurement techniques - deploying and managing tests, for example to home CPE, is not standardised. A harmonised way to run end-to-end tests that can be better indicative of user QoE is lacking. A way of detecting performance problems at any point and at any layer in diverse networks is lacking.

Furthermore if the ISP manages or owns much of this equipment, then it is vital that they are able to get benefit from running tests on these devices. In my view if the ISP is not the primary beneficiary (along with ultimately the broadband user) of this activity then these tests will never get deployed on a large scale but will always be restricted to limited and costly panels.

Trevor.

>Sent: 08 November 2012 22:01
>To: Eardley,PL,Philip,DUB8 R; [email-masked]; [email-masked]
>We heard regulators telling us yesterday that they want to mandate
>operators provide them with performance measurements.
>I need to know what the minimum set of requirements is that an operator
>would need to comply with, in order to meet the regulatory requirements.
>The goal is to minimize the cost of the regulation.
>
>If the goals of other parties can be met without adding cost to the
>regulatorily-mandated architecture, then that's wonderful, and I'm happy to
>accommodate them. If the needs of others add cost to the mandated
>architecture, then that's very, very bad. To me, this means that all other use
>cases are optional.
>
>As I mentioned in my other email, we do extensive measurements today. We
>do this to increase customer satisfaction (which decreases churn and keeps
>customers loyal) and to decrease help desk and other operational costs, by
>having effective test mechanisms and data that allows us to do proactive
>network maintenance and management. But we don't need lmap to help us
>with this. If others do, ok, but it better not add to the cost of the solution
>that the regulators are going to make operators do to meet regulatory
>mandates.
>Barbara
>
>case
>their
>> >Proposed Scope: Performance measurements for mass market
>broadband
>> >connectivity services.
>> >
>> >Use Cases

---

From: Brian Trammell <[email-masked]>
Date: Fri, 9 Nov 2012 07:14:33 -0500

hi Trevor,

+1

I get that the way some companies account for such things may make this sound terribly naïve, but I think the approach I'd take here is if the deployment of measurement infrastructure within the network will be mandated in order to allow regulators to to verify that operators follow laws and regulations pertinent to delivered network performance, thereby forcing a cost on network operators, then it makes sense to explore whether this same infrastructure can be used to other (potentially profitable) purposes, thereby at least recovering some of the capital outlay therefor.

Now, if we end up in a situation that it is _clear_ that the regulatory use cases are far simpler than any useful troubleshooting use case, or any value-added "measure my network" use case, such that a regulatory infrastructure would be measurably cheaper than one capable of multiple uses, then I would buy the argument that these should be treated separately.

But that just doesn't seem plausible to me. Unless the regulatory use case is scoped to be severely restricted (say, one trivially computable metric with a tiny population sample, which would IMO be questionably useful), the basic metrics (measurement point capabilities), analyses (collector and post-collector capabilities), data volumes and deployment patterns (system scale) seem to me to be pretty much the same. (The only thing from the LMAP draft architecture I don't see as universally applicable to all of these is the network parameters server, but it's clear from earlier list discussion that a lot of people seem to be confused about what that's for in the first place.)

Indeed, I'd look at this whole argument from the other side: is it possible to define a useful reference architecture and protocol for measurement infrastructures which have as their _primary_ use the within-the-ISP and from-the-customer-POV troubleshooting use cases, which is _also_ useful for regulatory verification purposes, which is incrementally implementable, such that when the mandate does come down, the regulatory requirements can be met by infrastructure already being deployed for operations?

Best regards,

Brian

---

From: "STARK, BARBARA H" <[email-masked]>
Date: Fri, 9 Nov 2012 16:06:42 +0000

> I get that the way some companies account for such things may make this
> sound terribly naïve, but I think the approach I'd take here is if the
> deployment of measurement infrastructure within the network will be
> mandated in order to allow regulators to to verify that operators follow laws
> and regulations pertinent to delivered network performance, thereby
> forcing a cost on network operators, then it makes sense to explore whether
> this same infrastructure can be used to other (potentially profitable)
> purposes, thereby at least recovering some of the capital outlay therefor.
> 
> Now, if we end up in a situation that it is _clear_ that the regulatory use cases
> are far simpler than any useful troubleshooting use case, or any value-added
> "measure my network" use case, such that a regulatory infrastructure would
> be measurably cheaper than one capable of multiple uses, then I would buy
> the argument that these should be treated separately.
> 
> But that just doesn't seem plausible to me. 

As I said, I think understanding the needs of others is good, and *if* the needs of others can be accommodated without adding complexity, then we should do that. So as long as we allow for this to be a decision factor, and allow that a provider who only wants to meet the requirements of the regulator must be *allowed* to only meet those requirements, and should not be *required* to implement something that meets the needs of all others, in order to satisfy a regulatory mandate.

If we can agree on this, I suggest we proceed with identifying the business requirements of the various use cases so we have something to actually assess as to what it does or does not imply.

After assessing business requirements and identifying the functions they imply -- if there is a consensus desire to define the mega-"satisfies everyone"-functional-architecture and create various needed piece-parts, I wouldn't stand in the way of this, as long as it is very clear which are the parts that the regulator has an interest in mandating ("function Y must be accomplished by doing X"), vs. being a regulatory "don't care". If the regulator doesn't care how function Y is accomplished, and 20 different orgs define 20 different ways of accomplishing Y, then a provider should be free to choose which way to do Y. If the IETF wants to be one of those 20, and the IETF's solution makes sense to a provider (e.g., because it allows for all use cases in a single cohesive architecture and is brilliant in its design), then the provider can certainly choose that as their approach. But it should be understood that it would not be a regulatory mandate to select that approach.
Barbara

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Fri, 9 Nov 2012 17:23:46 +0100

On Fri, Nov 09, 2012 at 04:06:42PM +0000, STARK, BARBARA H wrote:

> After assessing business requirements and identifying the functions
> they imply -- if there is a consensus desire to define the
> mega-"satisfies everyone"-functional-architecture and create various
> needed piece-parts, I wouldn't stand in the way of this, as long as
> it is very clear which are the parts that the regulator has an
> interest in mandating ("function Y must be accomplished by doing
> X"), vs. being a regulatory "don't care". If the regulator doesn't
> care how function Y is accomplished, and 20 different orgs define 20
> different ways of accomplishing Y, then a provider should be free to
> choose which way to do Y. If the IETF wants to be one of those 20,
> and the IETF's solution makes sense to a provider (e.g., because it
> allows for all use cases in a single cohesive architecture and is
> brilliant in its design), then the provider can certainly choose
> that as their approach. But it should be understood that it would
> not be a regulatory mandate to select that approach.

Barbara, I think we heard your concerns. That said, lets also take
into account that we talk likely about multiple regulators if we take
a more global perspective and the regulators might agree on a single
set of requirements globally (not too likely if you ask me but
possible) or slightly different overlapping sets (perhaps more
likely). This is perhaps clear but I thought I spell this out since
people always write about _the_ regulator (as if it is only the FCC).

/js

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1, 28759 Bremen, Germany
Fax:   +49 421 200 3103         <http://www.jacobs-university.de/>

---

From: "Bugenhagen, Michael K" <[email-masked]>
Date: Mon, 12 Nov 2012 15:48:59 +0000

Just a FYI -

   The Broadband forum scope, and Project requirements are actually in the liaison they sent over in March this year when that project started.
Just as a reminder I'll ask them liaison the latest draft they have to the group as well.

These would need to be identified as the scope as an earlier poster noted.

https://datatracker.ietf.org/liaison/1179/

- Mike

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Brian Trammell
Sent: Friday, November 09, 2012 6:15 AM
To: [email-masked]
Cc: [email-masked]; [email-masked]; [email-masked]; [email-masked]
Subject: Re: [lmap] Scope, Use Cases and Users, and Which are Mandatory vs. Nice to Have

hi Trevor,

+1

Best regards,

Brian

On 9 Nov 2012, at 4:02, [email-masked] wrote:

>> I need to know what the minimum set of requirements is that an 
>> operator would need to comply with, in order to meet the regulatory requirements.
>> regulatorily-mandated architecture, then that's wonderful, and I'm 
>> happy to accommodate them. If the needs of others add cost to the 
>> mandated architecture, then that's very, very bad. To me, this means 
>> that all other use cases are optional.
>> 
>> As I mentioned in my other email, we do extensive measurements today. 
>> We do this to increase customer satisfaction (which decreases churn 
>> and keeps customers loyal) and to decrease help desk and other 
>> operational costs, by having effective test mechanisms and data that 
>> allows us to do proactive network maintenance and management. But we 
>> don't need lmap to help us with this. If others do, ok, but it better 
>> not add to the cost of the solution that the regulators are going to 
>> make operators do to meet regulatory mandates.
>> Barbara
>> 
>>> One difference between the use cases is the timescale of interest. 
>>> For example regulators are probably more interested in monthly 
>>> averages over large numbers of users whereas in #4 ISPs are probably 
>>> more interested in finer grain information.
>>> 
>>> Barbara,
>>> I'm interested in why you want to narrow the focus to #1 (is it that 
>>> your sceptical about the importance of the other use cases, or that 
>>> they don't need standardisation, or something else?)
>>> 
>>> thanks
>>> phil
>>> 
>>> customers and may instanciate a measurement client/controller/server 
>>> of their own accord.
>>> 
>>> -Marc-
>>> 
>>>> Following are my suggestions for problem scope, listing of use 
>>>> cases that have been mentioned to date, and my thoughts on which we 
>>>> need to focus on solving.
>>>> 
>>>> network connectivity services. Regulators are the primary user in 
>>>> this use case. Regulators may have business requirements that 
>>>> involve access by others to the data collected for this use case 
>>>> (e.g., researchers are to be allowed access to data, or broadband 
>>>> customers should be allowed to access data for their own service), 
>>>> but these others are not considered primary users in this case.
>>>> 
>>>> 2. Researchers want to be able to run and collect data from a 
>>>> variety of tests performed against broadband connections 
>>>> (end-to-end, segment by segment, maybe or maybe not limited to mass 
>>>> market or the access network). Researchers are the primary user.
>>>> 
>>>> 3. Broadband "mass market" customers want to run tests against 
>>>> their broadband connections and see results of tests performed 
>>>> against their broadband connection. Regulators, researchers, and 
>>>> customers are the 3 sets of "users" that are covered by these 3 use 
>>>> cases. Broadband customers are the primary user.
>>>> 
>>>> architecting and engineering. [Note that I'm just listing this as a 
>>>> use case that has been mentioned, and not as something that is 
>>>> being requested by all ISPs
>>>> case is mandatory. The others are fine to consider, if they don't 
>>>> get in the way of solving use case #1.
>>>> 
>>>> identify the Business Requirements of regulators. Ideally, we would 
>>>> get input from regulators around the world. We certainly need to 
>>>> try to solicit their input. However, I think it's a good idea to 
>>>> start by documenting the business requirements of the one regulator 
>>>> who we do have. Getting started may incite others to join in. Other 
>>>> "users" (or proxies of other users) are also welcome to provide 
>>>> their business requirements -- but they need to understand that 
>>>> satisfying their requirements is considered "nice to have" and not mandatory.
>>>>

---

From: Rolf Winter <[email-masked]>
Date: Mon, 12 Nov 2012 17:20:58 +0100

Hi,

I really wonder what exactly you mean with business requirements here 
and what the IETF can or should do about those. Could you spell some of 
these requirements out?

Best,

Rolf

---

From: "STARK, BARBARA H" <[email-masked]>
Date: Mon, 12 Nov 2012 18:32:12 +0000

> I really wonder what exactly you mean with business requirements here and
> what the IETF can or should do about those. Could you spell some of these
> requirements out?

Yes. I've been trying to write down what I think I heard as being the basic "business" requirements coming from Henning and James at the IETF.
I haven't managed to write it up into something formal, but here are some high-level statements:

Regulators must be able to 
1)  specify what tests get run; the set of tests needs to cover all access technologies and allow "apples to apples" comparison of different access network providers, the technologies they use, and among results of different customers; the set of tests is expected to change over time (but hopefully not too quickly, since some tests may require considerable lead time to ensure that test infrastructure is in place, and to understand any performance/security risks that the tests my incur); note that different regulators may choose to require different sets of tests.
2) ensure that there is sufficient sampling of customers (of access network providers) to ensure coverage of all access network providers, geographies, and access technologies, where the sample size is large enough to ensure anonymity of results and limit the effect of statistical variation
3) be confident that data needed to provide meaningful test results on a per-customer level will exist, be collected, and collated 
4) have per-customer anonymous test results be available to the regulator and researchers

The first item implies to me that there is a need for a set of recommended tests. The set of tests will drive requirements not only for endpoints within the customer premises network, but also for test endpoints in the access network or beyond.
The second item, to me, looks less like an architectural element and more like a recommendation for what constitutes an appropriate sample size. 
The third implies to me that access network providers will need to be able to ensure that this is the case.
The fourth implies to me that access network providers will need to make sure that they have some way of making results available, and that the results are per-customer but anonymous.

So when I look at items 1, 3 and 4, I see that these cause access network providers to have requirements.
Not all access network providers are alike.
Some access network providers may want to have a 100% proprietary test control and management and internal data collection architecture.
Some access network providers may want to re-use their existing TR-069/TR-143/IPDR/OMA-DM/SNMP/etc. architectures.
Some access network providers may want to put a new box (like SamKnows) or an app (on a PC/smart phone/USB stick/etc.) in the customer premises network, and have the option of doing this themselves or contracting it out to a 3rd party.
Some access network providers may want a new architecture that perhaps re-uses some existing architectural elements, but maybe adds a new "control plane" protocol. Some may want to use a proprietary control plane protocol, and some may want to get together with other access network providers and standardize such a protocol.

If I have accurately represented the James/Henning "business requirements" that I heard, then I see nothing in them that would prohibit any of the access network provider options that I've listed. Which would make me happy. On the other hand, if a regulator is coming forward and saying "I have a requirement that the architecture for providing metrics must use a specific control plane protocol", then I'm concerned; because it would prohibit all architectures that do not use it. If an access network provider is coming forward and saying "I want a new control plane protocol to exist because I want to meet the regulatory requirements with an architecture that makes use of it", then I'm happy to support them (well, if they can get other providers to also want it -- I dislike creating "standards" that are only used by one provider). I'm also happy if the regulators express a desire to help the providers achieve this standardized control plane protocol.  I just get worried when the regulators state that a new standardized control plane protocol *must* exist to meet the regulatory use case.

Anyway, from "business requirement" #1, I see the need for a set of tests to be identified. It's clear that some PHY/access technologies and other special interest groups and standards orgs are interested in providing input to this set. It's also clear that the IETF has expertise in Layer 3 and above metrics.  

I've heard some access network providers say that they want a new control plane protocol for the architecture they want to use to meet #3 and #4. The access providers who want such a protocol should be free to have this protocol worked in whatever organization they wish.

As it relates to other use cases (customer wants to run tests, application service provider wants to run tests, service provider wants test environment to meet a variety of non-regulatory needs), I need to leave it up to the proponents of those use cases to provide the business requirements.
Barbara

---

From: Shane Amante <[email-masked]>
Date: Mon, 12 Nov 2012 21:24:24 -0700

Barbara,

> Some access network providers may want a new architecture that perhaps re-uses some existing architectural elements, but maybe adds a new "control plane" protocol. Some may want to use a proprietar…

First, I think the correct term to use, above, is: "management protocol" or OAM protocol, not "control plane protocol".  Examples of the former in the IETF are: SNMP, NETCONF.  In the IETF, I think if/when people see "control plane" protocol, they will instantly assume you mean: BGP, OSPF, LDP, etc.  Sorry to be the "word police" :-).

Second, I think it's important to clarify your comments in light of "straw man" architectural proposal in draft-schulzrinne-lmap-requirements-00.  I only say this because, IMO, I think that provides a decent starting point to ensure we share the same terminology and points-of-reference in this dialogue.  Specifically, it sounds like you're solely (?) focused on the "measurement client" in the above, correct?

> If I have accurately represented the James/Henning "business requirements" that I heard, then I see nothing in them that would prohibit any of the access network provider options that I've listed. Which would make me happy. On the other hand, if a regulator is coming forward and saying "I have a requirement that the architecture for providing metrics must use a specific control plane protocol", then I'm concerned; because it would prohibit all architectures that do not use it. If an access network provider is coming forward and saying "I want a new control plane protocol to exist because I want to meet the regulatory requirements with an architecture that makes use of it", then I'm happy to support them (well, if they can get other providers to also want it -- I dislike creating "standards" that are only used by one provider). I'm also happy if the regulators express a desire to help the providers achieve this standardized control plane protocol.  I just get worried when the 
> regulators state that a new standardized control plane protocol *must* exist to meet the regulatory use case.

I would point out a few things:
1)  I don't think anyone has stated the "must" word, even lower-case :-) -- but, I would welcome being corrected if you believe that to be the case.  Furthermore, regulators can't state things in the IETF or in any IETF documents, since the IETF is a global standards organization and regulations differ (widely) all across the world.
2)  Second, I believe the main thrust of the present straw man proposal is to point out the need for a _unified_ methodology in which to conduct network performance measurements on a much broader scale than exists, at the moment.  Personally, I read the straw man more as a "call to action" than any specific, detailed proposal of "a solution", but maybe that's me.
3)  Finally, without an objective set of requirements it will be difficult to impossible to objectively evaluate existing protocols, which you mention above, to see if all of them (or only some of them or none of them) will wholly satisfy or only partially satisfy the needs outlined in draft-schulzrinne.  IMO, I think it's premature to assume that there can only exist a single OAM protocol, at this point.  Obviously, from an Engineering PoV, I think many would agree that there are benefits to a single OAM protocol (development is focused on one thing); however, there are also downsides, as well.  Namely, that it *will* take quite a bit of time to specify in the IETF, procure and longer still to get adopted.  Thus, if it's /feasible/ to fit most/all the requirements incrementally into an existing protocol(s), then time to develop, procure and deploy is decreased (considerably).  Regardless, I think those are questions to be answered when evaluating the solution space against a widely agreed upon set of objective requirements, but we're not there yet.

In light of the above, it would be helpful to characterize your concerns in light of draft-schulzrinne-lmap-requirements-00, since that's the only proposal on the table at the moment.  If there is text you can cite from that draft that concerns you, it would be helpful to get it clarified, ask to get it *out* of that draft or whatever.  :-)  If you really want, focus just on Section 6, "Requirements", which is a mere 3 pages long, at this point.  :-)

-shane

---

From: "Bugenhagen, Michael K" <[email-masked]>
Date: Tue, 13 Nov 2012 16:04:44 +0000

Shane - 

   The only requirement I am fairly clear on is that some of the ISP's *(US based at least)  are required to produce performance results per speed tier for Customers.   This was verbally mentioned in the BAR BOF, but I didn't look for it in the draft.  

  So from a written requirements priority view I would think defining the what those PM's are, and how to perform the PM's, along with how to report the PM (Validating / cleaning the data) are key.

Regards,
Mike

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Shane Amante
Sent: Monday, November 12, 2012 10:24 PM
To: STARK, BARBARA H
Cc: Rolf Winter; [email-masked]
Subject: Re: [lmap] Scope, Use Cases and Users, and Which are Mandatory vs. Nice to Have

Barbara,

On Nov 12, 2012, at 11:32 AM, "STARK, BARBARA H" <[email-masked]> wrote:
>> and what the IETF can or should do about those. Could you spell some 
>> of these requirements out?
> 
> 2) ensure that there is sufficient sampling of customers (of access 
> network providers) to ensure coverage of all access network providers, 
> geographies, and access technologies, where the sample size is large 
> enough to ensure anonymity of results and limit the effect of 
> statistical variation
> 3) be confident that data needed to provide meaningful test results on 
> a per-customer level will exist, be collected, and collated

> If I have accurately represented the James/Henning "business 
> requirements" that I heard, then I see nothing in them that would 
> prohibit any of the access network provider options that I've listed. 
> Which would make me happy. On the other hand, if a regulator is coming 
> forward and saying "I have a requirement that the architecture for 
> providing metrics must use a specific control plane protocol", then 
> I'm concerned; because it would prohibit all architectures that do not 
> use it. If an access network provider is coming forward and saying "I 
> want a new control plane protocol to exist because I want to meet the 
> regulatory requirements with an architecture that makes use of it", 
> then I'm happy to support them (well, if they can get other providers 
> to also want it -- I dislike creating "standards" that are only used 
> by one provider). I'm also happy if the regulators express a desire to 
> help the providers achieve this standardized control plane protocol.  
> I just get worried when th
 e 

I would point out a few things:
1)  I don't think anyone has stated the "must" word, even lower-case :-) -- but, I would welcome being corrected if you believe that to be the case.  Furthermore, regulators can't state things in the IETF or in any IETF documents, since the IETF is a global standards organization and regulations differ (widely) all across the world.
2)  Second, I believe the main thrust of the present straw man proposal is to point out the need for a _unified_ methodology in which to conduct network performance measurements on a much broader scale than exists, at the moment.  Personally, I read the straw man more as a "call to action" than any specific, detailed proposal of "a solution", but maybe that's me.
3)  Finally, without an objective set of requirements it will be difficult to impossible to objectively evaluate existing protocols, which you mention above, to see if all of them (or only some of them or none of them) will wholly satisfy or only partially satisfy the needs outlined in draft-schulzrinne.  IMO, I think it's premature to assume that there can only exist a single OAM protocol, at this point.  Obviously, from an Engineering PoV, I think many would agree that there are benefits to a single OAM protocol (development is focused on one thing); however, there are also downsides, as well.  Namely, that it *will* take quite a bit of time to specify in the IETF, procure and longer still to get adopted.  Thus, if it's /feasible/ to fit most/all the requirements incrementally into an existing protocol(s), then time to develop, procure and deploy is decreased (considerably).  Regardless, I think those are questions to be answered when evaluating the solution space against a
  widely agreed upon set of objective requirements, but we're not there yet.

-shane

---

From: Rolf Winter <[email-masked]>
Date: Tue, 13 Nov 2012 17:13:52 +0100

Hi,

if you have to report them today, then it should be trivial to define 
them as I assume they have already been defined. I would expect that at 
least some of the other questions should have an answer, too? Is there 
official information on those measurements available? I mean not the 
data itself but what is requested by the regulator (not just US but 
other countries as well)?

Best,

Rolf

On 11/13/12 5:04 PM, Bugenhagen, Michael K wrote:
> 3)  Finally, without an objective set of requirements it will be difficult to impossible to objectively evaluate existing protocols, which you mention above, to see if all of them (or only some of them or none of them) will wholly satisfy or only partially satisfy the needs outlined in draft-schulzrinne.  IMO, I think it's premature to assume that there can only exist a single OAM protocol, at this point.  Obviously, from an Engineering PoV, I think many would agree that there are benefits to a single OAM protocol (development is focused on one thing); however, there are also downsides, as well.  Namely, that it *will* take quite a bit of time to specify in the IETF, procure and longer still to get adopted.  Thus, if it's /feasible/ to fit most/all the requirements incrementally into an existing protocol(s), then time to develop, procure and deploy is decreased (considerably).  Regardless, I think those are questions to be answered when evaluating the solution space agains!
 t a

---

From: "Weil, Jason" <[email-masked]>
Date: Wed, 14 Nov 2012 11:43:36 -0500

I'll attempt to aggregate feedback on the draft as well as the Use Case
discussion in a single email.

Scope:

Proposed Scope: Performance measurements for mass market broadband
connectivity services.

Suggested Tweak:Large-scale Performance Measurement for broadband data
services
Rationale - I don't think we should use the term mass market as it seems
to me to imply only to large operators. Small operators (arguably not mass
market) should be inclusive of the architecture as well.

SECTION 2 -Use Case/s:

I believe the 'Provider network measurements' Use Case in
draft-schulzrinne-lmap-requirements-00 is itself generic enough to vet the
requirements needed in the overall architecture. The other use cases
presented seem to be a subset of this first imo. Whether the end user, the
service provider or a 3rd party has access to the data stored in the
collector feels like a system policy question. I believe we are putting
too much emphasis on the regulatory requirement in the use case
discussion.

We should also capture the different network segments as this goes into
the full quality of experience discussion included in this use case
description. For example, the current measurement systems do a good good
at measuring the access link capacity, but not a good job extending beyond
that to the application source. A method for comparing application
performance by region would be very useful for measuring the end-to-end
customer experience and the measurement servers sitting in the path would
be perfect for measuring the segment between application provider and
network midpoint.

Section 3 - Architecture

In general I would refer to the items in this list as functional units as
opposed to actual hardware "boxes" which is what it sounds like here. A
physical device may have a number of these functional items included such
as a measurement collector in response to a downstream clietn and a
measurement client to perform upstream testing to an app servers.

3.5 - As others have mentioned I am not sure these needs to exist. A
simple API to embedded systems could provide this information.

SECTION 4 - Protocols:

Measurement client to measurement controller - Need to add wording that
the measurement controller may also return the actual test code/applet to
run on the client as well as or as part of the firmware update
periodically required on the measurement client.

SECTION 6 - Requirements:

What is alluded to but currently missing in the current version of the
draft is a requirement for the active measurement testing to not
negatively impact the Network under test. This is a key feature in the
Measurement controller functional requirements that needs to be included
from day one.

Proposed C-6: The measurement controller MUST ensure that the test
schedule it provides to measurement clients in its domain is generated in
a manner that minimizes aggregate test load on the measured network.

Another topic missing from the measurement controller discussion is the
concept of topology awareness. In order to meet the requirement in C-6
above, a measurement controller requirement should include some form of
network topology awareness in order to schedule tests in a controlled and
deterministic manner.

Proposed C-7: The measurement controller SHOULD have some awareness of the
network topology for the set of measurement clients it is responsible for
administering.

And if we assume that there will be multiple measurement controllers that
need to interoperate, then we also need some form of hierarchy such that
one controller can be elected master for some subset of test nodes
(possibly based on geography or administrative domain.) Other controllers
could then request test scheduling from the elected master controller.

Thanks,

Jason

On 11/12/12 11:24 PM, "Shane Amante" <[email-masked]> wrote:

>>>and
>> Yes. I've been trying to write down what I think I heard as being the
>>basic "business" requirements coming from Henning and James at the IETF.
>> 1)  specify what tests get run; the set of tests needs to cover all
>>access technologies and allow "apples to apples" comparison of different
>>access network providers, the technologies they use, and among results
>>of different customers; the set of tests is expected to change over time
>>(but hopefully not too quickly, since some tests may require
>>considerable lead time to ensure that test infrastructure is in place,
>>and to understand any performance/security risks that the tests my
>>incur); note that different regulators may choose to require different
>>sets of tests.
>> The first item implies to me that there is a need for a set of
>>recommended tests. The set of tests will drive requirements not only for
>>endpoints within the customer premises network, but also for test
>>endpoints in the access network or beyond.
>> The second item, to me, looks less like an architectural element and
>>more like a recommendation for what constitutes an appropriate sample
>>size.
>> The fourth implies to me that access network providers will need to
>>make sure that they have some way of making results available, and that
>>the results are per-customer but anonymous.
>>
>> Some access network providers may want to have a 100% proprietary test
>>control and management and internal data collection architecture.
>> Some access network providers may want to re-use their existing
>>TR-069/TR-143/IPDR/OMA-DM/SNMP/etc. architectures.
>> Some access network providers may want to put a new box (like SamKnows)
>>or an app (on a PC/smart phone/USB stick/etc.) in the customer premises
>>network, and have the option of doing this themselves or contracting it
>>out to a 3rd party.
>> Some access network providers may want a new architecture that perhaps
>>re-uses some existing architectural elements, but maybe adds a new
>>"control plane" protocol. Some may want to use a proprietary control
>>plane protocol, and some may want to get together with other access
>>network providers and standardize such a protocol.
>
>First, I think the correct term to use, above, is: "management protocol"
>or OAM protocol, not "control plane protocol".  Examples of the former in
>the IETF are: SNMP, NETCONF.  In the IETF, I think if/when people see
>"control plane" protocol, they will instantly assume you mean: BGP, OSPF,
>LDP, etc.  Sorry to be the "word police" :-).
>
>Second, I think it's important to clarify your comments in light of
>"straw man" architectural proposal in
>draft-schulzrinne-lmap-requirements-00.  I only say this because, IMO, I
>think that provides a decent starting point to ensure we share the same
>terminology and points-of-reference in this dialogue.  Specifically, it
>sounds like you're solely (?) focused on the "measurement client" in the
>above, correct?
>
>
>>providing metrics must use a specific control plane protocol", then I'm
>>concerned; because it would prohibit all architectures that do not use
>>it. If an access network provider is coming forward and saying "I want a
>>new control plane protocol to exist because I want to meet the
>>regulatory requirements with an architecture that makes use of it", then
>>I'm happy to support them (well, if they can get other providers to also
>>want it -- I dislike creating "standards" that are only used by one
>>provider). I'm also happy if the regulators express a desire to help the
>>providers achieve this standardized control plane protocol.  I just get
>>worried when th
> e
>1)  I don't think anyone has stated the "must" word, even lower-case :-)
>-- but, I would welcome being corrected if you believe that to be the
>case.  Furthermore, regulators can't state things in the IETF or in any
>IETF documents, since the IETF is a global standards organization and
>regulations differ (widely) all across the world.
>2)  Second, I believe the main thrust of the present straw man proposal
>is to point out the need for a _unified_ methodology in which to conduct
>network performance measurements on a much broader scale than exists, at
>the moment.  Personally, I read the straw man more as a "call to action"
>than any specific, detailed proposal of "a solution", but maybe that's me.
>3)  Finally, without an objective set of requirements it will be
>difficult to impossible to objectively evaluate existing protocols, which
>you mention above, to see if all of them (or only some of them or none of
>them) will wholly satisfy or only partially satisfy the needs outlined in
>draft-schulzrinne.  IMO, I think it's premature to assume that there can
>only exist a single OAM protocol, at this point.  Obviously, from an
>Engineering PoV, I think many would agree that there are benefits to a
>single OAM protocol (development is focused on one thing); however, there
>are also downsides, as well.  Namely, that it *will* take quite a bit of
>time to specify in the IETF, procure and longer still to get adopted.
>Thus, if it's /feasible/ to fit most/all the requirements incrementally
>into an existing protocol(s), then time to develop, procure and deploy is
>decreased (considerably).  Regardless, I think those are questions to be
>answered when evaluating the solution space against a
>In light of the above, it would be helpful to characterize your concerns
>in light of draft-schulzrinne-lmap-requirements-00, since that's the only
>proposal on the table at the moment.  If there is text you can cite from
>that draft that concerns you, it would be helpful to get it clarified,
>ask to get it *out* of that draft or whatever.  :-)  If you really want,
>focus just on Section 6, "Requirements", which is a mere 3 pages long, at
>this point.  :-)
>
>-shane
>
>
>
>
>> Anyway, from "business requirement" #1, I see the need for a set of
>>tests to be identified. It's clear that some PHY/access technologies and
>>other special interest groups and standards orgs are interested in
>>providing input to this set. It's also clear that the IETF has expertise
>>in Layer 3 and above metrics.
>>
>> I've heard some access network providers say that they want a new
>>control plane protocol for the architecture they want to use to meet #3
>>and #4. The access providers who want such a protocol should be free to
>>have this protocol worked in whatever organization they wish.
>>
>> As it relates to other use cases (customer wants to run tests,
>>application service provider wants to run tests, service provider wants
>>test environment to meet a variety of non-regulatory needs), I need to
>>leave it up to the proponents of those use cases to provide the business
>>requirements.
>> Barbara

This E-mail and any of its attachments may contain Time Warner Cable proprietary information, which is privileged, confidential, or subject to copyright belonging to Time Warner Cable. This E-mail is intended solely for the use of the individual or entity to which it is addressed. If you are not the intended recipient of this E-mail, you are hereby notified that any dissemination, distribution, copying, or action taken in relation to the contents of and attachments to this E-mail is strictly prohibited and may be unlawful. If you have received this E-mail in error, please notify the sender immediately and permanently delete the original and any copy of this E-mail and any printout.

---

From: Rolf Winter <[email-masked]>
Date: Wed, 14 Nov 2012 20:20:58 +0100

Hi,

I would suggest to replace "mass market" with "residential" IFF we want 
to reduce scope. I would assume however that many things that will be 
developed for this will be broadly applicable and useful in other 
settings too. We might have to develop metrics of interest and maybe 
develop standard measurement techniques and parameters, however I'd urge 
to make the protocol mechanics as generic and extensible as possible. 
Remember RFC 5218 (of cause having the cautious considerations of RFC 
6709 in mind).

Best,

Rolf

---

From: "STARK, BARBARA H" <[email-masked]>
Date: Wed, 14 Nov 2012 22:30:14 +0000

> Suggested Tweak:Large-scale Performance Measurement for broadband
> data services Rationale - I don't think we should use the term mass market as
> it seems to me to imply only to large operators. Small operators (arguably not
> mass market) should be inclusive of the architecture as well.

Just to explain -- I was trying to distinguish consumer and small business "mass market" offerings from large business and enterprise offering that often have SLAs associated with them. Many small operators also target what I consider to be mass market customers, so I did see those as being included. The testing done for SLA customers is pretty large-scale, so I'm worried that "large-scale" might be considered to include them? But I'm happy with alternate wordings, as long as we agree that we aren't trying to tackle testing for SLA customers. Or, if we are, that should be specifically called out, since it drives additional use cases and additional functionality.
Barbara