Subject: [lmap] comments on draft-schulzrinne-lmap-requirements

From: "STARK, BARBARA H" <[email-masked]>
Date: Wed, 14 Nov 2012 22:20:28 +0000

Shane suggested that the draft is a good start and asked me to provide comments against it. I've been avoiding doing that, because I'm not very fond of this draft, and am not sure if it can be "fixed", or if it should be completely overhauled, or totally scrapped. But others apparently do like it, so let me explain some of my objections to the current draft.

Section 2 Use Cases
This section could probably be fixed.
The last sentence of the Provider network measurements use case should be deleted, as it goes beyond use case, to describe how elements of this draft's proposed architecture would be used to meet the use case. I would recommend changing the title and "user" of the use case to "Access Network Providers", since the use case is about the needs of Access Network Providers for measuring their own network. Note that it is possible for an ISP not to be an access network provider. This is not as prevalent as it once was, but may still be a useful distinction in some parts of the world. The measurement needs of an ISP who does not own the access network may call for them to get measurement data from the access network provider. I believe that case is covered by the ASP case (below).

User network diagnostics I would recommend renaming to just "End User" (since it seems to be concerned with the end user's own network, and the end user's subscribed broadband connection), and I would prefer to see running on-demand tests as a distinct use case from accessing historical measurements (relating to themselves). I see that the end user seems to have an interest in testing their own network, the access provider connection, and end-to-end testing against an ASP. Trying to describe all of these within a single architecture may be challenging, scope-wise.

Multi-provider network measurements is a mixture of a variety of use cases that need to be split out. There is a use case of Application Service Provider (ASP), who wants to tests against that ASP's customers. There is a separate use case of regulatory entities. I can't tell if the researcher use case is distinct from the regulator use case or is covered by it. The ASP (which for me covers the access-reseller ISP) use case may also suggest the need for testing within their own networks, which could be very extensive (depending on their network). If ASPs have a need for on-demand testing, as well as large-scale testing, then I would prefer to see these use cases separated. Trying to describe the internal architecture for ASP network testing in the same architecture as access provider (internal or external) testing could be challenging, scope-wise. Including on-demand with large-scale use cases could also be challenging.

The statement that "it should be understood that a data collection platform must serve multiple stakeholder interests" is problematic to me, because it suggests that an access network provider may not be permitted to deploy data collection platforms that collect data intended only for use by that access network provider. This statement suggests that any platform that an access network provider deploys must also meet the needs of others. This sentence should either be removed or reworded (and moved to a paragraph where it would apply to all use cases) along the lines of "The data collection architecture should identify and define the (internal and external, or just external/interworking?) capabilities needed to satisfy the interests of all (some subset?) of these use cases." Defining an architecture that describes functionality for meeting the needs of use cases is very different from insisting that a deployment of a measurement platform must meet the needs of all stakeholders / use cases. Whether or not to deploy or include a function should be a business decision. A well-defined functional architecture will provide the ability to make this business decision, but will not attempt to mandate the business decision.

Section 3
I see 2 very big problems with the architecture.
 1) It is very much focused on the access provider's internal capabilities. It seems to try to generalize some of the architectural elements to be not necessarily an element of the access provider, but that just confuses things for me; because the requirements are very much geared towards assuming that the architecture elements are under the control of the access provider. For the most part, the draft defines one particular way in which the access provider could (must) architect running tests and collecting data internal to their network, and provide others with access to the test capabilities and resulting data. This is not the architecture I would recommend to the company I work for. As such, I don't think it's all that interesting. Nor do I consider defining such an architecture to be a necessary step before identifying what the appropriate set of tests may be for doing access network performance measurements to satisfy regulatory needs (or internal needs or customer trouble-shooting needs). Quite frankly, I'm not particularly interested in getting the IETF to help with defining an internal architecture for me to recommend. I'm not particularly interested in getting any org to do that; but if I were, there are probably other orgs where I would go first, because they have greater expertise in the idiosyncrasies of my employer's access network technologies and existing management framework. With that said -- if there are access providers who do want IETF to help them with this, I have no problem with that, and I might even be willing to help. But if there are no access providers (or their proxies) asking for IETF to define this internal architecture for them, then I would recommend against the IETF insisting on defining such an architecture. I've been witness to way too many IETF discussions of the sort "I want services providers to do X and they don't want to do X. Those bad, horrible, evil service providers. We need to find some way to force them to do X." I did read the disclaimer about the draft expressing personal views, and I'm willing to trust that this is truly the case. But the fact that the authors of the draft are employees of an agency that *does* have the ability to force access providers to do X, together with the language of the draft, leaves me incredibly uneasy. I hope that this work does not devolve into some attempt by non-access providers to force business models, business decisions, and specific architectures on access providers, with subsequent access provider bashing when it becomes clear that not all access providers are willing to actually deploy (all aspects of) the architecture. 
 2) The described architecture requires that many different functional components be present in single architectural elements. I prefer architectures that define distinct functional components separately, and allow for the functional components to be physically placed where others feel it makes sense for them. It's ok to provide examples of how functions *could* exist in a single physical box, but I generally find that *mandating* physical combinations within a functional architecture isn't a very good approach.

Here are some specific comments on the architectural elements:
Measurement client: I agree that if measurements (active or passive) are going to be made, then one or more Measurement clients are a mandatory element. I agree with the first 3 sentences of the Measurement client definition. But the idea that *all* Measurement clients must be manageable and "provide for communications within different network segments" is not something I agree with. It seems to be implied that there is exactly one Measurement client for any given broadband customer. I disagree with that. A good performance measurement architecture tends to consist of multiple Measurement clients, each with different measurement capabilities and interfaces. The various Measurement clients may also come from or be "owned" by various entities. The requirements section requires that this element communicates in certain ways with other architectural elements. I disagree that these are requirements of all Measurement clients. IMO, the only other element that a generic Measurement client is required to communicate with is the Measurement server, and that only if the client does active tests. If the Measurement client is intended to be an "Access provider managed measurement client", then this should be made explicit.

Active tests involve a Measurement server. I agree with the existence of such a Measurement server as an element of a performance measurement architecture that includes active tests. I have no problem with the definition of Measurement server. This does assume that some of the as-yet-to-be-identified tests will be active. But I'm ok with that assumption. The requirements section assumes the Measurement server includes much more functionality than this definition implies. I disagree that all Measurement servers must have such functionality. As with Measurement clients, I would expect there to be multiple Measurement servers, and they may or may not be dedicated to specific active tests, may or may not serve other function or have other interfaces, and are very generic architecture elements that may be "owned" by various entities. For example, I (personally) frequently use "www.google.com" as a Measurement server that I conduct active (ping, traceroute) tests against, using my PC OS "Command Prompt" interface to provide me with the client. I also conduct tests against this Measurement server using my router as the client, and interfacing to the router through a browser on my PC.

Measurement controller: This is not a necessary architectural element. This is an element specific to this proposed physical architecture. However, some of the external-facing functionality attributed to this element (in the requirements section) may be desirable to define in the context of a functional architecture, which attempts to understand the external-facing "interworking" functions that might be useful for accomplishing the various use cases. An example of this are the requirements around a function that allows 3rd parties to control and schedule tests to be run by elements under the control of the access network provider. If there are no access providers asking for assistance with design of their internal architectures, it might be good to focus on understanding these external interfaces, more than the internal ones. I do want to stress, though, that I consider it important to express functional requirements in such a way that it is clear "If this function is desired/implemented, then the function must/should..." as opposed to "The (access provider's) measurement architecture must provide this function."

Data collector: I do not see this as a necessary architectural element, either. As with the Measurement controller, it has been assigned the role of providing multiple, separable functions: collecting data, storing data, and providing access to the data.

Network parameter server: I do not see this as a necessary architectural element. If there is a need for an externally-facing function that provides access to access-provider-collected data for the regulatory use case to include corresponding (anonymous) information related to subscribed line rate, etc., then that is a requirement on that external-facing function. If an access provider wants help in figuring out how to get this information to the external-facing function, then I would prefer for that access provider (or their proxy) to ask for such help. 

If there is a desire to define an interworking function that would allow a 3rd party to get (not anonymous) customer-specific line information directly from the access provider (so the 3rd party could correlate it with data they collect from their own Measurement client(s)), then that would be a distinct function, as well. Personally, I would shy away from trying to define this function. But if someone else wants to tackle it, I might be willing to provide comments on it. 

Before I move on, I'll briefly summarize the external-facing "interworking" functions that I see the various use cases implying for an access network provider (Again, in the context of: If the access network provider wants to support a particular use case and implement the functions needed for it, then the requirements for those functions are that the function must/should...", and not "The access provider's architecture must...").
The Access Provider use case (measurements for internal access provider purposes) does not drive the need for any external interworking functions.
The End User on-demand or historical performance use case for their home network does not need external interworking functions from the access network provider.
The End User on-demand use case for testing of the access network (with tools from the access network provider) would require a user interface that allows the end user to launch user-specified tests against their own connection and view the results.
The End User historical performance use case for data from the access network provider would require a user interface that allows the end user to access historical performance metrics of their broadband connection. There seems to also be a requirement to allow a user to specify what tests to run (and parameters of those tests and schedule for running the tests).
The Regulator use case would require access to anonymous line/connection specific measurement data. It appears there are also requirements for a function that allows tests to be specified, manipulated, and scheduled by the regulator or regulator-approved 3rd parties.
The ASP use case for measurements without participation of an access provider would require no access provider interworking function.
The ASP use case for on-demand customer-specific testing with help from an access provider seem to require some function, but I'm not sure what it is. The requirements section seems to imply a function that would allow the ASP to request (from the access provider) on-demand and scheduled tests of the ASP's customers. And to be able to specify test parameters.
The ASP use case for access to large-scale historical performance data for their customers, with info from an access provider, does seem to require some function, but I'm not sure what it is. Again, there seems to be a requirement for the ASP to specify what tests get run, and test parameters.
There also seem to be some requirements around the formatting and timestamping of data provided by the access provider via the interworking functions.

Having said all of this, I don't think there's much sense in my commenting on specific requirements. I think the overall architecture is too flawed to try to deal with the specific flaws of each and every requirement. And I don't even know if there's an access provider interested in having the internal elements of this particular architecture properly specified.

If there's an access provider who wants help in designing their internal architecture, I think it would be nice if they spoke up and described their needs. Similarly, if someone else (e.g., ASPs, or those who speak on behalf of end users) wants help in designing an internal architecture (for testing of their private networks), it would be good to understand their needs. If there's a desire to identify and better understand/characterize interworking functions, then I'd be happy to participate in that. I'm not ready to agree to specifying detailed requirements of interworking functions. But if we can understand and characterize them, then we can make an informed decision regarding the need for detailed specification.
Barbara

---

From: Shane Amante <[email-masked]>
Date: Thu, 15 Nov 2012 14:54:44 -0700

Barbara,

If you don't mind, I'm going to attempt to separate this very detailed, but long :-), response out into separate threads to, hopefully, make the dialogue somewhat more manageable.

For now, I'm starting with your comments on Section 3.  Other threads will follow.

On Nov 14, 2012, at 3:20 PM, "STARK, BARBARA H" <[email-masked]> wrote:
[---snip---]
> 1) It is very much focused on the access provider's internal capabilities. It seems to try to generalize some of the architectural elements to be not necessarily an element of the access provider, but that just confuses things for me; because the requirements are very much geared towards assuming that the architecture elements are under the control of the access provider.

This seems like a good place to start, because I get the sense you may be concerned about another party (e.g.: regulator) taking over control of -- or dictating that Access Network Provider's install -- LMAP architectural elements on the Access Network Provider's network.  (Let me know if I'm misunderstanding).  My interpretation of the draft-schulzrinne-lmap-requirements was that LMAP was aiming more to build a cooperative 'federation' of these architectural elements.  However, I would agree that that interpretation, on my part, (or even the word "federation") does not appear to be in the draft.  Certainly either interpretation, but hopefully the latter one :-), should be clarified in a future revision of that draft, IMO.

IOW, at a very high-level, let's say that the ANP has deployed their own Measurement Controller.  And, a Regulator, or other third-party, has their own Measurement Controller.  One potential solution I could see emerging is that:
a)  Regulator's Measurement Controller asks ANP's Measurement Controller for a "token" or set of "tokens" to authenticate itself for some period of time to speak _directly_, using some IP-based protocol (NETCONF, who knows?), with a group of the ANP's Measurement Clients.
b)  Assuming the Regulator's Measurement Controller receives those tokens, then the Regulator's Measurement Controller speaks directly to the ANP's Measurement Clients to ask them to perform tests, presumably using some IP-based application protocol, (e.g.: SNMP, NETCONF, other?), to make the request and send the results back to regulator's Data Controller(s).

I would note that the 'elegance' of the above model is that the ANP is still "in control" insomuch as it needs to hand authentication tokens (and, IP addresses) of Measurement Clients before the third-party Measurement Controller can even ask the ANP's Measurement Clients to do anything.  No tokens == no access to clients.  One downside is that Measurement Clients would clearly have to support whatever "new", or _modified_, protocol it is that enables the Measurement Controller to Measurement Client communication to take place.  However, I'm personally *not* worried about that at this stage, because we're still talking about requirements.

Finally, I would note that the above is just _one_ illustration of a potential outcome in this space that could potentially satisfy the current "straw man" requirements.  I want to be clear that I'm not stating that the above is the _only_ potential solution, nor is it even be the best potential solution.  But, that's why we're here talking about this.  :-)

> For the most part, the draft defines one particular way in which the access provider could (must) architect running tests and collecting data internal to their network, and provide others with access to the test capabilities and resulting data. This is not the architecture I would recommend to the company I work for. As such, I don't think it's all that interesting. Nor do I consider defining such an architecture to be a necessary step before identifying what the appropriate set of tests may be for doing access network performance measurements to satisfy regulatory needs (or internal needs or customer troub
> le-shooting needs). Quite frankly, I'm not particularly interested in getting the IETF to help with defining an internal architecture for me to recommend. I'm not particularly interested in getting any org to do that; but if I were, there are probably other orgs where I would go first, because they have greater expertise in the idiosyncrasies of my employer's access network technologies and existing management framework. With that said -- if there are access providers who do want IETF to help them with this, I have no problem with that, and I might even be willing to help. But if there are no access providers (or their proxies) asking for IETF to define this internal architecture for them, then I would recommend against the IETF insisting on defining such an architecture. I've been witness to way too many IETF discussions of the sort "I want services providers to do X and they don't want to do X. Those bad, horrible, evil service providers. We need to find some way to force 
> them to do X." I did read the disclaimer about the draft expressing personal views, and I'm willing to trust that this is truly the case. But the fact that the authors of the draft are employees of an agency that *does* have the ability to force access providers to do X, together with the language of the draft, leaves me incredibly uneasy. I hope that this work does not devolve into some attempt by non-access providers to force business models, business decisions, and specific architectures on access providers, with subsequent access provider bashing when it becomes clear that not all access providers are willing to actually deploy (all aspects of) the architecture. 

Re: comments in the last two sentences, I'm not clear that the present draft was 'mandating' combinations of or placement of particular functions.  Rather, my read of the draft was that some functions or placement seemed to make intuitive sense, i.e.: the Measurement Client needs to be at the customer premise.

I did not detect a mandate on placement of the Measurement Servers in the draft.  I can see how the "Network Parameter Server" may be hosted on the ANP network, (because it happens to be an existing box/server they already have).  I can also see another potential answer, as well.  Namely, the "Network Parameter Server" may be a function/server hosted at a third-party (e.g.: regulator) that periodically receives a 'push' of updated data from the ANP's "Network Parameter Server"/system(s).  I can also see both functions co-existing at the same time with no harm, as well, e.g.: one ANP may choose one method and a second ANP may choose another ... both will work.  I would make the same comment about the "Data Collector".  With respect to the "Measurement Controller" and the case where the ANP owns/maintains the Measurement Client, its placement may have to be at both the third-party and the ANP, due to (to be defined?) needs of the ANP to authenticate requests, grant access to speak to Measurement Clients, etc.  This does not preclude the third-party Measurement Controller from, ultimately, speaking directly to a Measurement Client, but before the third-party Measurement Controller does so it needs to first ask the ANP's Measurement Controller "nicely".  :-)  At a high-level, this may be similar to how, for example, OAuth works in the Application Layer (third-party HTTPS authentication[1]).  I'm not suggesting that OAuth is the only or even best idea here, just that it's the closest analogy I can think of for how this may ultimately be realized.

-shane

[1] https://en.wikipedia.org/wiki/OAuth

---

From: Shane Amante <[email-masked]>
Date: Thu, 15 Nov 2012 19:10:35 -0700

Barbara,

Continuing the discussion specific to your comments on the Measurement Client.

[--snip--]
> Measurement client: I agree that if measurements (active or passive) are going to be made, then one or more Measurement clients are a mandatory element. I agree with the first 3 sentences of the Measurement client definition.

Well 3 out of 4 (sentences) is a good start. :-)

> But the idea that *all* Measurement clients must be manageable and "provide for communications within different network segments" is not something I agree with.

I'm not sure I detect an implication that all measurement clients must be manageable.  What the fourth sentence says is the following:
"Client measurement functionality must be implementable in a variety of user contexts ..."
                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
... at this point, I believe the above sentence is stating that as we think about developing solutions there may be various classes of "measurement client" devices, e.g.: CPE modem, WiFi AP's?, Home routers? (e.g.: D-Link, Netgear, Linksys, etc.), PC's, laptops, smartphones?, other?  Some of those devices are, naturally, going to be more resource-constrained (CPU, flash, DRAM) than others.  Thus, we need to either consider:
a) Designing a single solution for the lowest common denominator measurement client possible; and/or,
b) Designing "profiles" that various classes of measurement client can conform to.  For example, a "Basic profile" measurement client may only be able to do basic throughput, latency & loss measurements, etc.  An "Advanced profile" measurement client may be able to do all of the "Basic profile", plus fancier stuff like test for receipt of ICMPv6 Packet Too Big packets to ensure that Path MTU is working ... or something.  I'm sure there are much better examples one can come up with than that.  :-)

Perhaps we could ask the authors to modify the first part of that sentence as follows:
---snip---
There is expected to be a variety of measurement clients in the architecture.  Some classes of measurement clients may be more resource constrained than others in terms of CPU, DRAM and Flash memory.  The primary example are CPE modems.  Secondary examples are: WiFi Access Points, Home Routers/Switches, Personal Computers, Laptops, Tablets, Smartphones, etc.  Client measurement functionality must be implementable across various classes of devices, in a variety of user contexts ...
---snip---

> It seems to be implied that there is exactly one Measurement client for any given broadband customer. I disagree with that. A good performance measurement architecture tends to consist of multiple Measurement clients, each with different measurement capabilities and interfaces.

Hopefully, the additional context I provided in my above text may address the above?

> The various Measurement clients may also come from or be "owned" by various entities.

That's a really good point that the current text does not capture.  How about we ask for the following text to be folded in:
---snip---
Measurement clients may be owned and operated by an ISP and other measurement clients may be owned and operated by the ISPs customer, (e.g.: home user's PC or tablet).
---snip---

> The requirements section requires that this element communicates in certain ways with other architectural elements. I disagree that these are requirements of all Measurement clients. IMO, the only other element that a generic Measurement 
> client is required to communicate with is the Measurement server, and that only if the client does active tests.

Hrm.  I'm not sure I'd agree with the above.  How do you get the results, from a just completed performance measurement test, off the measurement client?  IOW, the measurement client will need to "push" its test results to a "data collector" function (server in "the cloud"), which presumably has lots of disk to store results?  Presumably if the CPE devices are resource constrained, then they won't be able to keep perf. measurement results around that long in DRAM, or Flash.

> If the Measurement client is intended to be an "Access provider managed measurement client", then this should be made explicit.

Perhaps we should not prematurely constrain the definition of the "measurement client" here in the architecture section, since this is supposed to be representative of all classes of measurement client.  Rather, we should look at (if necessary) incorporating your above sentence in the requirements section?  Or, perhaps, we end up with requirements per Use Case where that belongs?

Thanks,

-shane

---

From: Shane Amante <[email-masked]>
Date: Thu, 15 Nov 2012 20:02:37 -0700

Barbara,

Continuing the discussion specific to your comments on the Measurement Server.

> Active tests involve a Measurement server. I agree with the existence of such a Measurement server as an element of a performance measurement architecture that includes active tests. I have no problem with the definition of Measurement server. This does assume that some of the as-yet-to-be-identified tests will be active. But I'm ok with that assumption.

As am I.

> The requirements section assumes the Measurement server includes much more functionality than this definition implies.

Actually, it's funny you should mention this, because Section 6, Requirements, doesn't have a single requirement tagged with "S-*", which would presumably apply _specifically_ to the measurement server function.  :-)  But, there is this disclaimer, though: 
                                                         In many cases,
   a single requirement governs more than one entity or protocol, so the
   labeling should be considered rough.

Anyway, are the following requirements, the two that you find most concerning wrt the Measurement Server?
---snip---
   A-2:  Measurement clients and servers MUST support an extensible set
      of performance metrics.

   D-3:  The data exchange protocol between measurement server and data
      collector SHOULD allow the definition of common data elements,
      e.g., for network addresses and timestamps.
---snip---

> I disagree that all Measurement servers must have such functionality. As with Measurement clients, I would expect there to be multiple Measurement servers, and they may or may not be dedicated to specific active tests, may or may not serve other function or have other interfaces, and are very generic architecture elements that may be "owned" by various entities.

First, wrt Requirement A-2, I can see how a "MUST" may be concerning in light of the physical capabilities of a particular class/type of measurement server.  However, I read that requirement more in light of "The LMAP Architecture ...", not a specific implementation mandate.  Perhaps to avoid any doubt, maybe something like the following sentence needs to get added to the introduction paragraph to section 6:
---snip---
The following requirements are applicable to the overall LMAP Reference Architecture, as described in this document.  How each of these requirements are applied to specific implementation of a particular type, category or class of measurement client, server, etc. device is out-of-scope for this document.
---snip---
Is the above text good enough, for now?

Re: D-3, I would note that says "SHOULD", which should provide some flexibility in ensuring that future measurement servers need not adhere to this specific requirement.

But, going back to Section 3.2, Measurement Server, I can see how it may be useful to add some of the points you raised above.  How about suggesting to add something like the following to Section 3.2?
---snip---
Measurement servers likely will be owned and operated by a variety of administrative entities, some of which may be unrelated to the ISP providing service to a measurement client.  Therefore, measurement servers may need a facility where they can describe their capability to perform one, or more, suites of active performance measurement tests to a measurement controller. In this way, the measurement controller may facilitate coordination of measurement clients to measurement servers that have matching active performance measurement capabilities.  
---snip---

Thanks,

-shane

---

From: Shane Amante <[email-masked]>
Date: Thu, 15 Nov 2012 20:39:00 -0700

Barbara,

I'm still going ... on to the Measurement Controller. :-)

> Measurement controller: This is not a necessary architectural element. This is an element specific to this proposed physical architecture.

I'm unclear if your concern is related to:
1) having _strictly_ a single measurement controller be part of, or in control of a single measurement "panel" (e.g.: group of measurement clients & measurement server); or,
2) two, or more, measurement controllers (likely owned/operated by different parties) acting in concert to coordinate the rendezvous between an appropriate panel of measurement clients with measurement servers.  See my previous e-mail re: "Section 3" for an example of what I'm trying to describe here.

If neither of those are what concern you, can you point me in the right direction?  :-)

> However, some of the external-facing functionality attributed to this element (in the requirements section) may be desirable to define in the context of a functional architecture, which attempts to understand the external-facing "interworking" functions that might be useful for accomplishing the various use cases. An example of this are the requirements around a function that allows 3rd parties to control and schedule tests to be run by elements under the control of the access network provider. If there are no access providers asking for assistance with design of their internal architectures, it might be good to focus on understanding these external interfaces, more than the internal ones.

Good points.  That said, I would offer two points:
A)  We may also want to develop companion drafts/RFCs down the road that describe the "internal architectures" if only in the context of an Informational doc, so that (for example) developers and outside operators, of third-party measurement controllers, can see the larger picture and understand that when a third-party developed/owned/operated measurement controller 'calls' to ISP-owned/operated measurement controller they have a (high-level) picture of what's happening behind the scenes.
B)  It is conceivable that there may be measurement platforms (potentially encompassing everything from the measurement controller through measurement client) that are not owned or operated by the ISP.  In this case, I believe, we definitely do want to clearly specify what the requirements for standards-based interoperability of the measurement controller to measurement client.

> I do want to stress, though, that I consider it important to express functional requirements in such a way that it is clear "If this function is desired
> /implemented, then the function must/should..." as opposed to "The (access provider's) measurement architecture must provide this function."

Again, if we're talking about a broad set of requirements, applicable to a larger set of potential constituents I think it's premature to restrict the scope of applicability of the architectural elements, for now. 

-shane

---

From: Shane Amante <[email-masked]>
Date: Thu, 15 Nov 2012 21:03:39 -0700

Barbara,

OK, on to the Data Collector & Network Parameter Server.

> Data collector: I do not see this as a necessary architectural element, either. As with the Measurement controller, it has been assigned the role of providing multiple, separable functions: collect…

I'm not sure I understand your concern.  Given the resource constraints on some classes of measurement clients, doesn't it make sense for the measurement client to send it's performance measurement results to "the cloud" (a.k.a.: data collector) for longer-term archival, storage and centralized access by the ISP and/or 3rd-parties?

> Network parameter server: I do not see this as a necessary architectural element. If there is a need for an externally-facing function that provides access to access-provider-collected data for the…

I would think that if, say, an ISP already has a "network parameter server" they already own & operate, then that's fine ... they can continue to run it as they do today.  However, the key aspect we probably want to work on is defining a _standards-based_ "data model" (e.g.: XML or JSON schema?) that can encapsulate appropriate/relevant bits of data about subscribed lines in that (existing) Network Parameter Server.  This would allow for much more straightforward access to and parsing of "network parameters" by, for example, 'authorized parties' as the draft states (Section 3.4).

> If there is a desire to define an interworking function that would allow a 3rd party to get (not anonymous) customer-specific line information directly from the access provider (so the 3rd party co…

Hrm.  You raise a good point above about a 3rd-party acquiring "non-anonymous" customer line information from the access provider so that the 3rd-party could correlate it with data they collect from their own (3rd-party) measurement clients.  
1)  I'm not sure what, if any, technical mechanisms the IETF could develop to prevent such a thing from happening, other than to NOT specify any data-models/schemas that encapsulate such sensitive, non-anonymous information in a 'standards-based' data-model in the first place.
2)  That said, I would think that "privacy laws" in place in the U.S. and other parts of the world (perhaps most significantly, the EU) would prevent the ISPs from [ever] offering access to that data for fear of violating both the law and, more importantly, their customer's trust as well.

OK, I think I'm going to stop replying to this e-mail chain for now to: a) let you catch up; and, b) for me to get some sleep.  :-)

Thanks again,

-shane

---

From: <[email-masked]>
Date: Fri, 16 Nov 2012 09:26:45 +0000

Shane,

As you say your scenario of multiple measurement controllers is not the only way to achieve federation. Nor do I believe it is the most appropriate. 

I'm much more inclined to believe that an organisation (regulator, ISP or other) would deploy both their own clients and controller. Federation can be achieved by sharing the data - e.g. an individual user or ISP can share data for their line(s) with the regulator. For many purposes this data can be pre-processed (errors removed, limits/filters applied, lines anonymised, aggregate values calculated) that will be appropriate for different relationships.

The reasons I'm not convinced about multiple controllers:
- the test controller will gain many benefits by being integrated with other management systems - for example a home gateway management system. The device security can be managed much more easily. In addition the device and line characteristics gathered by such management systems can be used to tailor the test schedule and configuration. For example higher speed lines may run shorter throughput tests or older CPE may run lighter loads. Lines with boost capabilities can be set to run longer throughput tests to produce boost and off-boost values.
- Two controllers means that there is potential for conflict between the tests. How do we know we are not overloading the device? How do we assign priority between tests when they both want to run at the same time? Is the non-ISP party able to force a test schedule that would impact on a user experience? The ISP is responsible for maintaining the user experience, not the regulator.

Trevor Burbridge

>-----Original Message-----
>From: [email-masked] [mailto:[email-masked]] On Behalf Of
>Shane Amante
>Sent: 15 November 2012 21:55
>To: STARK, BARBARA H
>Cc: Rolf Winter; [email-masked]
>Subject: Re: [lmap] comments on draft-schulzrinne-lmap-requirements:
>Section 3
>
>Barbara,
[---snip---]
>IOW, at a very high-level, let's say that the ANP has deployed their own
>Measurement Controller.  And, a Regulator, or other third-party, has their
>own Measurement Controller.  One potential solution I could see emerging is
>that:
>a)  Regulator's Measurement Controller asks ANP's Measurement Controller
>for a "token" or set of "tokens" to authenticate itself for some period of time
>to speak _directly_, using some IP-based protocol (NETCONF, who knows?),
>with a group of the ANP's Measurement Clients.
>b)  Assuming the Regulator's Measurement Controller receives those tokens,
>then the Regulator's Measurement Controller speaks directly to the ANP's
>Measurement Clients to ask them to perform tests, presumably using some
>IP-based application protocol, (e.g.: SNMP, NETCONF, other?), to make the
>request and send the results back to regulator's Data Controller(s).
>
>I would note that the 'elegance' of the above model is that the ANP is still "in
>control" insomuch as it needs to hand authentication tokens (and, IP
>addresses) of Measurement Clients before the third-party Measurement
>Controller can even ask the ANP's Measurement Clients to do anything.  No
>tokens == no access to clients.  One downside is that Measurement Clients
>would clearly have to support whatever "new", or _modified_, protocol it is
>that enables the Measurement Controller to Measurement Client
>communication to take place.  However, I'm personally *not* worried about
>that at this stage, because we're still talking about requirements.
>
>Finally, I would note that the above is just _one_ illustration of a potential
>outcome in this space that could potentially satisfy the current "straw man"
>requirements.  I want to be clear that I'm not stating that the above is the
>_only_ potential solution, nor is it even be the best potential solution.  But,
>that's why we're here talking about this.  :-)

---

From: Capello Alessandro <[email-masked]>
Date: Fri, 16 Nov 2012 11:54:26 +0100

Hi all,

IMO IETF activities should focus only on identifying a set of abstract functions and on defining a flexible data model and extensible protocols (or choosing/adapting existing ones) to enable communications between these functions.

Basic function could be, for instance:
.       ability to perform a measurement
.       ability to collect measurement results from other entities and to perform aggregation/pre-elaboration
.       ability to trigger a specific measurement on a different entity
.       .

Some of these functions will be mandatory, other optional.
Also an agreement on metrics is necessary but without constraining how they are measured.

Any other specification (e.g. where the functions are placed, who owns them, etc.) is IMO out of scope for IETF and falls into the "over-specification" category, something I consider a mistake in general (just a personal opinion).

The draft actually contains some elements of over-specification, probably because it was written having a specific implementation in mind. But it also catches some important technical issues: lack of a proper data model and lack of standard interfaces/protocols for handling measurements. These technical issues should be solved without imposing unnecessary constraints in order to let anyone run its own business at best.

Regards,
Alessandro

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of [email-masked]
Sent: venerdì 16 novembre 2012 10.27
To: [email-masked]; [email-masked]
Cc: [email-masked]; [email-masked]
Subject: Re: [lmap] comments on draft-schulzrinne-lmap-requirements: Section 3

Shane,

Trevor Burbridge

[---snip---]
>that's why we're here talking about this.  :-)

_______________________________________________
lmap mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/lmap

Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle persone indicate. La diffusione, copia o qualsiasi altra azione derivante dalla conoscenza di queste informazioni sono rigorosamente vietate. Qualora abbiate ricevuto questo documento per errore siete cortesemente pregati di darne immediata comunicazione al mittente e di provvedere alla sua distruzione, Grazie.

This e-mail and any attachments is confidential and may contain privileged information intended for the addressee(s) only. Dissemination, copying, printing or use by anybody else is unauthorised. If you are not the intended recipient, please delete this message and any attachments and advise the sender by return e-mail, Thanks.

---

From: <[email-masked]>
Date: Fri, 16 Nov 2012 10:59:11 +0000

I agree.

However, that doesn't mean that we don't have to decide (a) if federation is required; and (b) how we would achieve it through a technical solution.

Trevor Burbridge

>From: Capello Alessandro [mailto:[email-masked]]
>Sent: 16 November 2012 10:54
>To: Burbridge,T,Trevor,DUB8 R; [email-masked]; [email-masked]
>Subject: RE: [lmap] comments on draft-schulzrinne-lmap-requirements:
>Section 3
>
>Hi all,
>
>IMO IETF activities should focus only on identifying a set of abstract
>functions and on defining a flexible data model and extensible protocols (or
>choosing/adapting existing ones) to enable communications between these
>functions.
>
>Any other specification (e.g. where the functions are placed, who owns them,
>etc.) is IMO out of scope for IETF and falls into the "over-specification"
>category, something I consider a mistake in general (just a personal opinion).
>
>The draft actually contains some elements of over-specification, probably
>because it was written having a specific implementation in mind. But it also
>catches some important technical issues: lack of a proper data model and
>lack of standard interfaces/protocols for handling measurements. These
>technical issues should be solved without imposing unnecessary constraints
>in order to let anyone run its own business at best.
>
>Regards,
>Alessandro
>
>
>[email-masked]
>As you say your scenario of multiple measurement controllers is not the only
>way to achieve federation. Nor do I believe it is the most appropriate.
>
>I'm much more inclined to believe that an organisation (regulator, ISP or
>other) would deploy both their own clients and controller. Federation can be
>achieved by sharing the data - e.g. an individual user or ISP can share data for
>their line(s) with the regulator. For many purposes this data can be pre-
>processed (errors removed, limits/filters applied, lines anonymised,
>aggregate values calculated) that will be appropriate for different
>relationships.
>
>- the test controller will gain many benefits by being integrated with other
>management systems - for example a home gateway management system.
>The device security can be managed much more easily. In addition the device
>and line characteristics gathered by such management systems can be used
>to tailor the test schedule and configuration. For example higher speed lines
>may run shorter throughput tests or older CPE may run lighter loads. Lines
>with boost capabilities can be set to run longer throughput tests to produce
>boost and off-boost values.
>- Two controllers means that there is potential for conflict between the tests.
>How do we know we are not overloading the device? How do we assign
>priority between tests when they both want to run at the same time? Is the
>non-ISP party able to force a test schedule that would impact on a user
>experience? The ISP is responsible for maintaining the user experience, not
>the regulator.
>
>Trevor Burbridge
>
>>IOW, at a very high-level, let's say that the ANP has deployed their
>>own Measurement Controller.  And, a Regulator, or other third-party,
>>has their own Measurement Controller.  One potential solution I could
>>see emerging is
>>that:
>>a)  Regulator's Measurement Controller asks ANP's Measurement
>>Controller for a "token" or set of "tokens" to authenticate itself for
>>some period of time to speak _directly_, using some IP-based protocol
>>(NETCONF, who knows?), with a group of the ANP's Measurement Clients.
>>b)  Assuming the Regulator's Measurement Controller receives those
>>tokens, then the Regulator's Measurement Controller speaks directly to
>>the ANP's Measurement Clients to ask them to perform tests, presumably
>>using some IP-based application protocol, (e.g.: SNMP, NETCONF,
>>other?), to make the request and send the results back to regulator's Data
>Controller(s).
>>
>>I would note that the 'elegance' of the above model is that the ANP is
>>still "in control" insomuch as it needs to hand authentication tokens
>>(and, IP
>>Controller can even ask the ANP's Measurement Clients to do anything.
>>No tokens == no access to clients.  One downside is that Measurement
>>Clients would clearly have to support whatever "new", or _modified_,
>>protocol it is that enables the Measurement Controller to Measurement
>>Client communication to take place.  However, I'm personally *not*
>>worried about that at this stage, because we're still talking about
>requirements.
>>
>>Finally, I would note that the above is just _one_ illustration of a
>>potential outcome in this space that could potentially satisfy the current
>"straw man"
>>requirements.  I want to be clear that I'm not stating that the above
>>is the _only_ potential solution, nor is it even be the best potential
>>solution.  But, that's why we're here talking about this.  :-)
>
>Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle
>persone indicate. La diffusione, copia o qualsiasi altra azione derivante dalla
>conoscenza di queste informazioni sono rigorosamente vietate. Qualora
>abbiate ricevuto questo documento per errore siete cortesemente pregati di
>darne immediata comunicazione al mittente e di provvedere alla sua
>distruzione, Grazie.
>
>This e-mail and any attachments is confidential and may contain privileged
>information intended for the addressee(s) only. Dissemination, copying,
>printing or use by anybody else is unauthorised. If you are not the intended
>recipient, please delete this message and any attachments and advise the
>sender by return e-mail, Thanks.

---

From: "STARK, BARBARA H" <[email-masked]>
Date: Fri, 16 Nov 2012 19:06:44 +0000

> Hi all,
> 
> IMO IETF activities should focus only on identifying a set of abstract functions
> and on defining a flexible data model and extensible protocols (or

<bhs> I think that identifying abstract functions is reasonable. I believe that whether or not data models or protocols are created to enable the functions is dependent on whether there exist entities who are willing and wanting to implement the functions (and not just entities who want to try to require others to implement). Protocols developed to enable functions that have no willing deployers aren't a very good idea.

> Also an agreement on metrics is necessary but without constraining how

<bhs> Some functions may be *conditionally* mandatory -- that is, if a particular use case is to be supported, then a particular function is necessary. I hope that there is no attempt within the IETF to impose use case support on any entity. And, just because a function is needed to support a use case, doesn't mean it has to be fully architected and defined. If no potential implementers of a use case see a compelling reason to support that use case, then proceeding with detailed definition of the functions needed for the use case is not something I would recommend. Similarly, if no potential implementers of a function that has no external interfaces are asking for assistance with defining the function, I would recommend against detailed work on that function. 

Barbara

---

From: "STARK, BARBARA H" <[email-masked]>
Date: Mon, 19 Nov 2012 17:25:36 +0000

> [---snip---]
> My interpretation of the draft-schulzrinne-lmap-
> requirements was that LMAP was aiming more to build a cooperative
> 'federation' of these architectural elements.  However, I would agree that
> that interpretation, on my part, (or even the word "federation") does not
> appear to be in the draft.  Certainly either interpretation, but hopefully the
> latter one :-), should be clarified in a future revision of that draft, IMO.

<bhs> But is such "federation" really needed? IMO, no. The complexity it creates around control and management and security (management of credentials, permissions, ensuring that all necessary safeguard are in place to prevent 3rd parties from negatively impacting a customer's service), with a need for execution environments in often simple client devices is considerable. I believe that insisting on the existence of such "federation" is overkill for the problem of providing metrics, measurements and tools sufficient for any of the base use cases (around enabling assessment of access networks and allow for troubleshooting and diagnosis of problems).

I believe that it is possible for a well-defined set of metrics to allow for identification of a reasonably static set of measurements and tests, that doesn't require constant updating of and changing of the Measurement Clients, or external management of the testing. While it is generally good to allow for evolution of the tests over time, I believe this can be accomplished in many cases through occasional firmware upgrades, handled directly by the device owner/manager, rather than a dynamic changing of test applications controlled by 3rd parties. I believe that good data can be provided to regulators and researchers without allowing them to control, schedule, or otherwise manage the tests. I believe consumers can be provided local control of their on-demand and scheduled (active) testing, that doesn't require all sorts of tokens and permissions and massive security, threat mitigation, and prevention of undesired impacts. I believe that 3rd parties who want to test with a specific customer, can work directly with that customer (and through the customer's local tools), and do not need direct access to Access Network Provider (ANP) elements.

I believe that focusing on identification of metrics would be much more beneficial to providing a solution for the basic use cases, rather than focusing on an architecture that requires massive security, new management and control interfaces, and client execution environments.

[---snip---]
> Rather, my read of the draft was that some functions or placement seemed
> to make intuitive sense, i.e.: the Measurement Client needs to be at the
> customer premise.

<bhs> In DSL environments, there is a single physical point-to-point connection between the customer premises and the DSLAM. For passive measurements, the DSLAM is a possible place to do passive measurement of DSL throughput, as well as some simple DSL/ATM/Ethernet-specific active tests. Even some simple active tests at IP (like a ping). In PON cases, the ONT at the side of the house or at the curb (for fiber to the curb) is not generally considered a part of the customer premises, but is an excellent place for doing passive (and some simple active-towards-the-access-network) tests. I disagree that the Measurement Client needs to be at or in the customer premises. Especially if ANP testing of their own network is part of the goal, or if regulator assessment of the end-to-end access network (which may have choke points further in) is part of the goal. Providing 3rd party control of testing within these network elements would be a very hard sell for the operators of these elements. From an ANP self-measuring and self-testing perspective, these are definitely Measurement Clients.
Barbara

---

From: "STARK, BARBARA H" <[email-masked]>
Date: Mon, 19 Nov 2012 18:58:40 +0000

[--snip--]
> > Measurement client: I agree that if measurements (active or passive) are
> > going to be made, then one or more Measurement clients are a mandatory
> > element. I agree with the first 3 sentences of the Measurement client
> > definition.
> 
> > But the idea that *all* Measurement clients must be manageable and
> > "provide for communications within different network segments" is not
> > something I agree with.
> 
> I'm not sure I detect an implication that all measurement clients must be
> manageable.  What the fourth sentence says is the following:

<bhs> As I mentioned, I thought it was possible to agree on the existence of a Measurement Client (as defined by the first 3 sentences) as an architectural element. The problem really arises with the assumptions as to the nature of this element that are implied by certain of the requirements:

   A-2:  Measurement clients and servers MUST support an extensible set
      of performance metrics.
   A-4:  Measurement clients MUST be able perform both active and
      passive measurements.
   M-3:  A measurement client MUST be able to register with the data
      collection platform automatically, announcing its availability and
      relevant system parameters.  (For example, a cable or DSL modem
      may indicate its make and model number.)
   M-4:  A measurement client MUST be able to declare what kind of
      measurements it can perform, e.g., by enumerating a set of
      measurement identifiers.
   C-4:  A measurement client MUST be able to authenticate and authorize
      the measurement controller.

As I mentioned in my last email, I believe Measurement Clients may exist inside ONTs, DSLAMs, and even very, very simple DSL modems (some of which are nothing more than a USB dongle that connects to a PC, and many of which are completely unmanaged, except locally by the end user). The idea that these devices / network elements must support these requirements in order to participate in a holistic scheme/architecture to collect meaningful performance data is something I do not agree with. I also believe that it is possible for a Management Client to be directly under the control of the end user, and potentially not be upgradable or capable of an extensible set of tests. I believe that the Windows Command Prompt is a very effective (for me) end-user-controlled Measurement Client for performing a variety of tests. But it does not meet the above requirements. Nor should it be expected to. I use it often, though, for my on-demand connectivity and performance testing.

To me, this architecture draft assumes that a "federated" architecture is necessary for useful collection of performance measurements and conducting tests. I disagree with that premise. There also seems to be an assumption that a "federated" architecture must be defined prior to defining appropriate metrics. I disagree. I believe that a federated architecture (that allows for all sorts of management and control and updating of tests/schedules/etc.) is unnecessary. I believe it is possible to re-use a lot of what exists, with minimal additions, to accomplish large scale performance testing and on-demand diagnostics, that meet the basic "needs" (not to be confused with "wants") of various parties. I believe it is possible for 3rd parties and ANPs to develop one-to-one relationships with an end user, to allow for on-demand and scheduled testing and collection of test data. The end user can even allow for 3rd parties to access tools under end-user control, which interact with ANP Measurement Servers (taking advantage of the end user to ANP relationship). For large scale performance data -- I believe it is possible to collect such data and provide access to the resulting anonymized data without enabling 3rd party management and control of the elements involved in data collection.

My recommendation would be to focus on defining appropriate metrics. Once we understand the metrics, we can work on understanding the appropriate measurements and tests, and the set of Measurement Clients and Measurements Servers that could be used to perform those measurements and tests. If something else is needed from a standards organization, that should become evident once the metrics are identified.
Barbara

---

From: "STARK, BARBARA H" <[email-masked]>
Date: Mon, 19 Nov 2012 21:52:04 +0000

> > The requirements section assumes the Measurement server includes
> > much more functionality than this definition implies.
> 
> Actually, it's funny you should mention this, because Section 6,
> Requirements, doesn't have a single requirement tagged with "S-*", which
> would presumably apply _specifically_ to the measurement server function.
> :-)  But, there is this disclaimer, though:
>                                                          In many cases,

<bhs> Those and 
   A-6:  All entities MUST be able to authenticate the entities they
      communicate with.
   D-3:  The data exchange protocol between measurement server and data
      collector SHOULD allow the definition of common data elements,
      e.g., for network addresses and timestamps.

This architecture seems to imply that it's relevant to the end user on-demand test case. My favorite test is "ping". I do my ping testing against a variety of Internet websites, but also my access/Internet provider's DNS servers and IP gateways. What I ping depends on what my problem is. Anything that responds to my ping is a Measurement Server in my book.

As an end user, I have no need for the set of tests that such Measurement Servers support.
I also have no need for my client to authenticate with them. I don't think they've ever required authentication of my client. I have no idea if they communicate with a Data Collector, and I really don't care.

Similarly, I often use "bandwidth test" websites for on-demand bandwidth tests. I really don't care whether they're extensible, authentication has never been an issue, and I have no requirement for them to communicate with a Data Collector

> ---snip---
> The following requirements are applicable to the overall LMAP Reference
> Architecture, as described in this document.  How each of these
> requirements are applied to specific implementation of a particular type,
> category or class of measurement client, server, etc. device is out-of-scope
> for this document.
> ---snip---

<bhs> No. Unless the described architectural elements are changed to be "LMAP Measurement Client" and "LMAP Measurement Server", with definition of these being that an LMAP Measurement Client/Server is one that meets the requirements enumerated in this document. I don't think it's possible to go from the generic architecture elements of Measurement Client/Server to the enumerated requirements that seem to have something very specific in mind that is not at all consistent with the generic definitions provided in the architecture section.
 
> But, going back to Section 3.2, Measurement Server, I can see how it may be
> useful to add some of the points you raised above.  How about suggesting to
> add something like the following to Section 3.2?
> ---snip---
> Measurement servers likely will be owned and operated by a variety of
> administrative entities, some of which may be unrelated to the ISP providing
> service to a measurement client.  Therefore, measurement servers may
> need a facility where they can describe their capability to perform one, or
> more, suites of active performance measurement tests to a measurement
> controller. In this way, the measurement controller may facilitate
> coordination of measurement clients to measurement servers that have
> matching active performance measurement capabilities.
> ---snip---

That still doesn't allow for a website that responds to a ping to be considered a Measurement Server. Or a website that provides bandwidth measurements. But they are Measurement Servers. They just aren't Measurement Servers that meet these "LMAP" requirements. Do any of the use cases create a need for all Measurement Servers (used by that use case) to meet these requirements? I don't think so. Even the 3rd party testing use case should allow for ping tests without insisting that the server supporting the ping tests meet these requirements.
 
This LMAP architecture is assuming that a requirement of the architecture is to support all use cases, within a single architecture. I disagree with that premise.
And I disagree that 3rd party management and control of "LMAP" architecture elements is a compelling use case. The protocols may be simple, but it's incredibly complex to manage and administer, and to put in all the controls needed to prevent end users from being negatively impacted by 3rd parties. A simple requirement of "make sure 3rd parties can't break anything" is insufficient and ignores the fact that ensuring this is incredibly complex and difficult.
Barbara

---

From: "STARK, BARBARA H" <[email-masked]>
Date: Mon, 19 Nov 2012 22:12:47 +0000

> > Measurement controller: This is not a necessary architectural element. This
> > is an element specific to this proposed physical architecture.
> 
> 1) having _strictly_ a single measurement controller be part of, or in control
> of a single measurement "panel" (e.g.: group of measurement clients &
> measurement server); or,
> 2) two, or more, measurement controllers (likely owned/operated by
> different parties) acting in concert to coordinate the rendezvous between an
> appropriate panel of measurement clients with measurement servers.  See
> my previous e-mail re: "Section 3" for an example of what I'm trying to
> describe here.
> 

<bhs> Measurement Controllers are not required by all use cases (e.g., end user on-demand testing that is controlled and managed by the end user, using devices under the direct control and management of the end user, doesn't call for the existence of a Measurement Controller).
I do not find the use case of allowing 3rd party management and control of test infrastructure to be a compelling use case. IMO, this is not a necessary function of any of the more general use cases for support of end-user specific on-demand testing and large scale performance measurements.
Outside of this 3rd party use case, none of the other use cases really care about how control is achieved -- it's a function internal to an architecture of a single entity. And as I mentioned in my other emails, not all Measurement Clients or Measurement Servers need to allow for measurement-specific remote control.
We have no clue as to what the desired metrics are, or the recommended measurements and tests to achieve those metrics; so assuming a Measurement Controller is needed to coordinate any of these tests is premature. Therefore, I don't consider it to be a necessary element of an architecture.
Barbara

---

From: "STARK, BARBARA H" <[email-masked]>
Date: Tue, 20 Nov 2012 14:16:03 +0000

> [--snip--]
> > Data collector: I do not see this as a necessary architectural element, either.
> > As with the Measurement controller, it has been assigned the role of
> > providing multiple, separable functions: collecting data, storing data, and
> > providing access to the data.
> 
> I'm not sure I understand your concern.  Given the resource constraints on
> some classes of measurement clients, doesn't it make sense for the
> measurement client to send it's performance measurement results to "the
> cloud" (a.k.a.: data collector) for longer-term archival, storage and centralized
> access by the ISP and/or 3rd-parties?

<bhs> For the end user on-demand tests, I don't see that a distinct data collector necessarily needs to exist. The Measurement Client could present collected data to the end user directly through a local user interface. The end user may or may not want to have the data saved for future use. The end user may not have a requirement for on-demand test data to be shared with others. Also, since the LMAP Data Collector definition insists that many functions be combined together (data collection function, data storage function, and providing access to the data function), I do not agree that this combined-function Data Collector architectural element, as defined in the draft, is necessary to providing functionality that some of the use cases do call for. I agree that a data collection function, a data storage function, and a data access function are needed by some of the use cases. I believe it is possible for data collection and storage functions to be internal to a single entity's management architecture, and am not convinced that deployers and implementers of these functions are asking for IETF help in defining them.
 
> > Network parameter server: I do not see this as a necessary architectural
> element. If there is a need for an externally-facing function that provides
> access to access-provider-collected data for the regulatory use case to
> include corresponding (anonymous) information related to subscribed line
> rate, etc., then that is a requirement on that external-facing function. If an
> access provider wants help in figuring out how to get this information to the
> external-facing function, then I would prefer for that access provider (or
> their proxy) to ask for such help.
> 
> I would think that if, say, an ISP already has a "network parameter server"
> they already own & operate, then that's fine ... they can continue to run it as
> they do today.  However, the key aspect we probably want to work on is
> defining a _standards-based_ "data model" (e.g.: XML or JSON schema?)
> that can encapsulate appropriate/relevant bits of data about subscribed lines
> in that (existing) Network Parameter Server.  This would allow for much
> more straightforward access to and parsing of "network parameters" by, for
> example, 'authorized parties' as the draft states (Section 3.4).

<bhs> I don't object to data model definition. I do think it may be premature, in the absence of a good understanding of the metrics. Without understanding the metrics, I believe the data model would need to be very abstract. Metrics, to me, could include an expression of "subscribed line rate" that has consistent meaning across all access technologies. This information is something I would consider a "network parameter". I would prefer to see IETF focus on metrics, rather than a measurement management and control architecture. Metrics can drive a data model that is independent of the architecture used to deliver the data.

> > If there is a desire to define an interworking function that would allow a 3rd
> > party to get (not anonymous) customer-specific line information directly
> > from the access provider (so the 3rd party could correlate it with data they
> > collect from their own Measurement client(s)), then that would be a distinct
> > function, as well. Personally, I would shy away from trying to define this
> > function. But if someone else wants to tackle it, I might be willing to provide
> > comments on it.
> 
> Hrm.  You raise a good point above about a 3rd-party acquiring "non-
> anonymous" customer line information from the access provider so that the
> 3rd-party could correlate it with data they collect from their own (3rd-party)
> measurement clients.
> 1)  I'm not sure what, if any, technical mechanisms the IETF could develop to
> prevent such a thing from happening, other than to NOT specify any data-
> models/schemas that encapsulate such sensitive, non-anonymous
> information in a 'standards-based' data-model in the first place.
> 2)  That said, I would think that "privacy laws" in place in the U.S. and other
> parts of the world (perhaps most significantly, the EU) would prevent the
> ISPs from [ever] offering access to that data for fear of violating both the law
> and, more importantly, their customer's trust as well.

<bhs> I'm not aware of any access provider who *wants* to do this. I couldn't quite tell if some of the requirements were trying to provide support for this use case -- I could read some of the statements either way. I'm glad we agree that this is not a use case that we need to support.
Barbara

---

From: Marc Linsner <[email-masked]>
Date: Tue, 20 Nov 2012 09:22:15 -0500

Barbara,

>
><bhs> Measurement Controllers are not required by all use cases (e.g.,
>end user on-demand testing that is controlled and managed by the end
>user, using devices under the direct control and management of the end
>user, doesn't call for the existence of a Measurement Controller).
>I do not find the use case of allowing 3rd party management and control
>of test infrastructure to be a compelling use case. IMO, this is not a
>necessary function of any of the more general use cases for support of
>end-user specific on-demand testing and large scale performance
>measurements.
>Outside of this 3rd party use case, none of the other use cases really
>care about how control is achieved -- it's a function internal to an
>architecture of a single entity. And as I mentioned in my other emails,
>not all Measurement Clients or Measurement Servers need to allow for
>measurement-specific remote control.
>We have no clue as to what the desired metrics are, or the recommended
>measurements and tests to achieve those metrics; so assuming a
>Measurement Controller is needed to coordinate any of these tests is
>premature. Therefore, I don't consider it to be a necessary element of an
>architecture.
>Barbara

It's probably just me, but I'm getting confused about your usage of '3rd
party management and control of test infrastructure'.  Are you referring
to controlling tests run on a client not 'owned' by the entity operating
the controller?

I'm taking a more simplistic view of the architecture.  A measurement
client can reside 'anywhere', it's a software entity communicating with
the controller via layer 3/4/7 (not layer 2 control plane).  IMO, for
security reasons, the client will be configured to communicate with a
measurement controller (no open ports soliciting a controller).  If it is
the desire of the entity configuring the client to allow a 3rd party to
control the client, that is a perfectly valid use case/scenario.

As far as metrics, my assumption is any metric defined by IPPM or related
SDO body, with additions as desired. From the plenary presentation I would
expect at least the following metrics: Sustained Download, Sustained
Upload, Web Browsing Download, UDP Packet Loss, VoIP Measure, DNS
Failures, ICMP Packet Loss, Total Bytes Downloaded, Burst Download, Burst
Upload, UDP Latency, Video Streaming Measure, DNS Resolution, Latency
Under Load, Total Bytes Uploaded.  Obviously, extensibility is a key
feature.

-Marc-

---

From: "STARK, BARBARA H" <[email-masked]>
Date: Tue, 20 Nov 2012 15:31:36 +0000

> From: Marc Linsner [mailto:[email-masked]]
> Sent: Tuesday, November 20, 2012 9:22 AM
> To: STARK, BARBARA H
> Cc: [email-masked]
> Measurement Controller
> >premature. Therefore, I don't consider it to be a necessary element of
> >an architecture.
> >Barbara
> 
> It's probably just me, but I'm getting confused about your usage of '3rd party
> management and control of test infrastructure'.  Are you referring to
> controlling tests run on a client not 'owned' by the entity operating the
> controller?

<bhs> Yes. That use case is called out in this draft.
 
> I'm taking a more simplistic view of the architecture.  A measurement client
> can reside 'anywhere', it's a software entity communicating with the
> controller via layer 3/4/7 (not layer 2 control plane).  IMO, for security
> reasons, the client will be configured to communicate with a measurement
> controller (no open ports soliciting a controller).  If it is the desire of the
> entity configuring the client to allow a 3rd party to control the client, that is a
> perfectly valid use case/scenario.

<bhs> I agree that Measurement Clients can reside anywhere. I disagree that all Measurement Clients must be controlled. Measurement Clients for use by the end user only do not necessarily require a Measurement Controller. I agree with your statement on control configuration with the caveat that it only applies *if* a client is controlled. I agree that 3rd party control is a valid use case. What I believe I said (in another email) is that I consider it to be an incredibly complex use case and not very compelling. By that I mean that I don't want to spend my time working on it. If others do, that's fine, as long as they aren't surprised and angered by the fact that not everyone will want to implement and deploy it, and that there may be active resistance to any attempt to mandate it. 

> expect at least the following metrics: Sustained Download, Sustained Upload,
> Web Browsing Download, UDP Packet Loss, VoIP Measure, DNS Failures,
> ICMP Packet Loss, Total Bytes Downloaded, Burst Download, Burst Upload,
> UDP Latency, Video Streaming Measure, DNS Resolution, Latency Under
> Load, Total Bytes Uploaded.  Obviously, extensibility is a key feature.

<bhs> How key is extensibility, how dynamic does extensibility have to be, and are various approaches to achieving extensibility allowed? Can extensibility be achieved by firmware upgrades that happen maybe once a year? Do all clients and servers have to be extensible, or is it ok to achieve extensibility through the addition of new clients and servers, potentially in new/different/separate physical devices/elements? Does extensibility imply near infinite extensibility or is it reasonable for extensibility to be constrained by lack of resources available on the device that hosts a client?
Barbara

---

From: Nicholas Weaver <[email-masked]>
Date: Tue, 20 Nov 2012 08:01:33 -0800

> <bhs> How key is extensibility, how dynamic does extensibility have to be, and are various approaches to achieving extensibility allowed? Can extensibility be achieved by firmware upgrades that hap…

I've said it once, I'll say it one final time:

This focus on architecture is deeply misguided.  As someone who's built one such architecture (Netalyzr), working on two others (Fathom, a browser extension, and techniques using GuruPlugs/Raspberry Pis), the architecture is the easy part.

In all this work, having some "standard" way to communicate between my measurement clients and servers is useless.  I just use HTTP, HTTPS, or SSH depending on whether quick & easy or server side or mutual authentication is required, with whatever load-balancing is needed handled by random assignment of remote test node.   

The internal data store is whatever is appropriate for the project (eg. for Netalyzr its python pickled objects for storage on our web server, data in a Postgress database for analysis).  

Updates are handled appropriately (Netalyzr is updateless, its an applet so its always up-to-date.  Our android version will just use the app store update mechanism.  Fathom we use Firefox's update mechanism.  The plugs are a hack using SSH, but its a fair amount of custom goop, and they update nightly).  

And all three end up having rather different architectural requirements.

The hard part is developing the test metrics and measurements, both in the abstract and within the API constraints of the target platform.  E.g. why I like the GuruPlugs and Raspberry Pi is that I can run Java, so therefore I can just "Run Netalyzr" for all Netalyzr tests.  But the same isn't the case for the Ripe Atlas or Bismark platform.

And Marc's list is actually just a start: you also have generic proxy presence & location detection, application specific proxy detection and analysis, packet injection detection, differential application behavior detection, application agnostic traffic shaping detection, DNSSEC support, DNS manipulation detection, path MTU, IPv6 for all of the previous, and a ton of others.  Netalyzr is currently composed of >100 distinct tests.  And this amount just continues to grow.

If the IETF actually wants to have an impact in this area, don't bother focusing on protocols for communication, focus on tests with given attributes.  Because people like me who actually build these things are going to ignore an architectural focus because that has no value.  But having ACCEPTED STANDARD tests would be very valuable.  

E.g. a good, standard latency-under-load test that can be written in Flash (so TCP/UDP connections only to cooperating servers) would be great.  Our current one in Netalyzr is only OK: we believe that one could do a lot better.

If you could then figure out WHERE the bottleneck is (using a server that can also do simultaneous traceroutes), this would be even more valuable.

---

From: Marc Linsner <[email-masked]>
Date: Tue, 20 Nov 2012 11:09:06 -0500

>>client
><bhs> I agree that Measurement Clients can reside anywhere. I disagree
>that all Measurement Clients must be controlled. Measurement Clients for
>use by the end user only do not necessarily require a Measurement
>Controller. I agree with your statement on control configuration with the
>caveat that it only applies *if* a client is controlled. I agree that 3rd
>party control is a valid use case. What I believe I said (in another
>email) is that I consider it to be an incredibly complex use case and not
>very compelling. By that I mean that I don't want to spend my time
>working on it. If others do, that's fine, as long as they aren't
>surprised and angered by the fact that not everyone will want to
>implement and deploy it, and that there may be active resistance to any
>attempt to mandate it.

<ml> IMO, remote management of a measurement client is a feature, but in
the context of LMAP, it is required.  Nothing has been stated that client
local UI will not be able to invoke any/all tests, I believe that is a
client implementor decision.

I am curious why you believe 3rd party control "to be an incredibly
complex use case and not very compelling".  Outside of the obvious
security considerations, then from a protocol perspective, what makes 3rd
party control any different from 1st party control?

>>Upload,
><bhs> How key is extensibility, how dynamic does extensibility have to
>be, and are various approaches to achieving extensibility allowed? Can
>extensibility be achieved by firmware upgrades that happen maybe once a
>year? Do all clients and servers have to be extensible, or is it ok to
>achieve extensibility through the addition of new clients and servers,
>potentially in new/different/separate physical devices/elements? Does
>extensibility imply near infinite extensibility or is it reasonable for
>extensibility to be constrained by lack of resources available on the
>device that hosts a client?
>Barbara

<ml> I'm not sure your concerns wrt extensibility are a LMAP issue.  If
LMAP does their job correctly, the controller-client protocol will be
extensible.  Whether/when a v1 client/controller is upgradable to v2 can't
be dictated/demanded by LMAP/IETF.  With that said, it would behove LMAP
to include any/all known/reasonable use cases in v1, and at a future time,
v2 taking into consideration v1 capabilities/backward compatibility.

-Marc-

---

From: Marc Linsner <[email-masked]>
Date: Tue, 20 Nov 2012 11:18:21 -0500

Nicholas,

Thanks for jumping in. I believe most in the IETF rely on the IPPM wg to
develop the test metrics.  LMAP is focused on client/controller management
issues.

I'm curious how you propose vendor A's controller to work with vendor B's
client if the communication protocol/architecture isn't written down for
all to see?

-Marc-

-----Original Message-----
From: Nicholas Weaver <[email-masked]>
Date: Tuesday, November 20, 2012 11:01 AM
To: "STARK, BARBARA H" <[email-masked]>
Cc: Nicholas Weaver <[email-masked]>, Marc Linsner
<[email-masked]>, "[email-masked]" <[email-masked]>
Subject: Focus on Tests, Not Architectures...

>This focus on architecture is deeply misguided.  As someone who's built
>one such architecture (Netalyzr), working on two others (Fathom, a
>browser extension, and techniques using GuruPlugs/Raspberry Pis), the
>architecture is the easy part.
>
>In all this work, having some "standard" way to communicate between my
>measurement clients and servers is useless.  I just use HTTP, HTTPS, or
>SSH depending on whether quick & easy or server side or mutual
>authentication is required, with whatever load-balancing is needed
>handled by random assignment of remote test node.
>
>The internal data store is whatever is appropriate for the project (eg.
>for Netalyzr its python pickled objects for storage on our web server,
>data in a Postgress database for analysis).
>
>Updates are handled appropriately (Netalyzr is updateless, its an applet
>so its always up-to-date.  Our android version will just use the app
>store update mechanism.  Fathom we use Firefox's update mechanism.  The
>plugs are a hack using SSH, but its a fair amount of custom goop, and
>they update nightly).
>
>The hard part is developing the test metrics and measurements, both in
>the abstract and within the API constraints of the target platform.  E.g.
>why I like the GuruPlugs and Raspberry Pi is that I can run Java, so
>therefore I can just "Run Netalyzr" for all Netalyzr tests.  But the same
>isn't the case for the Ripe Atlas or Bismark platform.
>
>And Marc's list is actually just a start: you also have generic proxy
>presence & location detection, application specific proxy detection and
>analysis, packet injection detection, differential application behavior
>detection, application agnostic traffic shaping detection, DNSSEC
>support, DNS manipulation detection, path MTU, IPv6 for all of the
>previous, and a ton of others.  Netalyzr is currently composed of >100
>distinct tests.  And this amount just continues to grow.
>
>
>If the IETF actually wants to have an impact in this area, don't bother
>focusing on protocols for communication, focus on tests with given
>attributes.  Because people like me who actually build these things are
>going to ignore an architectural focus because that has no value.  But
>having ACCEPTED STANDARD tests would be very valuable.
>
>E.g. a good, standard latency-under-load test that can be written in
>Flash (so TCP/UDP connections only to cooperating servers) would be
>great.  Our current one in Netalyzr is only OK: we believe that one could
>do a lot better.
>
>If you could then figure out WHERE the bottleneck is (using a server that
>can also do simultaneous traceroutes), this would be even more valuable.
>

---

From: Nicholas Weaver <[email-masked]>
Date: Tue, 20 Nov 2012 08:47:39 -0800

> client if the communication protocol/architecture isn't written down for

Its not a problem:  Vendor A's controller with Vendor B's client can only talk to each other anyway if they are running the same test.  Which is often insanely test specific what they need to talk about: you aren't going to be able to standardize that away in any generic fashion.

Then for the protocol to communicate results & control, just use HTTP/HTTPS/SSH (pick your transit protocol) and JSON/XML/HTML (pick your data framing) which meets your requirements for convenience and authentication.

E.g. we already have this for Netalyzr.  

If you run the Netalyzr command-line client, you can get the results in a nice JSON blob through HTTP from our server when everything is said and done, but that the JSON blob's content and structure is Netalyzr specific is not a surprise, it needs to be Netalyzr specific.

http://n1.netalyzr.icsi.berkeley.edu/json/id=43ca253f-20573-9eafe28f-27fa-4bb9-b262

As you can see, we chose HTTP (no need for authentication) and JSON (easier to parse & unparse), and we don't have all the server side rendering in the JSON representation (yet) but all client-side data is included.  If someone wants a JSON rendering for a server side result we don't yet do, tell us and we will add it in.

This creates a nice standard way for someone else to communicate and fetch results from our service, and they can use our binary client as a black box (since actually implementing all of the Netalyzr test suite would be a big endeavor, and pointless if your client can run Java).

Similarly, we have a visiting student building a really cool proxy-traceroute service (where does a proxy exist between you and the Internet, measured from the Internet side, on a specific port?).  

Getting the results will be just an HTTP fetch, with a specific HTML framing.  In this case we chose HTML so its both human readable and likely to be (relatively) unmangled at least in semantics.

---

From: Brian Trammell <[email-masked]>
Date: Tue, 20 Nov 2012 17:59:25 +0100

Hi, Nick,

(Copying this over to IPPM as well, as the focus-on-metrics discussion will be interesting there too).

> And all three end up having rather different architectural requirements.

Architectures, I would submit (and as I've said here before), are useful to hang terminology off of: you need boxes and lines so everyone understands (roughly) the same thing when you say "client" and "server" and "controller" with reference to a set of measurements you define.

But yes, specifying a way to specify a measurement is largely arbitrary. I'd even advocate separating the representation from the transport, especially as the general problem of measurement interoperability has data sources operating and wildly different scales, but as you say deciding this is the easy part.

> If the IETF actually wants to have an impact in this area, don't bother focusing on protocols for communication, focus on tests with given attributes.  Because people like me who actually build the…

+ 1e<large number>. Stated another way, the protocol gives you the grammar. What you need to make this at all interoperable is the vocabulary. 

IPPM has defined a several of these at a low level, with a focus on active measurement methods that produce comparable results. There appears to be interest in IPPM both in adding passive methods for the existing metrics (see draft-trammell-ippm-hybrid-ps) as well as adding definitions of more complex metrics based in part on the simple ones (see draft-mathis-ippm-model-based-metrics).

> If you could then figure out WHERE the bottleneck is (using a server that can also do simultaneous traceroutes), this would be even more valuable.

(This seems to make the assumption that traceroute-measurable latency is strongly correlated with application latency at a given hop, but I get your point...)

The efforts of IPPM to date have been focused on simply describable, easily repeatable metrics... there's not a lot of specification of arbitrarily complex or unknown preconditions (as would be useful in the latency-under-load example you give), because these tend to be counter to the goals of maximizing repeatability and minimizing uncertainty. There's also a focus on active measurement for the same reason. On the other hand, a lot of work in distributed measurement systems is focused on measuring the network as it is, with implicit or explicit passive components (again, returning to "latency under load", the load traffic will be made up of generated as well as existing background load). 

So, what might be useful would be to open another line of work within IPPM on interoperable descriptions of some of these existing tests as empirically specified metrics, perhaps with relaxed compliance with the IPPM framework as appropriate. This could be used to provide a more complex vocabulary to LMAP, but could also be useful independently -- if LMAP is eventually defined to apply only to a restricted set of use cases (i.e. specifically mandated tests for regulatory compliance, but not research or troubleshooting), comparable metrics with incompatible architecture and protocols could still lead to wider interoperability in the measurement space, the need for protocol shims (which are after all easy to write if the number inside has the same meaning) aside.

Best regards,

Brian

---

From: "STARK, BARBARA H" <[email-masked]>
Date: Tue, 20 Nov 2012 17:02:44 +0000

> If the IETF actually wants to have an impact in this area, don't bother focusing
> on protocols for communication, focus on tests with given attributes.
> Because people like me who actually build these things are going to ignore an
> architectural focus because that has no value.  But having ACCEPTED
> STANDARD tests would be very valuable.

+1

There are clients that will support control via OMA-DM, clients that will support control via TR-069, clients that will support control by SNMP, clients that will support control by some yet-to-be-defined protocol, clients where client and controller vendor are the same and the protocol is proprietary, etc. It's generally a good idea to know how to control a controllable client before deploying it.

Since client and controller are deployed by the same entity, it should be up to that entity to figure out their control architecture, and that entity should have full say in how to do their own control architecture.

Is there any entity asking for help from IETF in figuring out what they want to internally deploy?
Barbara

---

From: <[email-masked]>
Date: Tue, 20 Nov 2012 17:09:06 +0000

agree with initial focus on tests.

even if you think architecture(s) should be in scope, I think it's better to start by defining the work that everyone agrees would be valuable to do. Otherwise we'll end up not starting the metrics/tests work until we've done a bunch of architecture work! Perhaps once the tests/metrics work is defined, we can work out whether there is any new protocol needed (for reporting results or initiating tests say)

phil

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Nicholas Weaver
Sent: 20 November 2012 16:02
To: STARK, BARBARA H
Cc: Marc Linsner; Nicholas Weaver; [email-masked]
Subject: [lmap] Focus on Tests, Not Architectures...

On Nov 20, 2012, at 7:31 AM, STARK, BARBARA H wrote:
>> From: Marc Linsner [mailto:[email-masked]] As far as metrics, my 
>> assumption is any metric defined by IPPM or related SDO body, with 
>> additions as desired. From the plenary presentation I would expect at 
>> least the following metrics: Sustained Download, Sustained Upload, 
>> ICMP Packet Loss, Total Bytes Downloaded, Burst Download, Burst 
>> Under Load, Total Bytes Uploaded.  Obviously, extensibility is a key feature.
>

---

From: KEN KO <[email-masked]>
Date: Tue, 20 Nov 2012 17:53:19 +0000

>Since client and controller are deployed by the same entity, it should be up to that entity to figure out their control architecture, and that entity should have full say in how to do their own cont…

I don't think it's a universally held assumption that the client and controller are deployed by the same entity. For instance, a regulator use case spanning multiple operators could use a controller function (and, a data collection function) that communicates with clients deployed by each of those operators. So I don't think we should ignore architecture.

That said, I do believe that focusing on tests is an excellent idea and that that work could progress in ippm while other lmap issues are still being discussed.

---

From: Shane Amante <[email-masked]>
Date: Mon, 26 Nov 2012 21:40:02 -0700

On Nov 20, 2012, at 10:53 AM, KEN KO <[email-masked]> wrote:
> I don't think it's a universally held assumption that the client and controller are deployed by the same entity.

+1.

> For instance, a regulator use case spanning multiple operators could use a controller function (and, a data collection function) that communicates with clients deployed by each of those operators. So I don't think we should ignore architecture.

Agreed.  I also want to stress that there are other applicable, non-regulatory use cases, as well, where (for example) a single controller speaks to LMAP measurement clients deployed across a multitude of operators.

> That said, I do believe that focusing on tests is an excellent idea and that that work could progress in ippm while other lmap issues are still being discussed.

Agreed.

I'd also add that there may be two solution spaces emerging wrt "metrics" (in IPPM) that we discover.  First, as Brian Trammell pointed out in his e-mail on this thread:
> The efforts of IPPM to date have been focused on simply describable, easily repeatable metrics... there's not a lot of specification of arbitrarily complex or unknown preconditions (as would be
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> useful in the latency-under-load example you give), because these tend to be counter to the goals
                                                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> of maximizing repeatability and minimizing uncertainty. There's also a focus on active measurement
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>  for the same reason.

... these may be applicable for an extremely large number of (or, all?) "LMAP Measurement Clients" and provide 'basic' measurement capabilities, e.g.: UDP packet loss, UDP latency, etc.

On the other end of the spectrum, I think this is where the research community is still (?) in the midst of exploring + defining "new" metrics that can then be used in small to medium-size experiments, (e.g.: on a smaller base of "LMAP Measurement Clients"), whose scientific results can ultimately inform which of these tests may get formalized into standards by the IETF and, if appropriate, incorporated as "new" tests or replace one or more existing tests (completely).  Obviously, how those tests ultimately get rolled out to the field (potentially through firmware updates or only when new generations of HW + firmware are shipped), will be determined by the extent of the change, etc.

While I have an interest in following the new(er) metrics being explored, in the 2nd paragraph, I am more interested in and do not want us to lose sight of those basic metrics, in the first paragraph, as I believe a lot of valuable data can still be gleaned from them, today.

-shane