Subject: [lmap] Comments on draft-schulzrinne-lmap-requirements-00

From: KEN KO <[email-masked]>
Date: Fri, 21 Sep 2012 15:09:21 +0000

Thank you for posting this draft - it's an excellent introduction to the issues and requirements associated with large scale network performance testing. I think it will generate much needed discussion. To help kick that discussion off, here are some comments based on an initial reading of the material.

In section 3, the descriptions of the measurement client and measurement server seem to be inconsistent. The server can both sink and source traffic but the client is only a source (or an observer for passive measurements). I think both the client and the server will source and sink traffic. A more general comment is the need for unambiguous definition of terms including measurement client and measurement server. This should probably be coordinated with framework definition work in progress at the Broadband Forum.

In section 4, the "measurement client to measurement controller" protocol is described in terms of "pull" - that is, with each client querying the controller for its measurement schedule. This may be preferable for an architecture where the network is sparsely populated with measurement clients and a high percentage of the installed clients participate in a test program. Looking forward to an environment where test clients become ubiquitous and the number of clients participating in test program may be a tiny fraction of the total, a "push" protocol where the controller initiates communication only to those clients actively involved in a test program should result in far less control traffic. We may need to consider both alternatives. A solution hinted at in Section 5 might be where measurement clients initially register with a controller and are notified of push or pull mode as part of the registration process.

There is a need to identify different types of measurement clients to the measurement controller. Some clients, for instance located at residential or business gateways adjacent to UNIs, are primary candidates for scheduled test activities. Other clients located at intermediate nodes within the network may be used only for troubleshooting, or at least may have a different set of tests scheduled than a gateway. Still others such as software-based clients installed on fixed or mobile endpoint devices might serve either purpose, depending on whether there the associated gateway is a measurement client and whether the endpoint is considered "reliable" for the purposes of a specific test program. Classification of measurement client both by feature set and by network location may be desirable.

Requirements comments:
A-3: Not really a requirement - this is informative as a use case.
C-3: Depending on how registration gets defined, this may be a MUST.
D-6: It isn't clear how "interleave" is being used here or why clients MUST be able to interleave data samples from different measurement metrics. What is the problem with a file transfer that groups samples by metric type?
D-9: seems redundant given A-6.
N-1: Consider making this capability a MUST. Whether it is used in a specific environment is out of standards scope.
N-6, N-7: redundant given A-6. Authentication features should be MUSTs.

Finally, many of the descriptions and requirements in this document are relevant to the performance testing architecture and framework being developed in the Broadband Forum. I recommend liaising this document, or the updated document resulting from the upcoming IETF meeting, to the BBF.

Best regards,
Ken Ko

---

From: Al Morton <[email-masked]>
Date: Mon, 24 Sep 2012 14:48:38 -0400

LMAP co-authors,

Some further comments on your Internet Draft (I also appreciate reading your
views on this measurement architecture, it is a challenging topic as
you rather thoroughly described it in the 00 version).

The comments concern the measurement entity roles, both those currently
identified and possible additions. Also, there are complexity limitations
to consider, especially when the devices perform both active and passive
measurements.

As Ken Ko pointed out, the names and definitions of measurement client
and measurement server are somewhat ambiguous. Although they are familiar
terms, they imply a hierarchy which may not be present. It seems that
these two entities are most easily distinguished by their location in
the architecture, and depending how the measurement protocol works and
the metric measured, the entities could take on multiple roles. It is
worth considering more generic names, like "user agent" and "network agent".
Location names hold-up when testing between two network agents.

The generic names avoid two problems:
  - Some candidate test protocols already use client and server
  - IPPM's Source and Destination host designations were fine for
    one-way testing, but were much less clear with round-trip metrics.

The names could be even more specific, i.e. "user active measurement agent",
and this would help to categorize requirements when the details of
active and passive measurements are decided. One could imagine categories
of user agents with ability to test rates up to XX Mbps, others up to
YY (> XX) Mbps, while maintaining other accuracy requirements.

It's also useful to consider less complex measurement participation
at the distant edge of the network, where the scale is the largest.
Some test protocols and performance metrics already take this reality
into account, and should be candidates for the job.

I think the demands of timing distribution in a packet network
and the fundamental importance of time synchronization and stability
in measurements warrant adding a time distribution entity to the
architecture, with the additional need to identify the protocol between
all entities requiring accurate estimates of time.

I have some detailed comments on the requirements to share, but
they will have to wait for a future message.

regards,
Al

At 11:09 AM 9/21/2012, KEN KO wrote:
>Thank you for posting this draft – it’s an 
>excellent introduction to the issues and 
>requirements associated with large scale network 
>performance testing. I think it will generate 
>much needed discussion. To help kick that 
>discussion off, here are some comments based on 
>an initial reading of the material.
>
>In section 3, the descriptions of the 
>measurement client and measurement server seem 
>to be inconsistent. The server can both sink and 
>source traffic but the client is only a source 
>(or an observer for passive measurements). I 
>think both the client and the server will source 
>and sink traffic. A more general comment is the 
>need for unambiguous definition of terms 
>including measurement client and measurement 
>server. This should probably be coordinated with 
>framework definition work in progress at the Broadband Forum.
>
>In section 4, the “measurement client to 
>measurement controller” protocol is described in 
>terms of “pull” – that is, with each client 
>querying the controller for its measurement 
>schedule. This may be preferable for an 
>architecture where the network is sparsely 
>populated with measurement clients and a high 
>percentage of the installed clients participate 
>in a test program. Looking forward to an 
>environment where test clients become ubiquitous 
>and the number of clients participating in test 
>program may be a tiny fraction of the total, a 
>“push” protocol where the controller initiates 
>communication only to those clients actively 
>involved in a test program should result in far 
>less control traffic. We may need to consider 
>both alternatives. A solution hinted at in 
>Section 5 might be where measurement clients 
>initially register with a controller and are 
>notified of push or pull mode as part of the registration process.
>
>There is a need to identify different types of 
>measurement clients to the measurement 
>controller. Some clients, for instance located 
>at residential or business gateways adjacent to 
>UNIs, are primary candidates for scheduled test 
>activities. Other clients located at 
>intermediate nodes within the network may be 
>used only for troubleshooting, or at least may 
>have a different set of tests scheduled than a 
>gateway. Still others such as software-based 
>clients installed on fixed or mobile endpoint 
>devices might serve either purpose, depending on 
>whether there the associated gateway is a 
>measurement client and whether the endpoint is 
>considered “reliable” for the purposes of a 
>specific test program. Classification of 
>measurement client both by feature set and by 
>network location may be desirable.
>
>Requirements comments:
>A-3: Not really a requirement – this is informative as a use case.
>D-6: It isn’t clear how “interleave” is being 
>used here or why clients MUST be able to 
>interleave data samples from different 
>measurement metrics. What is the problem with a 
>file transfer that groups samples by metric type?
>N-1: Consider making this capability a MUST. 
>Whether it is used in a specific environment is out of standards scope.
>Finally, many of the descriptions and 
>requirements in this document are relevant to 
>the performance testing architecture and 
>framework being developed in the Broadband 
>Forum. I recommend liaising this document, or 
>the updated document resulting from the upcoming IETF meeting, to the BBF.
>
>Best regards,
>Ken Ko
>_______________________________________________
>lmap mailing list
>[email-masked]
>https://www.ietf.org/mailman/listinfo/lmap