Subject: [lmap] About draft-schulzrinne-lmap-requirements-00

From: marcelo bagnulo braun <[email-masked]>
Date: Tue, 25 Sep 2012 09:46:47 +0200

Hi,

Thanks for posting the draft and starting the discussion.

Some comments and questions.

About the Multi-provider network measurements support. In the draft you 
mention that you want to support this, even though it is beyond the 
scope of the document. I guess that the way the architecture is defined 
will implicitly enable some forms of collaboration and made other forms 
of collaboration harder. As per the protocols defined in the current 
document, I understand you could support that Measurements clients 
(M-clients for short) interacting with M-servers and M-controllers and 
even Data collectors from different organizations. This would allow for 
different organizations to schedule tests in the M-clients.
However, there is no protocol for the interaction between Data 
collectors from different entities. I guess this would be abohter way to 
enable interaction between different providers, where they coordiante 
through the M-controller to M-controller protocol a set of tests to be 
executed in different (disjoint) sets of clients and then the data 
collectros exchange the results of the tests. I guess you could achieve 
the same thing by allowing the M-clients to report to multiple data 
collectors, but i would assume you want to minimize the traffic sent by 
the M-clients.
This would allow for a sort of federation of multiple measurement 
platforms, that i would find appealing at first sight. So, I guess my 
question is whether you discarded this type of scenario for some reason, 
or you simply believe that the data collector to data collector protocol 
could reuse an existent protocol or else?

About the Network paramenter server. I am not sure I understand what is 
the role you have in mind for this element.
In the definition, you state that:
"   In some of the use cases, it is necessary for the analysis to compare
    the measured against the nominal network performance, or correlate
    measured parameters with the type and key parameters of the userOs
    network connection."

This may lead to think that the network parameter server provides 
information that is relevant for the analysis of the data rather than 
for the performance of the tests. However, the only protocol defiend 
involving the network parameter server is with the M-client, which 
implies that the parameters are needed to actually perform the tests 
rather than analyzing the data. I think this should be clarified.
Even in the case the parameters are needed to perform the tests, it is 
not clear to me why should the clients retrieve the parameters directly 
from the Network parameters server rather than obtain it from the 
M-controller along with the tests. I mean, defining a protocol between 
the M-controller and the Network parameter server rather than a protocol 
between the network parameter server and the M-cllients. Could you 
expnad on the rationale for this?

Finally, I agree with the comments form Al and Ken about the M-clients 
and M-servers. As presented in the documents, it seems that the M-server 
cannot report results back to the data collector. Wouldnt this be needed 
in some forms of tests?
In addition, I think it would make sense to perform tests between 
M-clients like testing features useful for peer to peer applications. 
While i guess you could place a M-server in the device that is runnig 
the M-client, this seems a bit of the stretch in the terminology to me. 
I think Al's suggestion about agents would accomodate this much more 
naturally.

Regards, marcelo

---

From: Al Morton <[email-masked]>
Date: Tue, 25 Sep 2012 08:59:25 -0400

Hi Marcelo,

At 03:46 AM 9/25/2012, marcelo bagnulo braun wrote:
>...About the Network paramenter server. I am not sure I understand 
>what is the role you have in mind for this element.
>information that is relevant for the analysis of the data rather 
>than for the performance of the tests. However, the only protocol 
>defiend involving the network parameter server is with the M-client, 
>which implies that the parameters are needed to actually perform the 
>tests rather than analyzing the data. I think this should be clarified.

One case that occurred to me as I read the requirement is where the
test might be run very efficiently if a network parameter only needs to
be verified (e.g., capacity), rather than searched for.  And even when
iterating to find an unknown, it helps to know the range of the possible
given that there are orders of magnitude difference between networks.

hope this helps,
Al

>Even in the case the parameters are needed to perform the tests, it 
>is not clear to me why should the clients retrieve the parameters 
>directly from the Network parameters server rather than obtain it 
>from the M-controller along with the tests. I mean, defining a 
>protocol between the M-controller and the Network parameter server 
>rather than a protocol between the network parameter server and the 
>M-cllients. Could you expnad on the rationale for this?

---

From: "Bugenhagen, Michael K" <[email-masked]>
Date: Tue, 25 Sep 2012 13:30:57 +0000

Marcelo,

     I believe the multi-provider configurations are due to one of the test segments being the primary Internet Drain locations.

In this case smaller ISP's may not have a test server at the Main Internet location and may have to work with a third party test server or test server / test controller depending upon how the framework is completed.  From this point of view there are multiple, multi-party configurations.

We may want to use the name "3rd party" vs. multi-operator in that case, however there is yet another case where a 3rd party could be involved.

The two cases I'm thinking of follow :

1)  Third party originates a test, via a ISP test system
2)   ISP uses a 3rd party test server as a target

Depending upon how we frame the inter-system communication I think 
Case #2  would be "east - west" or peer probe, or test controller communications
and Case #1 would very possibly a north bound interface or API, I'm assuming this because I suspect anyone supporting an external 3rd party originating the testing would also like to keep a copy of the results, and the ability to stop the test if it becomes intrusive.

I think the submission is a great start - at the moment I'm wondering about the customers ability to gain information on how their LAN, and Access segments are performing as well.  More to come on those.

Regards,
Mike 

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of marcelo bagnulo braun
Sent: Tuesday, September 25, 2012 2:47 AM
To: [email-masked]
Subject: [lmap] About draft-schulzrinne-lmap-requirements-00

Hi,

About the Multi-provider network measurements support. In the draft you mention that you want to support this, even though it is beyond the scope of the document. I guess that the way the architecture is defined will implicitly enable some forms of collaboration and made other forms of collaboration harder. As per the protocols defined in the current document, I understand you could support that Measurements clients (M-clients for short) interacting with M-servers and M-controllers and even Data collectors from different organizations. This would allow for different organizations to schedule tests in the M-clients.
However, there is no protocol for the interaction between Data collectors from different entities. I guess this would be abohter way to enable interaction between different providers, where they coordiante through the M-controller to M-controller protocol a set of tests to be executed in different (disjoint) sets of clients and then the data collectros exchange the results of the tests. I guess you could achieve the same thing by allowing the M-clients to report to multiple data collectors, but i would assume you want to minimize the traffic sent by the M-clients.
This would allow for a sort of federation of multiple measurement platforms, that i would find appealing at first sight. So, I guess my question is whether you discarded this type of scenario for some reason, or you simply believe that the data collector to data collector protocol could reuse an existent protocol or else?

About the Network paramenter server. I am not sure I understand what is the role you have in mind for this element.

This may lead to think that the network parameter server provides information that is relevant for the analysis of the data rather than for the performance of the tests. However, the only protocol defiend involving the network parameter server is with the M-client, which implies that the parameters are needed to actually perform the tests rather than analyzing the data. I think this should be clarified.
Even in the case the parameters are needed to perform the tests, it is not clear to me why should the clients retrieve the parameters directly from the Network parameters server rather than obtain it from the M-controller along with the tests. I mean, defining a protocol between the M-controller and the Network parameter server rather than a protocol between the network parameter server and the M-cllients. Could you expnad on the rationale for this?

Finally, I agree with the comments form Al and Ken about the M-clients and M-servers. As presented in the documents, it seems that the M-server cannot report results back to the data collector. Wouldnt this be needed in some forms of tests?
In addition, I think it would make sense to perform tests between M-clients like testing features useful for peer to peer applications. 
While i guess you could place a M-server in the device that is runnig the M-client, this seems a bit of the stretch in the terminology to me. 
I think Al's suggestion about agents would accomodate this much more naturally.

Regards, marcelo

_______________________________________________
lmap mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/lmap

---

From: marcelo bagnulo braun <[email-masked]>
Date: Tue, 25 Sep 2012 16:46:10 +0200

Hi Al,

below...

> given that there are orders of magnitude difference between networks.

That makes sense to me.

Two replies. If that is the case, it should be clarified in the 
description of the network parameter server (that would my understanding 
easier)
Second, It is not clear to me what are the assuptions that lead to the 
partiuclar design that the M-client contacts the controller for the 
tests and the network parameter server for the paramters. Wouldnt be 
more efficient for the controller to deliver all the test information 
needed to perfomr the tests? (possibly the controller could get the data 
from network parameter server)

I mean, i wonder if there is some assumption about where the client data 
lives and how to access it that is not spelled out in the document that 
serves as rationale for this particular choice.

Hope it makes sense.

Regards, marcelo

---

From: Robert Kisteleki <[email-masked]>
Date: Tue, 25 Sep 2012 17:04:28 +0200

Hi,

>>> information that is relevant for the analysis of the data rather than for
>>> the performance of the tests. However, the only protocol defiend

Not so much to me. The client should measure, that's its function. Yes, we
can give it hints about what it should expect, but that really shouldn't
affect the measurement itself. In existing measurement systems it's the
presentation or result processing layer's function to make conclusions about
whether the measurement result matches the expectations (e.g. capacity).

Besides, requiring the client to gather input from multiple sources in order
to do its job adds a huge amount of unneeded complexity and reduces reliability.

Cheers,
Robert

---

From: Al Morton <[email-masked]>
Date: Wed, 26 Sep 2012 09:34:06 -0400

Hi Robert,

At 11:04 AM 9/25/2012, Robert Kisteleki wrote:
> >> {Al wrote}
> > {Marcelo wrote}

I tend to agree (and said previously) that the complexity of the
client/agent on the edge of the network needs to be minimized.
So this involves several changes to the division of roles in the
proposal: the client is more subservient to the server/controller when
setting-up tests and running them. I can see how some protocols
could accomplish a lot of the needed measurements working
like this. The details of network parameters would be known at
other testing entities, but not the edge client/agent.

But the requirement to open the firewall and establish NAT state falls
to the edge client/agent. That's a new wrinkle and needs to be sorted-out
with the rest of the details.

regards,
Al

---

From: Brian Trammell <[email-masked]>
Date: Wed, 26 Sep 2012 16:02:19 +0200

Hi, Al, Robert, all,

> other testing entities, but not the edge client/agent.

+1: stated differently, I'd think the general solution to this problem reduces to ensuring reliable metadata export: when the measurement point generates enough information along with the measurements to allow proper interpretation (and adjustment of the interpretation) later, it has to know less at measurement time. In an architecture where each distributed measurement has a central collection point anyway, the collector provides the most logical point to do both correlation among measurement points as well as with externally sourced information about the network.

On the other hand, network parameters that must influence the configuration of the measurement have to be translated into measurement configuration parameters by the controller.

> with the rest of the details.

If the controller and collector must have routable addresses, and the client always initiates the connection thereto, then the NAT problem goes away. But in general, yes, the fact that many, many measurement points in the envisoned scenarios are going to be behind various amounts of NAT will need to be addressed in more detail.

Cheers,

Brian

---

From: Al Morton <[email-masked]>
Date: Wed, 26 Sep 2012 10:24:38 -0400

At 10:02 AM 9/26/2012, Brian Trammell wrote:
>+1: stated differently, I'd think the general solution to this 
>problem reduces to ensuring reliable metadata export: when the 
>measurement point generates enough information along with the 
>measurements to allow proper interpretation (and adjustment of the 
>interpretation) later, it has to know less at measurement time. In 
>an architecture where each distributed measurement has a central 
>collection point anyway, the collector provides the most logical 
>point to do both correlation among measurement points as well as 
>with externally sourced information about the network.

well-stated Brian, added passive measurement perspective appreciated too.
Al