Subject: Re: [lmap] user-initiated testing

From: Paul Aitken <[email-masked]>
Date: Wed, 31 Jul 2013 11:30:00 +0100

Jean-Francois, Sharam, Mike, Juergen, All,

The user-initiated test scenario which you describe looks like this:

    ispController      ispCollector
                 \    /
                  \  /
MA--{test peers)
                  /  \
                 /    \
   userController      userCollector

This architecture is contradictory to the WG charter which states, "the 
measurement system is under the control of a single organization", 
because two organisations (ISP, user) now share resources (MAs).

If the WG wants to address this, we should acknowledge that and address 
it _later_, once we have the basic LMAP blocks in place. If we don't get 
the basic blocks in place then the WG will never get anything done.

This architecture is also quite different to the proposed framework and 
the assumption that the MA is under a single Controller at any point in 
time.

Last week some people argued strongly against multiple controllers...

This architecture presents multiple issues as I indicated in my email 
yesterday - which I saw nobody disagreed with. Should we now add that 
the MA may be conflicted by multiple Controllers?

P.

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Wed, 31 Jul 2013 12:38:34 +0200

On Wed, Jul 31, 2013 at 11:30:00AM +0100, Paul Aitken wrote:
> This architecture is contradictory to the WG charter which states,
> "the measurement system is under the control of a single
> organization", because two organisations (ISP, user) now share
> resources (MAs).

It seems you misread my message. Sorry for not having been clear
enough. The picture I drew was this:

		   MA

  ------------------------------------

                   MA
                  /  \
                 /    \

Both are simply comlete distinct LMAP systems, inline with the
assumption stated in the charter.

/js

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1, 28759 Bremen, Germany
Fax:   +49 421 200 3103         <http://www.jacobs-university.de/>

---

From: Paul Aitken <[email-masked]>
Date: Wed, 31 Jul 2013 11:50:21 +0100

Juergen,

>      ispController      ispCollector

If these are distinct MAs, then I completely agree; there is no issue 
here at all.

However, if these are simply different logical sub-divisions of a single 
MA, then the division is not so straightforward.

eg, if the ISP is running a CPU-loading test which the user is unaware 
of, the user-side results may be very poor (eg, user bandwidth seems 
very low).

Whereas a single controller would know that a 100% CPU test is 
scheduled, and therefore not schedule other tests at the same time.

Should we move conflict resolution into each MA? ie, should the MA 
refuse the user initiated test because it conflicts with the ISP test? 
What should it report? Ideally it should return as much information as 
possible as to why the test could not be run, so that the controller can 
potentially schedule an appropriately modified test.

However, if it returns too much information, it could be possible for 
the user to determine a lot of information about the ISP's testing. In a 
different scenario, this could be two different ISPs so privacy is an 
issue...

P.

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Wed, 31 Jul 2013 13:22:57 +0200

On Wed, Jul 31, 2013 at 11:50:21AM +0100, Paul Aitken wrote:
> 
> If these are distinct MAs, then I completely agree; there is no
> issue here at all.

This is how I look at it.

> However, if these are simply different logical sub-divisions of a
> single MA, then the division is not so straightforward.
>
> eg, if the ISP is running a CPU-loading test which the user is
> unaware of, the user-side results may be very poor (eg, user
> bandwidth seems very low).
> 
> refuse the user initiated test because it conflicts with the ISP
> test? What should it report? Ideally it should return as much
> information as possible as to why the test could not be run, so that
> the controller can potentially schedule an appropriately modified
> test.
> 
> However, if it returns too much information, it could be possible
> for the user to determine a lot of information about the ISP's
> testing. In a different scenario, this could be two different ISPs
> so privacy is an issue...

All good reasons to avoid this. (And note, the assumption of an MA is
controlled by a single Controller at any point in time does not
preclude implementations where the underlying runtime system can
enforce proper separation and provide multiple MAs on a single box.)

/js

---

From: Paul Aitken <[email-masked]>
Date: Wed, 31 Jul 2013 12:49:33 +0100

Juergen,

> enforce proper separation and provide multiple MAs on a single box.)

I've no objection to a single physical MA which is able to act as 
multiple virtual MAs, provided that they truly act separately and 
distinctly.

However there's almost certainly some artificial limitation (eg, maximum 
CPU, maximum bandwidth) imposed by the virtual system controller - so 
what's being measured is not the physical limitation, but the artificial 
limitation.

- which was said in yesterday's WG meeting: ensure you know what you're 
really measuring.

P.

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Wed, 31 Jul 2013 14:00:19 +0200

On Wed, Jul 31, 2013 at 12:49:33PM +0100, Paul Aitken wrote:
> However there's almost certainly some artificial limitation (eg,
> maximum CPU, maximum bandwidth) imposed by the virtual system
> controller - so what's being measured is not the physical
> limitation, but the artificial limitation.

Some virtualization systems allow you for example to assign CPU cores
to specific virtual domains. Of course, no system is going to be
perfect - even if you deploy separate hardware boxes, they still do
have limitations.

> - which was said in yesterday's WG meeting: ensure you know what
> you're really measuring.

Yes. This is why I am a big fan of MAs that are not shared because at
the MA software level, it is really difficult to control resource
usage.

/js

---

From: [email-masked]
Date: Wed, 31 Jul 2013 10:09:20 -0400

>    userController      userCollector

This isn't what I had in mind. The example I used was more of a ispMA
on a user device, talking to a second ispCollector2. 

                      \
                       \
                        ispCollector2

I understand that a second collector might be out of scope for 
initial work here, but I'm trying to clear some of the confusion 
on wording and open up the possibilities. It's possible, for 
example, to have an ISP owned MA, ISP-initiated on an end-user 
device. 

user-device != user-initiated != user-owned MA

For example, what if, as an ISP, I built a Windows 8 LMAP app 
that the user can download and install. By running the app, the
user agrees to have this app run ISP-triggered tests once a day 
or upon demand (when he calls support for example). In this case, 
we have: 
Domain = ISP
Device ownership = user
MA ownership = ISP (I own the code of app)
MA control = ISP 
MA initiation = ISP

Another example. Same type of app, this time on Android, on a 
wifi mobile device:
Domain = ISP
Device ownership = user
MA ownership = ISP
MA control = ISP
MA initiation = user

In the above case, I agree with Barbara that a need for user 
authentication might exist, but it could also be done automatically
through source IP address. 

This type of measurement can actually be useful in the regulatory 
case as well. As an ISP, I could run MAs both on user devices and
on ISP devices (integrated broadband routers for example). By 
correlating results between both groups, I could show regulatory 
instances that most speed issues do not arise in the last mile
but actually in the last feet of the connection, in the user
network (with wifi for example). 

I'm not advocating that we bloat the scope of LMAP to huge 
proportions here. My point is simpler: we are putting artificial
and unnecessary boundaries on what LMAP can do if we assume 
an automatic link between device ownership, domain, MA ownership 
and measurement initiation. 

/JF

---

From: <[email-masked]>
Date: Wed, 31 Jul 2013 16:22:09 +0100

>-----Original Message-----
>From: [email-masked] [mailto:[email-masked]] On Behalf Of
>[email-masked]
>Sent: 31 July 2013 15:09
>To: [email-masked]
>Subject: Re: [lmap] user-initiated testing
>
>> Jean-Francois, Sharam, Mike, Juergen, All, The user-initiated test
>> scenario which you describe looks like this:
>>
>This isn't what I had in mind. The example I used was more of a ispMA on a
>user device, talking to a second ispCollector2.
>
>I understand that a second collector might be out of scope for initial work
>here, but I'm trying to clear some of the confusion on wording and open up
>the possibilities. It's possible, for example, to have an ISP owned MA, ISP-
>initiated on an end-user device.

There's no problem with multiple Collectors. The Information Model supports this through multiple Report Channels. Part of the motivation for this was to allow on-demand tests to be directed to Collectors which could be the user or call agent consoles.

Trevor.

---

From: "Bugenhagen, Michael K" <[email-masked]>
Date: Wed, 31 Jul 2013 16:08:58 +0000

Single vs. multiple controller question -

I don't think anything in the charter keeps us from describing the creation of a "client or user" interface for others to execute and or schedule tests via a single controller.

Customer Controller Question ---

   Do we really assume the customer would have their own controller ?

Personally I had assumed we may define a MA, on a ISP NID that has two interfaces, one facing the customer that doesn't require control (a responder), and another that in an initiator that can send tests in either direction.   This spit MA view enable a good amount of flexibility, however both still require some type of local resources available check to make sure they can perform the task at hand.

-----Original Message-----
From: Juergen Schoenwaelder [mailto:[email-masked]] 
Sent: Wednesday, July 31, 2013 7:00 AM
To: Paul Aitken
Cc: [email-masked]; [email-masked]; Bugenhagen, Michael K; [email-masked]
Subject: Re: [lmap] user-initiated testing

On Wed, Jul 31, 2013 at 12:49:33PM +0100, Paul Aitken wrote:
> controller - so what's being measured is not the physical limitation, 
> but the artificial limitation.

Some virtualization systems allow you for example to assign CPU cores to specific virtual domains. Of course, no system is going to be perfect - even if you deploy separate hardware boxes, they still do have limitations.

> - which was said in yesterday's WG meeting: ensure you know what 

Yes. This is why I am a big fan of MAs that are not shared because at the MA software level, it is really difficult to control resource usage.

/js

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Wed, 31 Jul 2013 18:36:01 +0200

> I don't think anything in the charter keeps us from describing the creation of a "client or user" interface for others to execute and or schedule tests via a single controller.

Not sure what "describing the creation of" means. If you want to have
the possibility of such an interface mentioned in say the framework,
fine. If you want to work on details of such an interface, I would say
that is not covered by the current charter. I think Phil had a much
larger 'picture' on his slides since there are a number of components
surrounding the components the LMAP working group is currently
chartered to work on.
 
> Personally I had assumed we may define a MA, on a ISP NID that has two interfaces, one facing the customer that doesn't require control (a responder), and another that in an initiator that can send…

What is a 'responder'? Are you saying you assume that Measurement
Agent and a Measurement Peer are always packaged together? I think the
functional separation should not require this. (And there are some
large measurement systems today where Measurement Agent and
Measurement Peer are clearly not packaged together.)

/js

---

From: "Bugenhagen, Michael K" <[email-masked]>
Date: Wed, 31 Jul 2013 16:40:31 +0000

Question from JS -
What is a 'responder'? Are you saying you assume that Measurement Agent and a Measurement Peer are always packaged together? I think the functional separation should not require this. (And there are some large measurement systems today where Measurement Agent and Measurement Peer are clearly not packaged together.)

/js

Answer -

   I'm assuming that a MA, can include a test initator, and or, a test responder function.
These are distinct existing functions in use today.

My Deeper Answer - we should have a Abstract MA concept here -

   My MA assumption is that a MA should in fact be capable of supporting any new test function without being re-invented.  
Therefore - a Test MA function is really an abstracted subset of functions that support any and all types of testing frameworks, and the MA abstraction model should allow any existing test to be ported to it.

Regards,
Mike

-----Original Message-----
From: Juergen Schoenwaelder [mailto:[email-masked]] 
Sent: Wednesday, July 31, 2013 11:36 AM
To: Bugenhagen, Michael K
Cc: Paul Aitken; [email-masked]; [email-masked]; [email-masked]
Subject: Re: [lmap] user-initiated testing

> I don't think anything in the charter keeps us from describing the creation of a "client or user" interface for others to execute and or schedule tests via a single controller.

Not sure what "describing the creation of" means. If you want to have the possibility of such an interface mentioned in say the framework, fine. If you want to work on details of such an interface, I would say that is not covered by the current charter. I think Phil had a much larger 'picture' on his slides since there are a number of components surrounding the components the LMAP working group is currently chartered to work on.
 
> Personally I had assumed we may define a MA, on a ISP NID that has two interfaces, one facing the customer that doesn't require control (a responder), and another that in an initiator that can send…

What is a 'responder'? Are you saying you assume that Measurement Agent and a Measurement Peer are always packaged together? I think the functional separation should not require this. (And there are some large measurement systems today where Measurement Agent and Measurement Peer are clearly not packaged together.)

/js

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Wed, 31 Jul 2013 21:55:47 +0200

> These are distinct existing functions in use today.

I think we are facing terminology clashes. Can we agree to use one
terminology? I assume what you call a 'test initiator' is what [1]
calles a Measurement Agent and what you call a 'test responder' is
what [1] calls a Measurement Peer. These are functions. How you
package them is an implementation decision.

> Therefore - a Test MA function is really an abstracted subset of functions that support any and all types of testing frameworks, and the MA abstraction model should allow any existing test to be po…

I am trying to translate from what I assume is your terminology but I
get somewhere lost in the second sentence.

/js

[1] http://tools.ietf.org/html/draft-eardley-lmap-terminology-02

---

From: Paul Aitken <[email-masked]>
Date: Wed, 31 Jul 2013 21:05:24 +0100

Jean-Francois,

>       ispController      ispCollector

This is the same topography that I showed in my later email:

      isp   .                   ispCollector
         \  .                  /
          \ .                 /
            controller ---- MA -- {test peers}
          / .                 \
         /  .                  \
     user   .                   userCollector
          LMAP
        boundary

We all seem to agree that this is a valid and likely use case.

> Another example. Same type of app, this time on Android, on a

In the architecture we're discussing, the MA is instructed from a 
controller.

Whether the end-user has access to the controller and what sort of 
access they have, is beyond the scope of LMAP.

P.

---

From: "Bugenhagen, Michael K" <[email-masked]>
Date: Wed, 31 Jul 2013 21:13:02 +0000

Nope I was decomposing a Measurement agent... somewhat like this

     			  	 --------------------
				|  Measurement Agent |
      			|   Control function |
				 --------------------   
					/	   \		
			           /          \
  	      --------------------        -----------------
           |  Test initiator    |      |  test responder |  
	     |    function        |      |    function     |  
		--------------------        ----------------- 

Where a Measurement agent has two separate sub-functions under it, both biding for resources of running a test.
The measurement agent control function talks to the Measurement controller.
The test initiator / responder functions roll up under the Measurement agent, but can work independently of each other.

-----Original Message-----
From: Juergen Schoenwaelder [mailto:[email-masked]] 
Sent: Wednesday, July 31, 2013 2:56 PM
To: Bugenhagen, Michael K
Cc: Paul Aitken; [email-masked]; [email-masked]; [email-masked]
Subject: Re: [lmap] user-initiated testing

> These are distinct existing functions in use today.

I think we are facing terminology clashes. Can we agree to use one terminology? I assume what you call a 'test initiator' is what [1] calles a Measurement Agent and what you call a 'test responder' is what [1] calls a Measurement Peer. These are functions. How you package them is an implementation decision.

> Therefore - a Test MA function is really an abstracted subset of functions that support any and all types of testing frameworks, and the MA abstraction model should allow any existing test to be po…

I am trying to translate from what I assume is your terminology but I get somewhere lost in the second sentence.

/js