Subject: [lmap] LMAP: situation: Question 4: Measurement Agent device type

From: Benoit Claise <[email-masked]>
Date: Mon, 06 May 2013 16:17:16 +0100

Dear all,

I'm not sure I've seen (much) discussion on this topic

    *Question 4: Measurement Agent device type*
    A branch office router, a home router, a dual home router, a laptop,
    a tablet, a mobile device, ...?
    The discussion started on the list with an email sent on March 7th,
    with the "[lmap] Which devices does LMAP target?" subject. This
    needs more discussion.
    The requirements will be different based on the device type (ex:
    router versus tablet versus mobile).
    On my mobile, for which I pay per transmitted bytes, I don't want to
    pay for those measurements. This comes back to the question of
    voluntary based measurements or not?

I'm concerned that the scope might be too big if we take all these 
device types into account.
People spoke on the list about "service parameters". I agree that these 
are important to combine with the measurement reports.
On a PC, table, or even a mobile device using wireless, how would I know 
the service parameters? How would I make the link with my router service 
status? Multi-provider home is not a uncommon case these days...
Mobiles have multiple connections these days: IPv4/IPv6, 3G, and change 
constantly based on the best QoE.
We would have to deal with a discovery protocol, or a traceroute-based 
type of measurements.

Regards, Benoit

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Mon, 6 May 2013 17:34:23 +0200

On Mon, May 06, 2013 at 04:17:16PM +0100, Benoit Claise wrote:
> People spoke on the list about "service parameters". I agree that
> these are important to combine with the measurement reports.
> On a PC, table, or even a mobile device using wireless, how would I
> know the service parameters? How would I make the link with my
> router service status? Multi-provider home is not a uncommon case
> these days...
> Mobiles have multiple connections these days: IPv4/IPv6, 3G, and
> change constantly based on the best QoE.
> We would have to deal with a discovery protocol, or a
> traceroute-based type of measurements.

I hear you talking about network access options that MAs have and not
really device types. But even then, it is unclear what your question
really is.

- For certain MAs, the measurement traffic volume may be an issue.

  Yes. So we may need to be able to announce the estimated network
  bandwidth consumed so that people can take an informed decisions
  whether the allow/disallow certain tests to be performed by MAs.

  For other deployments, e.g. the operator use case, one would hope
  that measurements that benefit the operator would not be counted
  towards the traffic volume of the subscriber (but that would be a
  contractual / policy issue beyond the IETF's scope).

- For certain MAs, it will be difficult to obtain the service line
  parameters.

  Yes. There is was a debate whether MAs should be involved in the
  communication of line parameters. Perhaps MAs be able to provide
  sufficient information about the test results such that the network
  access used can be determined if necessary. This might be all line
  parameters if that information is accessible to the MA or at least
  information sufficient to determine the beginning of the network
  path used.

I do no think that ruling out device types helps us much. I see it
more of a need to determine requirements about how much contextual
information is desirable and at the same time to discuss the privacy
issues involved with that. In other words, I do not see this as a
charter scoping issue (in case that is driving your question).

/js

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1, 28759 Bremen, Germany
Fax:   +49 421 200 3103         <http://www.jacobs-university.de/>

---

From: Benoit Claise <[email-masked]>
Date: Mon, 06 May 2013 23:06:02 +0100

> really device types. But even then, it is unclear what your question
Let me explain differently, by asking question.
How many MAs do you expect in a branch office?
     One on the router?
How many MAs do you expect to have in a home?
     One per exit router?
     One on my IPAD?
     One on my PC?
     One on my phone?
Those MAs are somehow coordinated?
     If yes, how?
     If not, we have 4 times the measurements.

The access options are also important
     One my phone/IPAD connected via wireless, that could be OK
     One my phone/IPAD connected via 3G, that's not OK with me

Focusing the MA on the router (even if there could be multiple access 
types) reduce the problem space, and this is something that the ISPs do 
already, except they have proprietary solutions

> charter scoping issue (in case that is driving your question).
It is!
Hence my clarifying questions in 
http://www.ietf.org/mail-archive/web/lmap/current/msg00368.html

Regards, Benoit

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Tue, 7 May 2013 07:03:48 +0200

>     If not, we have 4 times the measurements.

I think the answer will depend on who deploys MAs. For the ISP use
case, an MA in the access router (exit router or whatever you call
that box) is likely what the ISP needs. If we talk about say TV over
IP services, I could see that someone might embed an MA in a TV set
(which might do only passive measurements to report characteristics of
multi-media streams). If we allow regular users to deploy MAs in order
to make independent measurements, I would not be surprised is someone
builds an 'app' for popular devices that essentially includes a MA.
But then it is a human user who deploys this 'app', likely he/she
knows the consequences what what he/she is doing.

>     One my phone/IPAD connected via 3G, that's not OK with me

Although an MA doing passive measurements might still be OK on your 3G
link from a bandwidth point of view.

> Focusing the MA on the router (even if there could be multiple
> access types) reduce the problem space, and this is something that
> the ISPs do already, except they have proprietary solutions

Well, yes, such a restriction might allows the WG to sidestep some
discussions. However, from an architectural point of view, I would
hope that the resulting technology is at the end general enough to be
applied to other use cases and deployments as well.

/js

PS: And to cover some of today's hardware probe based systems, you
    would have to replace 'on the router' with 'on or close to the
    router'.

---

From: <[email-masked]>
Date: Tue, 7 May 2013 15:13:12 +0100

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Juergen Schoenwaelder
> Sent: 07 May 2013 06:04
> To: Benoit Claise
> Cc: [email-masked]
> Subject: Re: [lmap] LMAP: situation: Question 4: Measurement Agent
> device type
> 

Interesting point.

If all the MAs are under the control of the same measurement system, then there is some hope that they can be coordinated. In terms of Active Measurement Tasks, the Controller needs to understand that they all use the same access connection. I guess this should be possible as part of the process when the measurement system deploys the MA function onto the device.

However, it's possible that the MAs aren't all under the control of the same measurement system. Perhaps the ISP has deployed the MA on the exit router, the regulator the MA on the STB, the user the MA on the tablet etc. 
I don't think I'd try and coordinate all these MAs. this sounds hard (and if the home is connected to 2 access networks, it might be impossible to know which network an MA would use).
Perhaps we simply say it's good practice for the measurement tasks to 'listen before talk' (and perhaps have a back-off algorithm, especially if the task is long-running or heavy-loading)

> case, an MA in the access router (exit router or whatever you call that
> box) is likely what the ISP needs. If we talk about say TV over IP
> services, I could see that someone might embed an MA in a TV set (which
> might do only passive measurements to report characteristics of multi-
> media streams). If we allow regular users to deploy MAs in order to
> make independent measurements, I would not be surprised is someone
> But then it is a human user who deploys this 'app', likely he/she knows
> the consequences what what he/she is doing.
> 
> > types) reduce the problem space, and this is something that the ISPs
> > do already, except they have proprietary solutions
> 
> discussions. However, from an architectural point of view, I would hope
> that the resulting technology is at the end general enough to be

Agree with Juergen - don't mind this restriction if it helps the chartering process, but hope the standards would have wider applicability

phil

> _______________________________________________
> lmap mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/lmap

---

From: "MORTON JR., ALFRED C (AL)" <[email-masked]>
Date: Tue, 7 May 2013 13:20:05 -0400

I think overlapping measurement systems may be unavoidable.
There's lots of testing to web-based measurement sites today.
When users run tests on-demand, they have some purpose in mind
and agree to the traffic implications, and to the ISP it's 
just user traffic (as in no-gaming).

But the simplest coordination between systems is that they 
avoid cross-traffic - and that may mean that user host MAs
will not be able to detect when the router MA is trying to 
a test, for example, and the router test would back-off
(seeing user traffic). So here I agree with Phil, listening
before (possible during) tests is preferred.

Al

> [email-masked]
> Sent: Tuesday, May 07, 2013 10:13 AM
> To: [email-masked]; [email-masked]
> Cc: [email-masked]
> Subject: Re: [lmap] LMAP: situation: Question 4: Measurement Agent device
> type
> 
> 
> 
> If all the MAs are under the control of the same measurement system, then
> there is some hope that they can be coordinated. In terms of Active
> Measurement Tasks, the Controller needs to understand that they all use
> the same access connection. I guess this should be possible as part of the
> process when the measurement system deploys the MA function onto the
> device.
> 
> However, it's possible that the MAs aren't all under the control of the
> same measurement system. Perhaps the ISP has deployed the MA on the exit
> router, the regulator the MA on the STB, the user the MA on the tablet
> etc.
> I don't think I'd try and coordinate all these MAs. this sounds hard (and
> if the home is connected to 2 access networks, it might be impossible to
> know which network an MA would use).
> Perhaps we simply say it's good practice for the measurement tasks to
> 'listen before talk' (and perhaps have a back-off algorithm, especially if
> the task is long-running or heavy-loading)
> 
> >
> Agree with Juergen - don't mind this restriction if it helps the
> chartering process, but hope the standards would have wider applicability
> 
> phil
> 
> >
> > /js
> >

---

From: "Romascanu, Dan (Dan)" <[email-masked]>
Date: Wed, 8 May 2013 12:58:16 +0000

Hi Al,

Can you draft a sentence or two for charter clarification purposes? We'll see if we can get consensus around it and whether it answers Benoit's question. 

Thanks and Regards,

Dan

> MORTON JR., ALFRED C (AL)
> Sent: Tuesday, May 07, 2013 8:20 PM
> To: [email-masked]; [email-masked];
> [email-masked]
> Cc: [email-masked]
> When users run tests on-demand, they have some purpose in mind and agree
> to the traffic implications, and to the ISP it's just user traffic (as
> in no-gaming).
> 
> But the simplest coordination between systems is that they avoid cross-
> traffic - and that may mean that user host MAs will not be able to
> detect when the router MA is trying to a test, for example, and the
> router test would back-off (seeing user traffic). So here I agree with
> Phil, listening before (possible during) tests is preferred.
> 
> Al
> 
> 
> > From: [email-masked] [mailto:[email-masked]] On Behalf
> > Of [email-masked]
> > > From: [email-masked] [mailto:[email-masked]] On Behalf
> > > Of Juergen Schoenwaelder
> > > Sent: 07 May 2013 06:04
> > > To: Benoit Claise
> > > Cc: [email-masked]
> > If all the MAs are under the control of the same measurement system,
> > then there is some hope that they can be coordinated. In terms of
> > Active Measurement Tasks, the Controller needs to understand that they
> > all use the same access connection. I guess this should be possible as
> > part of the process when the measurement system deploys the MA
> > function onto the device.
> >
> > However, it's possible that the MAs aren't all under the control of
> > the same measurement system. Perhaps the ISP has deployed the MA on
> > the exit router, the regulator the MA on the STB, the user the MA on
> > the tablet etc.
> > I don't think I'd try and coordinate all these MAs. this sounds hard
> > (and if the home is connected to 2 access networks, it might be
> > impossible to know which network an MA would use).
> > 'listen before talk' (and perhaps have a back-off algorithm,
> > especially if the task is long-running or heavy-loading)
> >
> > >
> > > that
> > > services, I could see that someone might embed an MA in a TV set
> > > (which might do only passive measurements to report characteristics
> > > of multi- media streams). If we allow regular users to deploy MAs in
> > > order to make independent measurements, I would not be surprised is
> > > someone builds an 'app' for popular devices that essentially
> includes a MA.
> > > Although an MA doing passive measurements might still be OK on your
> > > 3G link from a bandwidth point of view.
> > >
> > > > access
> > > > types) reduce the problem space, and this is something that the
> > > > ISPs do already, except they have proprietary solutions
> > >
> > > hope that the resulting technology is at the end general enough to
> > > be applied to other use cases and deployments as well.
> > >
> >

---

From: "Livingood, Jason" <[email-masked]>
Date: Wed, 8 May 2013 21:02:09 +0000

>I think overlapping measurement systems may be unavoidable.

may be unavoidable --> is unavoidable

>before (possible during) tests is preferred.

Quite so. I think since MAs will test a range of protocols we can't just
watch for port XX and back off. So the better is to watch for some
threshold traffic level which, if exceeded, means no test is run.

What may be somewhat more challenging is deciding whether that level is
zero traffic (which with background / Internet of things / home automation
/ background apps could be increasingly unlikely) or some low single digit
% of the provisioned rate for an end user (requiring advance knowledge of
this and/or standardized access to discover it) or some small # of kbps.
Or it is configurable for every measurement system. Perhaps one of the
SamKnows guys can comment on how exactly their listener / test run
decision logic works...

In any case, it is good for us to assume that there will normally be
multiple overlapping test administrative domains (for lack of a better
description) within a given logical network such as a home network.

- Jason

---

From: "Guangqing Deng" <[email-masked]>
Date: Thu, 9 May 2013 11:28:14 +0800

>may be unavoidable --> is unavoidable
That's my point too.
>threshold traffic level which, if exceeded, means no test is run.
Totally agree with “listening before talking”. Besides background traffic, 
for those mobile devices like mobile phone and IPAD, the current level of 
battery energy may also need to be detected before measurement.

>description) within a given logical network such as a home network.

Guangqing Deng
CNNIC

---

From: Benoit Claise <[email-masked]>
Date: Mon, 13 May 2013 18:20:16 +0200

> knows the consequences what what he/she is doing.
Yes.
What I don't want is the ISP trying to install MAs on all my home 
devices without so sort of coordination. I should have make that clear. 
Actually, I don't want an ISP to install MAs on any of my devices.
However, if I chose to, then yes, I want to be able to install MA on any 
of my devices.
In that case, we have some more questions:
- what about service parameters, which my devices don't know about?
   Don't care, manual introduction in the MA, or automatic discovery?
- what about multiple interface (IPv4, IPv6, 3G). This is the MIF WG issue
- these should be the same tests as my ISP, so that I could compare. How?
Regarding measurement coordinations on my devices, I agree that this 
might unavoidable

> link from a bandwidth point of view.
For active monitoring, there is also a connection with the MIF WG.
I guess we want to know the access type (IPv4, IPv6, 3G) ... even if the 
measurement test always takes the default/best path, as this can change 
over the time.

Regards, Benoit

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Mon, 13 May 2013 19:52:31 +0200

On Mon, May 13, 2013 at 06:20:16PM +0200, Benoit Claise wrote:

> devices without so sort of coordination. I should have make that
> clear. Actually, I don't want an ISP to install MAs on any of my
> devices.

If your ISP can install software on your private devices, I agree you
have a serious problem. ;-)

> However, if I chose to, then yes, I want to be able to install MA on
> any of my devices.

In the general case, I would expect that your device would not be able
to discover any service parameters unless you have provided the
credentials to the device to access a service parameter service of
your ISP (in which case you have implicitely given your device the
permission to do so).

> - what about multiple interface (IPv4, IPv6, 3G). This is the MIF WG issue

Unclear question - skipped.

> - these should be the same tests as my ISP, so that I could compare. How?

The idea I think is that test may be standardized (in IPPM) including
the set of parameters that can be set. So if you can get all the
details, you should be able to compare results (minus variation
introduced by using different platforms and perhaps slightly different
MA locations). I guess I do not really understand the 'how' question
either.

/js

---

From: Ted Lemon <[email-masked]>
Date: Mon, 13 May 2013 19:05:48 +0000

> Unclear question - skipped.

Suppose you have a device with a 3G and a Wifi interface, and you want to do measurement from that device.   Does the measuring application just blindly connect, and then ask the stack which way the flow is going?   That would be valid.   Does it try to dictate which way the flow will go?   Also valid.   Does it attempt to establish a flow over each interface?   Also valid.   Does it have a UI to allow the user to exclude some interfaces from measurement?   Etc.

---

From: "MORTON JR., ALFRED C (AL)" <[email-masked]>
Date: Mon, 13 May 2013 15:33:57 -0400

> From: Juergen Schoenwaelder
>

All LMAP MAs use the Control and Reporting Protocols which should take advantage
of the new IPPM metric registry for metric selection, and with by-product that the
measurement implementation process can simplified when only registered metrics are
needed. 

MAs running on a user's unknown device introduce some variability as 
Juergen said, but that variation and the specific interface connectivity are 
likely to be key aspects of the user's performance 
(and why the user requested and installed an MA in the first place).

Al

> Sent: Monday, May 13, 2013 1:53 PM
> To: Benoit Claise
> Cc: Lemon Ted; [email-masked]

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Mon, 13 May 2013 22:53:10 +0200

> Suppose you have a device with a 3G and a Wifi interface, and you want to do measurement from that device.   Does the measuring application just blindly connect, and then ask the stack which way th…

The answer likely is 'it depends on what kind of interfaces the OS
provides to an MA and the nature of a specific test and the context in
which a measurement task is executed'.

I know, for example, people measuring how IPv6 access to web services
differs from IPv4 access to services. In this case, a test utilizing
both IPv4 and IPv6 is a rather natural thing.

I see questions here where I think we need to separate the mechanisms
from the policies that can be built with them and concentrate on the
development of mechanisms that allow to implement useful policies. We
should stay away from 'hard coding' certain policies into a solution
(if that is possible at all).

/js

---

From: Ted Lemon <[email-masked]>
Date: Mon, 13 May 2013 21:04:52 +0000

> (if that is possible at all).

Yup.   The reason I suggest considering the questions I asked is not so that you can come up with specific policies, but so that you can give implementors advice about what to do in these situations.   The MIF working group is working on an architecture document that'll talk about these problems in a more general way, but your application is a good example of one where these issues can definitely be expected to come up.   I don't have a strong opinion about what you should do to address this, but Benoit thought you probably ought to do something, and that makes sense to me too.

---

From: "STARK, BARBARA H" <[email-masked]>
Date: Mon, 13 May 2013 21:25:08 +0000

> However, if I chose to, then yes, I want to be able to install MA on any of my
> devices.
> Regarding measurement coordinations on my devices, I agree that this might
> unavoidable

I think I'll try one more time to make a case for being able to supply "inside the home network" devices with service attributes and other info.

Here's what I'm thinking:
We define a DNS-SD service (RFC 6763) that allows devices to discover the url for an xml page on a router. This xml page can provide service attributes (but it doesn't have to; or it can provide some and not others -- whatever it's designed or able to provide), WAN link utilization statistics, LAN link metrics, etc. This doesn't have to be limited to ISP routers; "retail" routers can also have this "service", even though they may not be able to provide service attributes. They could still provide WAN link utilization and LAN link statistics and such.

What would need to be defined: 
(1) The RFC 6763 Service Instance Name (Service Instance Name = <Instance> . <Service> . <Domain>).
(2) xml representation of possible data elements. The service parameter xml will be identical to that which we've already discussed defining for other uses. We would also need to define xml representation of WAN link utilization and LAN link metrics. 

Since privacy rules vary from country to country (some may think it's fine to provide some or all of this data internal to a home network, others may want it secured), it would be possible for the url to be defined so that it can be secure (e.g., https) or not secure (e.g., http). There's probably a fair amount to be said about privacy, but I think it can be said, by providing options and without attempting to legislate privacy in IETF.

Last time I suggested this, there were a number of people who expressed opposition, because they saw no reason to ever provide service attributes to a non-ISP MA inside the home network. But maybe I didn't explain it right. Just because this capability is defined, doesn't mean anyone has to implement it in their router. If an ISP doesn't want to implement this, then they don't implement it. This in no way is intended to prevent or be a substitute for ISP's managing service attributes in other ways, and it's not intended to be forced on anyone. But for those who do think defining such a function might be useful, I think it would be pretty easy to define and implement. And it's not just for "3rd party" MAs, but also for end user MAs that only the end user controls. Like when an end user wants to try to replicate an ISP's measurements.

I'd be willing to take a stab at writing a draft, if there were interest.
Barbara