Subject: Re: [lmap] degrees of Measurement Suppression

From: "Bugenhagen, Michael K" <[email-masked]>
Date: Tue, 10 Dec 2013 22:16:58 +0000

I concur with Barbara -

  Stated a bit differently - If LMAP says put this probe on everything (Part of the introduction statement)...

Then 

1)  They will want a suppression method that allows them to disable it in case of national emergency - saving lives via emergency communication takes priority in their minds.
    So they typically will ask for 2 each Safety breaks.
		1 from the controller
		A second from a element controller or whatever configures the element in the first place.   

   These "MA unavailable" states should also be recorded for transparency sake.

2)  Op's folks like to test - even when things are going bad.. otherwise fault isolation is hard to do.

	So suppression should have 3 levels (again this goes back to everyone has a probe) -
	Green = do as you will (test anything)
	Yellow - don't load anything - restrict testing that will load the network, or kick up CPU cycles ...but allow fault testing (NO Saturation tests, ....)(
	RED - Stop, don't pass go, ... ADHOC testing only

If USE case 1 is the ISP - not adopting those types of Op's requirements would get a significant amount of pushback on the implementation side.

Regards,
Mike

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of STARK, BARBARA H
Sent: Tuesday, November 12, 2013 10:50 AM
To: [email-masked]
Subject: [lmap] degrees of Measurement Suppression

A number of providers have discussed a model of Measurement Suppression that supports more granular degrees of suppression (other than just suppress and don't suppress). Michael Bugenhagen presented the original version of this model, and he said it would be ok if I brought it up to IETF.

Following are highlights of the proposal (with some modifications I've included as a result of additional discussion):

1. Measurement Methods (or perhaps Tasks) have a configurable parameter that indicates whether the Controller operator considers them to be "critical for OAM", "not critical, but not resource intensive", and "not critical and resource intensive".

2. The Controller can specify degrees of Measurement Suppression, which should include: halt all non-critical tasks immediately but allow OAM tasks; halt resource-intensive tasks immediately but allow non-resource-intensive tasks; finish resource intensive tasks but do not start new resource-intensive tasks and allow all non-resource-intensive tasks; allow all tasks.

3. It may also be possible for the unspecified bootstrap mechanism to instruct the MA to suppress measurements. Where multiple channels exist for Measurement Suppression (e.g., Controller and bootstrap), the MA is to use the most restrictive setting.

Barbara

_______________________________________________
lmap mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/lmap

---

From: Steve Miller <[email-masked]>
Date: Tue, 10 Dec 2013 17:39:47 -0500

If we are testing at a rate where leaving the tests running during a national emergency will make a meaningful difference as to whether or not lives are saved, we shouldn't be testing at that rate even during normal operation.  Relying on the ability to turn anything off -- in an active manner at least -- during an emergency seems like something we should avoid: I'd think that anyone who's experienced making a phone call just after a natural disaster would be able to testify as to the flakiness of any communications medium under those circumstances.

   A key (and maybe *the* key) element here is that in many cases the time you must want to hit the big red off switch is the time at which you are least able to communicate with the MAs.

   The other autoshutdown stuff that's been discussed so far would give people the flexibility to implement the only policy that truly seems important: the one that disables measurements after N seconds/minutes/hours/days in which the MA can't reach the controller.  It's been my experience that no matter how important that seems on paper, it ends up being much, much less critical than it seems in the Real World.  Still, leaving this binary on/off level of control in place makes some sense: it's easy to code, it's pretty innocuous, the additional load on the controller infrastructure shouldn't be that bad, and it avoids the "three million old devices hammering some poor destination that belongs to the people who bought some defunct ISP" problem.  It's also been my experience that even that level of control isn't needed and can in fact cause problems of its own, but I recognize that this scenario is a little different.  I still think we should not overcomplicate things.

	-Steve
	(who still needs to read the updated I-Ds, hopefully early next week, sorry)

---

From: Charles Cook <[email-masked]>
Date: Tue, 10 Dec 2013 16:24:08 -0700

Just some thoughts...

I just finished reading the framework draft.  As written, it talks about
large-scale measurements, but does not really define the magnitude of
large-scale.  It also states that end-users can request that
measurements be done on their behalf, and that measurements can
potentially run indefinitely.  Although not mentioned in the framework,
a service provider could offer a measurement service that under normal
conditions runs essentially continuously when the end user is not
transmitting their own data.    In other words, we don't know what the
upper bound is on what can be running in the network as measurements. 
Consequently, we need to assume worst case. 

Using the national emergency analogy, I suggest that during an emergency
is when you want to do everything possible to improve communications as
much as possible.  It is essential that there be a mechanism that
eliminates measurements almost immediately, if necessary.  If
"large-scale" measurements are being done, it probably is not scalable
for a Controller to send an individual message to each MA.  Furthermore,
the emergency may directly affect the Controller and prevent it from
sending messages.  The MA needs an easy way to assess the state of the
network and respond accordingly.  It is relatively easy for the MA to
poll the network before starting a measurement, and then continue to
periodically poll during the measurement.  If there is no response, the
MA assumes there is a network issue and terminates the measurement. 

There probably needs to be more than a go / no-go response from the
network to account for varying degrees of the condition of the network. 
There probably needs to be at least three states, but if there are too
many states it complicates the measurement system.  If there are three
states then an MA can run the measurement, terminate the measurement, or
complete certain measurements and then cease further measurements until
the appropriate network conditions are restored.  MA behavior in
response to network conditions to a certain extent could be
pre-provisioned in the MA by the operational domain owner responsible
for the MA.  There may also be some benefit in dictating behavior in the
Instruction from the Controller.

Charles

-- 

Charles Cook 
Principal Architect
Network
5325 Zuni Street; Suite 224
Denver, CO  80221
Tel:  303.992.8952  Fax:  925.281.0662
[email-masked]

---

From: Steven Miller <[email-masked]>
Date: Wed, 11 Dec 2013 05:42:52 -0500

[ This may not be a marvel of clarity: getting up at 4AM to write emails 
doesn't always end as well as one might hope. (-: ]

    I thought we'd discussed putting in text (and I don't remember if it 
was in the framework draft or elsewhere) that said "you could run a test 
forever but we really recommend that you don't." We'd talked about 
having measurements that might look like they run continuously but are 
just a sequence of "do this this often for a week" measurements, where 
at some point during the week when the MA checks in with the controller 
the controller hands it a different schedule with a different cutoff.

    If a user is requesting a test it seems like we could put a cap on 
that in the same way that we'd put a cap on the tests being requested by 
the "main" user of the MA (e.g., the ISP).  That is, if a user says "run 
this forever", we turn that into "until the end of time, run this for a 
week, renewing every week as part of some normal schedule download."  Or 
as an implementation detail we just don't generally let users run tests 
for further out than some period.

    About the national-emergency thing: I'm saying that, in essence, it 
doesn't matter whether or not you want to do everything in your power to 
improve communication, because once the emergency has started you won't 
be able to reach the MAs reliably.  Also, this seems to me like a 
slippery slope: "do anything possible" seems rather broad.  To take that 
a bit to the extreme for purposes of emphasis, do we somehow mandate 
that CPE all have a "kill Netflix and iTunes Store" switch for use in a 
national emergency?  Our test traffic had better be well down in the 
weeds in terms of being any sort of routine major contributor to load on 
the ISPs hosting MAs or on the Internet in general.

    I think that every bit of complexity we add here makes the overall 
system much more fragile, in ways that have huge potential for 
unintended consequences.  If we need the ability to talk to every MA in 
a hurry, initiated from the controller side, now we have all the fun of 
keeping 2-way communication open through a NAT all of the time (and 
given how most of those schemes do that, that'll be highly unreliable in 
the face of serious network issues, if you need a packet every 20 
seconds or so to keep the association valid in the CPE, 20 seconds is a 
short time when the network is so hammered that you have 90% loss).  If 
large-scale means we have millions of MAs, the load we impose to go try 
to talk to all of them in a hurry to tell them to stop is not entirely 
insignificant.  If we try to be clever about detecting network issues 
and shut down performing some or all measurements, the implementors will 
get the cleverness wrong and we'll lose the results from the 
measurements that are the most interesting -- the ones that tell us that 
the network is having issues.  It's hard to be clever at all, and it's 
even harder to be clever in a device with 8MB of RAM. (-:

    I think that the operations in the existing framework are adequate 
to do the intersection of what we want and what will work in times of 
network stress.

     -Steve

---

From: "Bugenhagen, Michael K" <[email-masked]>
Date: Wed, 11 Dec 2013 14:20:37 +0000

Steve -

    Operational design groups have these frameworks historically - and they control the deployment of test systems so adoption issues can and do occur with these types of systems that can load the network.
I think it's fairly simple to take away those concerns by addressing them vs. discarding what may be a legitimate issue for and ISP to adopt the framework.        

If we can't reach consensus on the hard requirement I recommend one of two paths.
1) Add those added safety measures but make them optional.
And or 
2) Partition the tests (which I think we're doing) so that they can be adopted into existing test control schemas that fit the ISP's need.

My concern on the way it is currently drafted -
  I'm just afraid that the current control schema might miss the target for the #1 use case for all ISP's, which "large" somewhat infers all ISP's would find the control framework suitable.
I understand the added complexity argument, but if it's required to meet hard operational criteria (and this has been discussed else where) then it's probably worth the effort to ensure that this work is reference-able and revelent to all.     There are a few more Key blind spots in this framework as well that make it's current form and longevity suspect.

To resolve the "is this a real op's issue"  -
Let's see if we can't get some IETF ISP members to ping their OP's groups on if they restrict load testing the network for any reason - if yes.. then we need to look at phases.
BTW - every Op's group I've been in has situations where they don't from the customer level on UP... aka you don't load a congested customer unless you're taking them out of service and they know it, and you don't stress a "trunk" of shared path that's severely congested and in trouble either through fault or due un-usual circumstance.   (an earthquake drops building in a city and you want every bit available to be there for use, not test traffic).

Cheers - 
Mike

-----Original Message-----
From: Steve Miller [mailto:[email-masked]] 
Sent: Tuesday, December 10, 2013 4:40 PM
To: Bugenhagen, Michael K
Cc: 'STARK, BARBARA H'; [email-masked]'
Subject: Re: [lmap] degrees of Measurement Suppression

	-Steve

---

From: "Sharam Hakimi" <[email-masked]>
Date: Wed, 11 Dec 2013 11:14:36 -0500

Mike,
There two ways of having this control which would work in the system and
do what you would like. They are

	1: The MA device MUST stop testing if it has not heard from the
MC in a designated time  ( we had talked about 		putting this in
the document.

	2: In an emergency the MC can issue a BLANK test schedule that
would override whatever the MA is scheduled to do.
		( and if MA cannot receive this it will automatically
stop)

Thanks,
Sharam 

-----Original Message-----
From: lmap [mailto:[email-masked]] On Behalf Of Bugenhagen,
Michael K
Sent: Wednesday, December 11, 2013 9:21 AM
To: 'Steve Miller'
Cc: 'STARK, BARBARA H'; [email-masked]'
Subject: Re: [lmap] degrees of Measurement Suppression

Steve -

    Operational design groups have these frameworks historically - and
they control the deployment of test systems so adoption issues can and
do occur with these types of systems that can load the network.
I think it's fairly simple to take away those concerns by addressing
them vs. discarding what may be a legitimate issue for and ISP to adopt
the framework.        

If we can't reach consensus on the hard requirement I recommend one of
two paths.
1) Add those added safety measures but make them optional.
And or 
2) Partition the tests (which I think we're doing) so that they can be
adopted into existing test control schemas that fit the ISP's need.

My concern on the way it is currently drafted -
  I'm just afraid that the current control schema might miss the target
for the #1 use case for all ISP's, which "large" somewhat infers all
ISP's would find the control framework suitable.
I understand the added complexity argument, but if it's required to meet
hard operational criteria (and this has been discussed else where) then
it's probably worth the effort to ensure that this work is
reference-able and revelent to all.     There are a few more Key blind
spots in this framework as well that make it's current form and
longevity suspect.

To resolve the "is this a real op's issue"  -
Let's see if we can't get some IETF ISP members to ping their OP's
groups on if they restrict load testing the network for any reason - if
yes.. then we need to look at phases.
BTW - every Op's group I've been in has situations where they don't from
the customer level on UP... aka you don't load a congested customer
unless you're taking them out of service and they know it, and you don't
stress a "trunk" of shared path that's severely congested and in trouble
either through fault or due un-usual circumstance.   (an earthquake
drops building in a city and you want every bit available to be there
for use, not test traffic).

Cheers - 
Mike

-----Original Message-----
From: Steve Miller [mailto:[email-masked]] 
Sent: Tuesday, December 10, 2013 4:40 PM
To: Bugenhagen, Michael K
Cc: 'STARK, BARBARA H'; [email-masked]'
Subject: Re: [lmap] degrees of Measurement Suppression

   If we are testing at a rate where leaving the tests running during a
national emergency will make a meaningful difference as to whether or
not lives are saved, we shouldn't be testing at that rate even during
normal operation.  Relying on the ability to turn anything off -- in an
active manner at least -- during an emergency seems like something we
should avoid: I'd think that anyone who's experienced making a phone
call just after a natural disaster would be able to testify as to the
flakiness of any communications medium under those circumstances.

   A key (and maybe *the* key) element here is that in many cases the
time you must want to hit the big red off switch is the time at which
you are least able to communicate with the MAs.

   The other autoshutdown stuff that's been discussed so far would give
people the flexibility to implement the only policy that truly seems
important: the one that disables measurements after N
seconds/minutes/hours/days in which the MA can't reach the controller.
It's been my experience that no matter how important that seems on
paper, it ends up being much, much less critical than it seems in the
Real World.  Still, leaving this binary on/off level of control in place
makes some sense: it's easy to code, it's pretty innocuous, the
additional load on the controller infrastructure shouldn't be that bad,
and it avoids the "three million old devices hammering some poor
destination that belongs to the people who bought some defunct ISP"
problem.  It's also been my experience that even that level of control
isn't needed and can in fact cause problems of its own, but I recognize
that this scenario is a little different.  I still think we should not
overcomplicate things.

	-Steve
	(who still needs to read the updated I-Ds, hopefully early next
week, sorry)

On Tue, Dec 10, 2013 at 10:16:58PM +0000, Bugenhagen, Michael K wrote:
> 1)  They will want a suppression method that allows them to disable it
in case of national emergency - saving lives via emergency communication
takes priority in their minds.
> 		A second from a element controller or whatever
configures the element in the first place.   
> 	Yellow - don't load anything - restrict testing that will load
the network, or kick up CPU cycles ...but allow fault testing (NO
Saturation tests, ....)(
> If USE case 1 is the ISP - not adopting those types of Op's
requirements would get a significant amount of pushback on the
implementation side.
> A number of providers have discussed a model of Measurement
Suppression that supports more granular degrees of suppression (other
than just suppress and don't suppress). Michael Bugenhagen presented the
original version of this model, and he said it would be ok if I brought
it up to IETF.
> 
> Following are highlights of the proposal (with some modifications I've
included as a result of additional discussion):
> 
> 1. Measurement Methods (or perhaps Tasks) have a configurable
parameter that indicates whether the Controller operator considers them
to be "critical for OAM", "not critical, but not resource intensive",
and "not critical and resource intensive".
> 
> 2. The Controller can specify degrees of Measurement Suppression,
which should include: halt all non-critical tasks immediately but allow
OAM tasks; halt resource-intensive tasks immediately but allow
non-resource-intensive tasks; finish resource intensive tasks but do not
start new resource-intensive tasks and allow all non-resource-intensive
tasks; allow all tasks.
> 
> 3. It may also be possible for the unspecified bootstrap mechanism to
instruct the MA to suppress measurements. Where multiple channels exist
for Measurement Suppression (e.g., Controller and bootstrap), the MA is
to use the most restrictive setting.

---

From: "Bugenhagen, Michael K" <[email-masked]>
Date: Wed, 11 Dec 2013 16:46:37 +0000

Sharam -

    Those are some options -

But again the Op's guys are responsible for keeping the customer "not impacted" and the network alive.

So they will want more than "on / off" for probe functions - they will want, load test (Service activation test), OAM / fault test, and Stop all testing controls.
	- Your suggestion doesn't cover the 3 states required for Op's...

  Let's put it this way - the Op's teams have either already driven this into probes, are as asking you probe makers to do this.
IF they have an option they are going to pick the probe that has the states that match their operational practices.
It always goes to the needs of the customer, and when we are talking about ISP's - that's the Op's organization.

Sorry if I appear a bit jaded on the subject, but in the past I've had to explain why some of these new technologies can't be tested and don't have these features when they break.
Not an enjoyable endeavor whatsoever...

   Otherwise stated - I do think it's worthwhile to have those 3 levels of testing control (or suppression), and 2 ways to shut it down (EMS or element manager, and Measurement Controller) available so the Op's guys can pass it without question. 

Cheers,
Mike     

-----Original Message-----
From: Sharam Hakimi [mailto:[email-masked]] 
Sent: Wednesday, December 11, 2013 10:15 AM
To: Bugenhagen, Michael K; Steve Miller
Cc: STARK, BARBARA H; [email-masked]
Subject: RE: [lmap] degrees of Measurement Suppression

Mike,
There two ways of having this control which would work in the system and do what you would like. They are

	2: In an emergency the MC can issue a BLANK test schedule that would override whatever the MA is scheduled to do.
		( and if MA cannot receive this it will automatically
stop)

Thanks,
Sharam 

-----Original Message-----
From: lmap [mailto:[email-masked]] On Behalf Of Bugenhagen, Michael K
Sent: Wednesday, December 11, 2013 9:21 AM
To: 'Steve Miller'
Cc: 'STARK, BARBARA H'; [email-masked]'
Subject: Re: [lmap] degrees of Measurement Suppression

Steve -

    Operational design groups have these frameworks historically - and they control the deployment of test systems so adoption issues can and do occur with these types of systems that can load the network.
I think it's fairly simple to take away those concerns by addressing them vs. discarding what may be a legitimate issue for and ISP to adopt
the framework.        

Cheers - 
Mike

-----Original Message-----
From: Steve Miller [mailto:[email-masked]] 
Sent: Tuesday, December 10, 2013 4:40 PM
To: Bugenhagen, Michael K
Cc: 'STARK, BARBARA H'; [email-masked]'
Subject: Re: [lmap] degrees of Measurement Suppression

	-Steve

---

From: "Sharam Hakimi" <[email-masked]>
Date: Wed, 11 Dec 2013 12:12:29 -0500

Mike,
My comments was to respond to your emergency case point. The other tests
are part of the normal test operation covered by the use cases.

Thanks,
Sharam

-----Original Message-----
From: Bugenhagen, Michael K
[mailto:[email-masked]] 
Sent: Wednesday, December 11, 2013 11:47 AM
To: Sharam Hakimi; 'Steve Miller'
Cc: 'STARK, BARBARA H'; [email-masked]'
Subject: RE: [lmap] degrees of Measurement Suppression

Sharam -

    Those are some options -

But again the Op's guys are responsible for keeping the customer "not
impacted" and the network alive.

So they will want more than "on / off" for probe functions - they will
want, load test (Service activation test), OAM / fault test, and Stop
all testing controls.
	- Your suggestion doesn't cover the 3 states required for
Op's...

  Let's put it this way - the Op's teams have either already driven this
into probes, are as asking you probe makers to do this.
IF they have an option they are going to pick the probe that has the
states that match their operational practices.
It always goes to the needs of the customer, and when we are talking
about ISP's - that's the Op's organization.

Sorry if I appear a bit jaded on the subject, but in the past I've had
to explain why some of these new technologies can't be tested and don't
have these features when they break.
Not an enjoyable endeavor whatsoever...

   Otherwise stated - I do think it's worthwhile to have those 3 levels
of testing control (or suppression), and 2 ways to shut it down (EMS or
element manager, and Measurement Controller) available so the Op's guys
can pass it without question. 

Cheers,
Mike     

-----Original Message-----
From: Sharam Hakimi [mailto:[email-masked]] 
Sent: Wednesday, December 11, 2013 10:15 AM
To: Bugenhagen, Michael K; Steve Miller
Cc: STARK, BARBARA H; [email-masked]
Subject: RE: [lmap] degrees of Measurement Suppression

Mike,

Thanks,
Sharam 

-----Original Message-----

Steve -

Cheers - 
Mike

-----Original Message-----
From: Steve Miller [mailto:[email-masked]] 
Sent: Tuesday, December 10, 2013 4:40 PM
To: Bugenhagen, Michael K
Cc: 'STARK, BARBARA H'; [email-masked]'
Subject: Re: [lmap] degrees of Measurement Suppression

	-Steve

---

From: Charles Cook <[email-masked]>
Date: Wed, 11 Dec 2013 14:49:06 -0700

Summarizing this email chain:

·         Although not prohibited, measurements without end times (ones
that could run forever) should be discouraged.  (I do think I saw a
sentence in the draft along these lines.)

·         It may be beneficial to place caps on end-user requested
measurements.

·         In the absence of a crystal ball, it is important to have a
measurement system that will scale, and be able to successfully turn off
measurement activities should there be a need to do so.

·         We don't want to literally do "anything possible" to improve
communications in an emergency.  But we do want to have
simple-to-implement mechanisms to control measurement traffic that will
help improve overall non-measurement communications.

·         We want the measurement system to have sufficient flexibility
that ISP Operations can use it to conduct various exercises in their own
network.

·         We don't want to make the measurement system complex.

·         Measurement traffic should be a small fraction, "way down in
the weeds", of the overall traffic in the network.

·         In an emergency, it may not be possible to reliably reach an MA.

Possible Suppression Solutions:

1)      Continual (periodic) 2-way communications between the
Measurement Controller and the MA.  This was given as an example, and it
was pointed out that in an emergency this approach may be unreliable.

2)      A two-state on/off mechanism that is controlled by the
Measurement Controller sending Instructions to the MA.  This has the
issue that communication between the Measurement Controller and MA may
not be reliable in an emergency.  Additionally, this requires the
Measurement Controller to send a message to each MA that has a schedule
that permits it to participate in measurements at the time of the emergency.

3)      A two-state on/off mechanism that is controlled by a timeout at
the MA if the MA has not heard anything from the Measurement
Controller.  This results in the MA eventually terminating measurements,
but it requires that the Measurement Controller to periodically send
keep-alive messages when the network is not in an emergency state. 
Also, the responsiveness of the network depends on what the Measurement
Controller keep-alive period is set to, and the length that the timer in
the MA is set to.

4)      A two-state on/off mechanism where the MA periodically checks
the state of the network and conducts behavior based on the state of the
network.  If the MA is unable to retrieve network state information, it
assumes the network is in an emergency state and acts accordingly.  The
state of the network could be provided by the Measurement Controller, a
server in the Operational Domain that is responsible for the MAs
belonging to that Operational Domain, or both.  The Operational Domain
can provision the MA as appropriate (frequency of network state checks,
address of network state server(s)).

I don't think that any of these solutions would preclude a Measurement
Controller sending a blank Instruction to the MA to shut down
measurements (assuming the network is able to reliably deliver the
Instruction to the MA). 

I am sure there are other solutions.  Of the ones listed so far, I
prefer #4.

There is also a question of whether two network states are sufficient. 
We don't want too many states because that results in additional
complexity.  Possible solutions include:

1)      Two states:  on (continue measurements), and off (cease
measurements immediately).

2)      Three states:  on (continue measurements), off (cease
measurements immediately), and limited (limit measurements per
parameters in the Instruction sent by the Measurement Controller). 

3)      N states:  ???

4)      Two states mandated, but additional states allowed.  Simple MAs
will support two states (on and off).  More advanced MAs can support
more elaborate responses to a network emergency.

Again, I am sure there are other solutions.  Of the ones listed so far,
I prefer #2 and #4.

Charles

-- 

Charles Cook 
Principal Architect
Network

---

From: <[email-masked]>
Date: Thu, 12 Dec 2013 09:06:36 +0000

> There are a few more Key blind

Please could you let us know as soon as possible what these are. As far as I know, suppression is the only open issue.

Thanks
phil

---

From: "Bugenhagen, Michael K" <[email-masked]>
Date: Fri, 13 Dec 2013 14:56:37 +0000

>From: [email-masked] [mailto:[email-masked]] 
>Sent: Thursday, December 12, 2013 3:07 AM
>To: Bugenhagen, Michael K; [email-masked]
>Cc: [email-masked]; [email-masked]
**********>Please could you let us know as soon as possible what these are. As far as I know, suppression is the only open issue.
>Phil
-----------------------------------------

Response starts here -

Ok -  3 each items I think we could use some more clear visionary planks on in terms of the state of the industry and the goals of the project (scope) 
** summary statement at the bottom 
  - please note I don't disagree with the current path, I think we need to ensure strong focus on the project concepts that will make the solution as relevant as possible (just like the rest of us).

So sharing some op's experience background.... and some hopefully reasonable observations
As you read this please consider I have a Strategic implementation consultant hat on - as to how do the roadmaps of the ISP, vendor, and our goal of low cost collide....

-----------

1) The Collapsing down of the number of MA's (aka ISP challenges with multiple probes and test solutions)
	Vendors produce testing solutions follow "operationalization" timelines of the ISP services.
	Eg.  General IP testing solutions appeared first, then VoIP testing solutions, then Video Over IP ....
			Different Operation centers "owned" different test solutions (unlike TDM where a single solution sufficed).
			Given there is always a "best in bread" test solution, and each vendor tends to target their market and focus their development this naturally leads to multiple test solutions.

The hitch -
	- Maturity wise ISP's who offer Broadband, VoIP and Video need MA's to test all of those services on the same device - but MA's aren't designed to co-exist which leads to implementation issues (collision space).
	
	Ideal things to address 
		-  Cost of 2-3 MA's and test systems vs. a single test system (the risk in LMAP is the "yet another test system") cost
		-  Portability of test so if an ISP is collapsing 2-3 test systems into one (ideal solution to lower cost) ** WE ARE DOING THIS...

	 My personal recommendation For #1- Cost is relevant so - focus a bit more on standardizing portable test that work under any controller / MA combination as this will lower cost vs. them being required to be under a new testing system.  Otherwise stated, ensure the sticky-ness of the solution.
------------------

2) New small form factor test devices (aka available Vendor solutions emerging on the scene)
	If we observe test solution vendors a good amount of them are now providing small form factor test engines due to the lower cost CPU's in Modems and set top boxes NOT having enough power to run probes.
	This is one heck of a strong signal that the multiple probe approach is failing....

	Currently we identify this as possible (hardware to use as a probe) but we don't define or categorize how that would work.
	This is a test solution evolution path where by some standard modularity or "abstraction model" involves a framework whereby a small form factor hardware is added in a standard fashion to an element.
	Given the industry headed this way we may want to consider figuring out how to frame that as a standard so those devices can be swapped out and support this testing (avoid proprietary nature of the implementation).

	My recommendation for #2 - This infers that each test we define needs to have a general abstraction model for the test that each vendor can implement (Standard MIB, UML per test) to enable ISP's to swap out vendor hardware yet	 have equivalent tests.   

------------

3) NFV (Network function virtualization - aka 2-3 year if not less frameworks)

	Aka Cloud, here we see that NFV under ETSI is actively defining a framework where by modems, STB, .... become virtualized elements, and can support virtualization.
	In this genre probes become virtualized applications much like the Android, and Iphone app's today where one can download them.

	Putting on the visionary goggles - that means that one day it would be reasonable for a ISP to provide a virtualized application one could download to your modem or mobile phone to conduct the testing.
	It also means that Tight integration between the Service attributes and the probe are now possible.
	** HUGE amount of man-hour savings here.
	
	Recommendations for #3 - NFV means stating that the MA becomes something that can be run in an virtualized environment as a application from the Infrastructure provider (STB, modem, ... and tweaked for the ISP)
			That is if you want lowest cost...

------

Summary statement --- 
Those are the big ones that have me concerned that all the LMAP controller work could result in Yet another Testing System (YATS) - that's NOT a bad thing, and may be required - but we should be aware of the nature of matter that we're asking for low or no cost, but building something that costs more unless we do the portability to match the industry roadmap. 
At the end of the day If I'm an ISP offering and testing services, to get the low / no cost as an ISP I'm going to have to wind the tests into my current deck of testing control vs. building another system

---

From: "Bugenhagen, Michael K" <[email-masked]>
Date: Fri, 13 Dec 2013 15:02:52 +0000

When I said "We are doing this" under test portability I'm saying that given the test definitions are under the registry they should be portable....

-----Original Message-----
From: lmap [mailto:[email-masked]] On Behalf Of Bugenhagen, Michael K
Sent: Friday, December 13, 2013 8:57 AM
To: [email-masked]'; [email-masked]'
Cc: [email-masked]'; [email-masked]'
Subject: Re: [lmap] degrees of Measurement Suppression

Response starts here -

So sharing some op's experience background.... and some hopefully reasonable observations As you read this please consider I have a Strategic implementation consultant hat on - as to how do the roadmaps of the ISP, vendor, and our goal of low cost collide....

-----------

The hitch -
	
	Ideal things to address 

------------

	

------

Summary statement ---