Subject: [lmap] preliminary agenda - please comment!

From: "Romascanu, Dan (Dan)" <[email-masked]>
Date: Thu, 3 Jul 2014 12:21:57 +0000

I uploaded a preliminary agenda for the LMAP meeting in Toronto at IETF-90.

It is available at http://www.ietf.org/proceedings/90/agenda/agenda-90-lmap.

Please comment!

Regards,

Dan

---

From: marcelo bagnulo braun <[email-masked]>
Date: Thu, 03 Jul 2014 15:52:59 +0200

looks good to me

El 03/07/14 14:21, Romascanu, Dan (Dan) escribió:
> It is available at 
> http://www.ietf.org/proceedings/90/agenda/agenda-90-lmap 
> <http://www.ietf.org/proceedings/90/agenda/agenda-90-lmap>.
>
> Please comment!
>
> Regards,
>
> Dan
>
>
>
> _______________________________________________
> lmap mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/lmap

---

From: "Bugenhagen, Michael K" <[email-masked]>
Date: Thu, 3 Jul 2014 14:56:29 +0000

Dan,

   I don't know if we've been actively tracking "gaps" in a list anywhere but that might be work setting aside at least 15 minutes for in the agenda...

See you in Toronto!

Cheers,
Mike

From: lmap [mailto:[email-masked]] On Behalf Of Romascanu, Dan (Dan)
Sent: Thursday, July 03, 2014 7:22 AM
To: '<[email-masked]>'
Subject: [lmap] preliminary agenda - please comment!

Please comment!

Regards,

Dan

---

From: "Romascanu, Dan (Dan)" <[email-masked]>
Date: Thu, 3 Jul 2014 17:22:33 +0000

Hi Michael,

Can you clarify what kind of gaps you are referring to?

Thanks and Regards,

Dan

From: Bugenhagen, Michael K [mailto:[email-masked]]
Sent: Thursday, July 03, 2014 5:56 PM
To: Romascanu, Dan (Dan); '<[email-masked]>'
Subject: RE: preliminary agenda - please comment!

Dan,

See you in Toronto!

Cheers,
Mike

From: lmap [mailto:[email-masked]] On Behalf Of Romascanu, Dan (Dan)
Sent: Thursday, July 03, 2014 7:22 AM
To: '<[email-masked]<mailto:[email-masked]>>'
Subject: [lmap] preliminary agenda - please comment!

Please comment!

Regards,

Dan

---

From: "Bugenhagen, Michael K" <[email-masked]>
Date: Thu, 3 Jul 2014 17:40:16 +0000

A list of the "scope" to work underway.

Example - in the information model we have identified "Objects" involved with testing such as cross traffic, but we haven't defined the "behavior" of the MA during that event.
So although we have a "registry" the registry itself is based upon existing test which don't describe how test behavior in the presence of live network and service states that can change the expected outcome of the test.

Therefore on a large part we have not met the target goal of designing a framework that can handle live testing in a in-service network without describing the testing states, and methods required to be addressed in that environment.
A litmus test of this is "if one were to have different behaviors to those factors do the test results vary" - the answer here I believe is yes - which directly infers to meet the goal of a framework that can support apples to apples test results in a live network we need to address that issue, and given we haven't addressed it yet - well, that's a gap.

I may have mentioned that before, but I haven't seen any of the current "framework" contributions propose solutions, even though we are saying the framework is becoming more mature... when it does those types of gaps should not exist IMO.

So I feel they warrant being called out at this point...

There may be more, it's just one I don't see being worked, but it's a critical one as well - so we should be trying to close the big gaps before continued work on the small stuff, at least IMO it is...

Mike

From: Romascanu, Dan (Dan) [mailto:[email-masked]]
Sent: Thursday, July 03, 2014 12:23 PM
To: Bugenhagen, Michael K; '<[email-masked]>'
Subject: RE: preliminary agenda - please comment!

Hi Michael,

Thanks and Regards,

Dan

From: Bugenhagen, Michael K [mailto:[email-masked]]
Sent: Thursday, July 03, 2014 5:56 PM
To: Romascanu, Dan (Dan); '<[email-masked]<mailto:[email-masked]>>'
Subject: RE: preliminary agenda - please comment!

Dan,

See you in Toronto!

Cheers,
Mike

Please comment!

Regards,

Dan

---

From: "Romascanu, Dan (Dan)" <[email-masked]>
Date: Thu, 3 Jul 2014 17:44:50 +0000

Why not discuss such issues as part of the comments to the framework and information model Internet-Drafts?

Time in meetings is expensive, and I prefer to focus on solving the open issues on the Internet-Drafts. What you call 'gaps' seem to belong here.

Do I miss anything?

Regards,

Dan

From: Bugenhagen, Michael K [mailto:[email-masked]]
Sent: Thursday, July 03, 2014 8:40 PM
To: Romascanu, Dan (Dan); '<[email-masked]>'
Subject: RE: preliminary agenda - please comment!

Mike

From: Romascanu, Dan (Dan) [mailto:[email-masked]]
Sent: Thursday, July 03, 2014 12:23 PM
To: Bugenhagen, Michael K; '<[email-masked]<mailto:[email-masked]>>'
Subject: RE: preliminary agenda - please comment!

Hi Michael,

Thanks and Regards,

Dan

Dan,

See you in Toronto!

Cheers,
Mike

Please comment!

Regards,

Dan

---

From: "Bugenhagen, Michael K" <[email-masked]>
Date: Thu, 3 Jul 2014 17:54:21 +0000

I think the conversation on the reflector servers that purpose fairly well... but I also expect the group to acknowledge it as we progress the document.
I will very likely get with the authors to see if they have any approach of closing this in mind - and if not make a proposal through one of them provided they bother to do so.

   My problem is time at the moment so I haven't had time to reach out to the authors and close this gap via collaborative contributions.

But on the subject of  project completeness and viability and  I believe this is a very legitimate issue - so I wanted to raise the awareness of this one,  and other issues before parts of the team start thinking the work is done.
i.e. -    I should have time to get with the different leads next week to see if it can't be closed appropriately in the given framework without tweaking it...  my fear is that this may interfere and cause rework in the current drafts, but that's life....

  So yep, if you don't want as a group to discuss it we can handle it off line.

M

From: Romascanu, Dan (Dan) [mailto:[email-masked]]
Sent: Thursday, July 03, 2014 12:45 PM
To: Bugenhagen, Michael K; '<[email-masked]>'
Subject: RE: preliminary agenda - please comment!

Do I miss anything?

Regards,

Dan

Mike

Hi Michael,

Thanks and Regards,

Dan

Dan,

See you in Toronto!

Cheers,
Mike

Please comment!

Regards,

Dan

---

From: "MORTON, ALFRED C (AL)" <[email-masked]>
Date: Thu, 3 Jul 2014 13:55:05 -0400

I think the details of testing states and policies related to the In-service
environment may be measurement study-specific decisions, and then they should
appear in regional Implementer's guides and the governing body's Characterization
Plan.

In other words, the Framework is not a region-specific Characterization Plan,
and the Charter for the initial LMAP work sets the scope of the Framework,
drawing the line which may exclude gaps that the Characterization Plan would fill.

Al

From: lmap [mailto:[email-masked]] On Behalf Of Romascanu, Dan (Dan)
Sent: Thursday, July 03, 2014 1:45 PM
To: Bugenhagen, Michael K; '<[email-masked]>'
Subject: Re: [lmap] preliminary agenda - please comment!

Do I miss anything?

Regards,

Dan

Mike

Hi Michael,

Thanks and Regards,

Dan

Dan,

See you in Toronto!

Cheers,
Mike

Please comment!

Regards,

Dan

---

From: "Bugenhagen, Michael K" <[email-masked]>
Date: Thu, 3 Jul 2014 18:02:25 +0000

Al,

    Then we may have another BBF to IETF LMAP gap.

The charter there says to define an Performance Metric standard for measuring Broadband performance - in fact it's the first sentence of the scope in the BBF project.
Regional or not a universal metric on a live network needs to include method for handling environmental impacts such as terms of service.

      I'm OK with LMAP making common pipe information objects, if the scope "line" is drawn there - but the projects origin scope may have been broader, I definitely know it was on the BBF side.

Let me talk with Charles Cook and Ken Ko from the BBF side of the house.

If nothing else  I think LMAP and IPPM should coordinate with BBF on who's doing what - I originally assumed the IETF would handle environmental and service state issues as part of the testing framework via writing some MA standard behavior standards.
IF this is not the case, then the BBF may have to tackle that one ... and the IETF simply put a "out of scope" statement that it's an issue to be addressed by the owner of the test platform.

Thoughts ?
Mike

From: MORTON, ALFRED C (AL) [mailto:[email-masked]]
Sent: Thursday, July 03, 2014 12:55 PM
To: Romascanu, Dan (Dan); Bugenhagen, Michael K; '<[email-masked]>'
Subject: RE: preliminary agenda - please comment!

Al

Do I miss anything?

Regards,

Dan

Mike

Hi Michael,

Thanks and Regards,

Dan

Dan,

See you in Toronto!

Cheers,
Mike

Please comment!

Regards,

Dan

---

From: "Romascanu, Dan (Dan)" <[email-masked]>
Date: Thu, 3 Jul 2014 18:09:36 +0000

Hi Michael,

See in-line.

Regards,

Dan

From: Bugenhagen, Michael K [mailto:[email-masked]]
Sent: Thursday, July 03, 2014 9:02 PM
To: 'MORTON, ALFRED C (AL)'; Romascanu, Dan (Dan); '<[email-masked]>'
Subject: RE: preliminary agenda - please comment!

Al,

DR:  This is not something that is within the scope of the current charter of LMAP.

What we do have now in what coordination with the BBF is concerned is:

Ø  The LMAP working group will coordinate with other standards bodies working in this area (e.g., BBF, IEEE 802.16, ETSI) regarding the information model, and with other IETF working groups in the areas of data models, protocols, multiple interface management, and measurement of performance metrics.

DR: In which of the document (or documents) you see the need for such a statement?

Thoughts ?
Mike

From: MORTON, ALFRED C (AL) [mailto:[email-masked]]
Sent: Thursday, July 03, 2014 12:55 PM
To: Romascanu, Dan (Dan); Bugenhagen, Michael K; '<[email-masked]<mailto:[email-masked]>>'
Subject: RE: preliminary agenda - please comment!

Al

Do I miss anything?

Regards,

Dan

Mike

Hi Michael,

Thanks and Regards,

Dan

Dan,

See you in Toronto!

Cheers,
Mike

Please comment!

Regards,

Dan

---

From: "Bugenhagen, Michael K" <[email-masked]>
Date: Thu, 3 Jul 2014 19:43:32 +0000

I'm good with working together -

  However this means that the method used by the BBF can't break the IETF framework or the two will end up with Different MA behaviors and not aligned.
Aka.  Two MA's with different Behaviors are "different MA's" all together ?

Also I'm concerned the solution will involve protocol exchange messages which are in the IETF domain - and frankly I'd rather have an open conversation with all the IETF folks to solve this one ....

But I'm open minded so whatever way we get to the goal is fine, it might simply take a bit longer if we can't extend the IETF framework at the BBF to support the Environmental "stuff"...

Happy 4th (reminder the US folks are off on Friday)...

Mike

From: Romascanu, Dan (Dan) [mailto:[email-masked]]
Sent: Thursday, July 03, 2014 1:10 PM
To: Bugenhagen, Michael K; 'MORTON, ALFRED C (AL)'; '<[email-masked]>'
Subject: RE: preliminary agenda - please comment!

Hi Michael,

See in-line.

Regards,

Dan

From: Bugenhagen, Michael K [mailto:[email-masked]]
Sent: Thursday, July 03, 2014 9:02 PM
To: 'MORTON, ALFRED C (AL)'; Romascanu, Dan (Dan); '<[email-masked]<mailto:[email-masked]>>'
Subject: RE: preliminary agenda - please comment!

Al,

Thoughts ?
Mike

Al

Do I miss anything?

Regards,

Dan

Mike

Hi Michael,

Thanks and Regards,

Dan

Dan,

See you in Toronto!

Cheers,
Mike

Please comment!

Regards,

Dan

---

From: "Romascanu, Dan (Dan)" <[email-masked]>
Date: Thu, 3 Jul 2014 20:02:21 +0000

Please do not mis-understand me - I am in favor of coordinating and dividing work. I am just concerned about using at best the time in the meeting - and this thread is about the meeting agenda! We can discuss coordination and gaps on the list, and in more or less formal manner, but if we allocate time in the meeting let us just make sure we prepare best and use the meeting time in an efficient manner.

Regards,

Dan

From: Bugenhagen, Michael K [mailto:[email-masked]]
Sent: Thursday, July 03, 2014 10:44 PM
To: Romascanu, Dan (Dan); 'MORTON, ALFRED C (AL)'; '<[email-masked]>'
Subject: RE: preliminary agenda - please comment!

Mike

From: Romascanu, Dan (Dan) [mailto:[email-masked]]
Sent: Thursday, July 03, 2014 1:10 PM
To: Bugenhagen, Michael K; 'MORTON, ALFRED C (AL)'; '<[email-masked]<mailto:[email-masked]>>'
Subject: RE: preliminary agenda - please comment!

Hi Michael,

See in-line.

Regards,

Dan

Al,

Thoughts ?
Mike

Al

Do I miss anything?

Regards,

Dan

Mike

Hi Michael,

Thanks and Regards,

Dan

Dan,

See you in Toronto!

Cheers,
Mike

Please comment!

Regards,

Dan

---

From: "STARK, BARBARA H" <[email-masked]>
Date: Sat, 5 Jul 2014 16:12:43 +0000

The liaison statement from BBF said everything that BBF officially agreed to say to IETF. Note that there is no mention from BBF regarding any "gaps" that are officially perceived by "BBF" or any requests from BBF for IETF to consider additional deliverables.

Anyone is welcome to provide their personal opinions regarding perceived "gaps" and regarding what organization is best positioned to do what work. But it's very, very important to recognize these as personal opinions. Please do not treat these personal opinions as official BBF statements. It would also be nice if these opinions were expressed in a way that made this clear.

IMO, it will still be needed to define somewhere the exact metrics and measurement methods for US-based measurement of broadband access services. But I see no need to try to drive this internationally, and do not think either BBF or IETF are well-suited to define this for the US. A US group that has strong technical representation from service providers, equipment manufacturers, and consumer advocates would probably be better suited to doing this. I can't speak for other regions.

IMO, the framework, information-model, and registry are the most important deliverables needed from IETF.
Barbara

From: lmap [mailto:[email-masked]] On Behalf Of Romascanu, Dan (Dan)
Sent: Thursday, July 03, 2014 2:10 PM
To: Bugenhagen, Michael K; MORTON JR., AL; '<[email-masked]>'
Subject: Re: [lmap] preliminary agenda - please comment!

Hi Michael,

See in-line.

Regards,

Dan

Al,

Thoughts ?
Mike

Al

Do I miss anything?

Regards,

Dan

Mike

Hi Michael,

Thanks and Regards,

Dan

Dan,

See you in Toronto!

Cheers,
Mike

Please comment!

Regards,

Dan

---

From: <[email-masked]>
Date: Mon, 7 Jul 2014 09:22:55 +0100

Before we invent a BBF/IETF division, I just like to clarify that the Information Model and Framework does support what I think Michael is asking for.

It is assumed that the test or scheduler has access to a library or local data model that it can use to query cross traffic. This is going to be device specific - e.g. this could be supported within the BBF data models.

The task configuration, either retrieved from the registry of configured on the MA can specify any number of open parameters, including how to test cross traffic before execution. After execution the IM supports the reporting of the simultaneous cross-traffic in the measurement results.

Trevor.

From: lmap [mailto:[email-masked]] On Behalf Of Bugenhagen, Michael K
Sent: 03 July 2014 19:02
To: 'MORTON, ALFRED C (AL)'; 'Romascanu, Dan (Dan)'; '<[email-masked]>'
Subject: Re: [lmap] preliminary agenda - please comment!

Al,

Thoughts ?
Mike

Al

Do I miss anything?

Regards,

Dan

Mike

Hi Michael,

Thanks and Regards,

Dan

Dan,

See you in Toronto!

Cheers,
Mike

Please comment!

Regards,

Dan

---

From: Sharam Hakimi <[email-masked]>
Date: Mon, 7 Jul 2014 15:51:18 +0000

Mike,
IMHO, I think we need to separate post processing necessities from methods of measurement. Terms of service, as an example, does not change how the test is performed on a predefined framework , rather what the results should be compared to.

Thanks,
Sharam

From: lmap [mailto:[email-masked]] On Behalf Of Bugenhagen, Michael K
Sent: Thursday, July 03, 2014 2:02 PM
To: 'MORTON, ALFRED C (AL)'; 'Romascanu, Dan (Dan)'; '<[email-masked]>'
Subject: Re: [lmap] preliminary agenda - please comment!

Al,

Thoughts ?
Mike

Al

Do I miss anything?

Regards,

Dan

Mike

Hi Michael,

Thanks and Regards,

Dan

Dan,

See you in Toronto!

Cheers,
Mike

Please comment!

Regards,

Dan

---

From: "STARK, BARBARA H" <[email-masked]>
Date: Tue, 8 Jul 2014 03:48:16 +0000

What I understand Michael to be asking for are the precise Measurement Method roles that must be implemented in US MAs used for the US regulator use case, and for the precise registry entries for these Measurement Methods. While IETF deliverables are useful (and necessary) to having a consistent way to configure and manage these Measurement Method roles, I'm pretty sure that this level of precision is not an IETF (or BBF) deliverable.

This is not a BBF/IETF division. I think it's pretty clear that there is no consensus among interested parties at this point as to how/where to create such detailed deliverables. So I'd describe this as an "interested parties" division.
Barbara

From: lmap [mailto:[email-masked]] On Behalf Of [email-masked]
Sent: Monday, July 07, 2014 4:23 AM
To: [email-masked]; MORTON JR., AL; [email-masked]; [email-masked]
Subject: Re: [lmap] preliminary agenda - please comment!

Trevor.

From: lmap [mailto:[email-masked]] On Behalf Of Bugenhagen, Michael K
Sent: 03 July 2014 19:02
To: 'MORTON, ALFRED C (AL)'; 'Romascanu, Dan (Dan)'; '<[email-masked]<mailto:[email-masked]>>'
Subject: Re: [lmap] preliminary agenda - please comment!

Al,

Thoughts ?
Mike

Al

Do I miss anything?

Regards,

Dan

Mike

Hi Michael,

Thanks and Regards,

Dan

Dan,

See you in Toronto!

Cheers,
Mike

Please comment!

Regards,

Dan

---

From: "Bugenhagen, Michael K" <[email-masked]>
Date: Tue, 8 Jul 2014 13:58:17 +0000

Barbara, Trevor,

   From a service provider Operational analysis hat on my head perspective (and remember this is in scope)....
Actionable live network test results must contain some condition information to enable sorting the "BIG DATA" performance results.

We have no such indicators....     Or the triggers and behaviors that enable capturing those "categories".

e.g. - Trevor's information model metaphorically has a toe in the water with the "cross traffic" object, but there is not a "threshold crossing event" object for that environmental object to use for the MA test behavior (reschedule, run, ...)
As a "framework" Large scale operational test results need to be root cause sort-able.

This directly infers we need result markings, with well understood MA behaviors.
All useful Operational tests that DO NOT yield the expected result get put into root cause analysis testing... what's the state of the service, was their cross traffic, .....

So - I'm simply pointing out that as the industry moves from DUT lab type testing to live network testing the "root cause" test result stamping requires some added MA behaviors, Threshold crossing events, and Service State information.
Otherwise it's not going to  as useful we are hoping it will be.

Requirements wise it looks like this ---

To date the registry isn't going to do this so I asked the question if LMAP would...  so far it's nope.

Thanks,
Mike

From: STARK, BARBARA H [mailto:[email-masked]]
Sent: Monday, July 07, 2014 10:48 PM
To: [email-masked]; Bugenhagen, Michael K; MORTON JR., AL; [email-masked]; [email-masked]
Subject: RE: preliminary agenda - please comment!

From: lmap [mailto:[email-masked]] On Behalf Of [email-masked]<mailto:[email-masked]>
Sent: Monday, July 07, 2014 4:23 AM
To: [email-masked]<mailto:[email-masked]>; MORTON JR., AL; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: Re: [lmap] preliminary agenda - please comment!

Trevor.

Al,

Thoughts ?
Mike

Al

Do I miss anything?

Regards,

Dan

Mike

Hi Michael,

Thanks and Regards,

Dan

Dan,

See you in Toronto!

Cheers,
Mike

Please comment!

Regards,

Dan

---

From: Sharam Hakimi <[email-masked]>
Date: Tue, 8 Jul 2014 18:30:55 +0000

Mike,
I do not think it is necessary to complicate the job of the MA.  The MA needs to gather the raw data ( e.g.  from, to, upload, download, results). The Data collector can put the raw data sent from MA in any format that is required by the specific provider, compare to specific metrics, and issue alerts, set Alarms or any requested action. Currently there is no path between the data collector and the measurement controller, however, we could setup something in the future that would turn an alert into an action that the MC can take to direct an MA to perform certain tests over or similar actions.

Sharam

From: lmap [mailto:[email-masked]] On Behalf Of Bugenhagen, Michael K
Sent: Tuesday, July 08, 2014 9:58 AM
To: 'STARK, BARBARA H'; [email-masked]'; 'MORTON JR., AL'; [email-masked]'; [email-masked]'
Subject: Re: [lmap] preliminary agenda - please comment!

Barbara, Trevor,

Thanks,
Mike

Trevor.

Al,

Thoughts ?
Mike

Al

Do I miss anything?

Regards,

Dan

Mike

Hi Michael,

Thanks and Regards,

Dan

Dan,

See you in Toronto!

Cheers,
Mike

Please comment!

Regards,

Dan