Subject: [lmap] Framework comments

From: <[email-masked]>
Date: Fri, 1 Aug 2014 16:08:37 +0100

Matt,
I wanted to follow up on a couple of comments you made at the mike during the Toronto meeting. They were both about avoiding sampling bias in results analysis - in particular, if the test has errors or is suppressed, then you think test results should be reported (as it's more likely that performance is bad).  I agree. I think the framework is ok, but it could make your point more clearly.  

Assuming you were reading S5.4 Report protocol http://tools.ietf.org/html/draft-ietf-lmap-framework-07#section-5.4  - here's what it says:-

<<
Optionally, a Report is not sent when there are no Measurement
   Results.

   In the initial LMAP Information Model and Report Protocol, for
   simplicity we assume that all Measurement Results are reported as-is,
   but allow extensibility so that a measurement system (or perhaps a
   second phase of LMAP) could allow a MA to:

   o  label, or perhaps not include, Measurement Results impacted by,
      for instance, cross-traffic or the Measurement Peer (or other
      Measurement Agent) being busy

   o  label Measurement Results obtained by a Measurement Task that
      overlapped with another

   o  not report the Measurement Results if the MA believes that they
      are invalid

   o  detail when Suppression started and ended

I think this text is OK, but we could add a pointer to Section 6.1, something like:-

<< As discussed in Section 6.1, data analysis of the results should carefully consider potential bias from any Measurement Results that are not reported, or from Measurement Results that are reported but may be invalid. >>

S6.1 says:
<<
An Instruction is likely to include several Measurement Tasks.
   Typically these run at different times, but it is also possible for
   them to run at the same time.  Some Tasks may be compatible, in that
   they do not affect each other's Results, whilst with others great
   care would need to be taken.
... 
The measurement system also needs to consider carefully how to
   interpret missing Results; for example, if the missing Results are
   ignored and the lack of a Report is caused by its broadband being
   broken, then the estimate of overall performance, averaged across all
   MAs, would be too optimistic.

Do you think more needs to be added to this?  maybe we should add:
<< The correction interpretation may depend on the specifics of the Measurement Task and Measurement Schedule.>>

S5.6 about " Items beyond the scope of the initial LMAP work" says
<<   5.  It does not define how to analyse Measurement Results, including
       how to interpret missing Results.

The other section that may need discussion is S5.2.2.1:
<<
 The Suppression information may include any of the following optional
   fields:
[list of bullets, including:]
  o  a demand that the MA immediately ends on-going Measurement Task(s)
      that are tagged for suppression (and deletes the associated
      partial Measurement Result(s)).  	
(Side-note: normally suppression stops new Measurement Tasks from starting; this is about Tasks that are stopped mid-test)
I think the text is reasonable - for most Tasks partial Measurement Results are basically meaningless. It's possible there are some Tasks where the partial results are useful, so perhaps we should change "and deletes the associated partial Measurement Result(s)" to "and most likely deletes the associated partial Measurement Result(s)"

If you have other review comments, they would be great - but please send them as soon as possible (I'll be working on an update early next week - Dan mentioned a deadline fo 31st July for comments.)

Thanks,
Best wishes
phil

---

From: "Bugenhagen, Michael K" <[email-masked]>
Date: Fri, 1 Aug 2014 15:52:51 +0000

I'll be sending out my comments by next Friday.

I'm traveling and in meetings all this week so unfortunately I couldn't address them this week.

Thanks,
Mike

-----Original Message-----
From: lmap [mailto:[email-masked]] On Behalf Of [email-masked]
Sent: Friday, August 01, 2014 10:09 AM
To: [email-masked]; [email-masked]
Subject: [lmap] Framework comments

Matt,

<<

S6.1 says:
<<

(Side-note: normally suppression stops new Measurement Tasks from starting; this is about Tasks that are stopped mid-test) I think the text is reasonable - for most Tasks partial Measurement Results are basically meaningless. It's possible there are some Tasks where the partial results are useful, so perhaps we should change "and deletes the associated partial Measurement Result(s)" to "and most likely deletes the associated partial Measurement Result(s)"

Thanks,
Best wishes
phil

_______________________________________________
lmap mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/lmap