Subject: [lmap] Only Report raw measurement results?

From: <[email-masked]>
Date: Wed, 27 Nov 2013 16:48:12 +0000

Following up on Juergen's suggestion, how about adding text on these lines:

In the initial LMAP Information Model and Report Protocol, for simplicity we assume that all Measurement Results are reported as-is, but allow extensibility so that a measurement system (or perhaps a second phase of LMAP) could allow a MA to pre-process Measurement Results before it reports them. Potential examples of pre-processing by the MA are:
* labelling, or perhaps not including, Measurement Results impacted by for instance cross-traffic or the MP being busy.  
* detailing the start and end of suppression.
* filtering outlier Results 
* calculating some statistic like average (beyond that defined by the Measurement Task itself)
* sending a special Report ("alarm") if recent Results match some pattern (for example, below a threshold)
* not sending a Report if there are no Measurement Results

Are any of these things important enough (&/or simple enough) that they should be included in LMAP1.0?
(some of them could be crafted by careful definition of Measurement Tasks - eg filtering, calculating statistics and perhaps alarms)

The above text could either fit at the end of S5.4 Report Protocol, or perhaps in a new section "Items beyond the scope of the *initial* LMAP Protocol Model".

This actually raises a general point: the Info Model (and Protocol) i-ds should add some indication about extensibility. 

Comments?
Thanks
phil

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Juergen Schoenwaelder
> Sent: 14 November 2013 12:35
> To: Burbridge,T,Trevor,TUB8 R
> Cc: [email-masked]; [email-masked]
> Subject: Re: [lmap] Splitting measurement tasks into measurement tasks
> plus reporting tasks?
> 
> Trevor,
> 
> concerning (b) and (c), I think we should not consider this as a
> requirement for LMAP 1.0 but perhaps we should consider how such
> filtering and processing steps could be added in a later version of
> LMAP. In other words, we may want to think about how we can make the
> information model extensible such that (b) and (c) could be added
> nicely in a later phase.
> 
> /js
> 
> On Wed, Nov 13, 2013 at 08:53:41AM +0000, [email-masked]
> wrote:
> > (d) we have in the information model.
> >
> > (a)-(c) are interesting questions. (a) could always be a configurable
> parameter. (b) and (c) I've kept away from for reasons of complexity,
> but they could be useful tools. My initial reaction is that we don't
> need pre-processing as (i) the amount of data returned is not likely to
> be huge in most scenarios; (ii) some of this can be done within the
> test - for example a stream of ping packets can be a single measurement
> and report avg, min, max etc.
> >
> > Trevor.
> >
> > >-----Original Message-----
> > >From: [email-masked] [mailto:[email-masked]] On Behalf
> > >Of Juergen Schoenwaelder
> > >Sent: 12 November 2013 19:26
> > >To: Steve Miller
> > >Cc: [email-masked]
> > >Subject: Re: [lmap] Splitting measurement tasks into measurement
> > >tasks plus reporting tasks?
> > >
> > >I think you touch on several issues:
> > >
> > >a) Are reports send if nothing is pending to be reported?
> > >
> > >   I think the answer should be no but this can be discussed.
> > >
> > >b) Should we support a notion of filters that can be applied, that
> is
> > >   only result records passing a filter expression are actually
> reported.
> > >
> > >c) In a previous email, someone suggested to have aggregation
> > >   functions that can be applied generically to test results
> > >   (e.g., calculation of avg, min, max, ...).
> > >
> > >d) Should we support a report timing that kind of says "report
> > >   immediately if a result record is available"
> > >
> > >These are all fair questions one can ask and it is going to be a
> > >trade-off between simplicity and functional richness. Perhaps we
> need
> > >to look at what deployed systems really do so we do not get over
> > >board building aggregation and filtering features into the model
> that
> > >can also easily be applied on the collector's end.
> > >
> > >/js
> > >
> > >On Tue, Nov 12, 2013 at 01:56:11PM -0500, Steve Miller wrote:
> > >>    I think that's close but I'm not quite sure it's right.  I
> think
> > >> part of the gotcha
> > >here is that the spec tends to express Measurement Tasks as things
> > >that have a definite start and end time, while I'm thinking of
> things
> > >that are running continuously.
> > >>
> > >>    That is, if we wanted to do a DNS query once a second, forever,
> > >> the current
> > >model would express that more as:
> > >>
> > >> 	Measurement Task "DNS Performance": Perform a DNS
> > >loss/latency/jitter check once a second, for 300 seconds, reporting
> > >avg/min/max loss, latency and jitter plus a histogram of the
> latencies.
> > >>
> > >> 	Report Channel "Collector C": Report to collector at
> > >0,5,10,15,20,25,30,35,40,45,50,55 minutes after the hour.
> > >>
> > >> 	Measurement Schedule: Run the Measurement Task "DNS
> Performance"
> > >at 0,5,10,15,20,25,30,35,40,45,50,55 minutes after the hour and
> > >report using Report Channel "Collector C".
> > >>
> > >> but each of those reports has a very definite start and end time -
> -
> > >> there's no
> > >way there to define a report via a rolling window.
> > >>
> > >>    In the current framework, to get what I'm thinking about, I can
> > >> *almost* do
> > >it, but I'd have to do that separately.  First, to get the
> > >performance numbers every five minutes, we'd have what's above, more
> or less:
> > >>
> > >> 	Measurement Task "DNS Performance": Perform a DNS
> > >loss/latency/jitter check once a second, for 300 seconds, reporting
> > >avg/min/max loss, latency and jitter plus a histogram of the
> latencies.
> > >>
> > >> 	Report Channel "Collector C 300Secs": Report to collector at
> > >0,5,10,15,20,25,30,35,40,45,50,55 minutes after the hour.
> > >>
> > >> 	Measurement Schedule: Run the Measurement Task "DNS
> Performance"
> > >at 0,5,10,15,20,25,30,35,40,45,50,55 minutes after the hour and
> > >report using Report Channel "Collector C 300Secs".
> > >>
> > >> and then to get some sort of report immediately, it seems like
> > >> about as close as
> > >we'd get would be:
> > >>
> > >> 	Measurement Task "DNS Badness": Perform a DNS
> loss/latency/jitter
> > >check once a second, for 300 seconds, reporting avg/min/max loss,
> > >latency and jitter immediately if the loss is worse than 1%, the
> > >latency is more than 200ms, or the jitter is more than 200ms.
> > >>
> > >> 	Report Channel "Collector C 1Sec": Report to collector once a
> second.
> > >>
> > >> 	Measurement Schedule: Run the Measurement Task "DNS Badness"
> at
> > >0,5,10,15,20,25,30,35,40,45,50,55 minutes after the hour and report
> > >using Report Channel "Collector C 1Sec".
> > >>
> > >> which has a couple of issues:
> > >>
> > >> 	- now we're sending twice as much traffic to each destination
> as
> > >> we were
> > >before (which in this example isn't such a big deal but it's not
> hard
> > >to imagine that a file-download test or an RTP test wouldn't be so
> > >forgiving (-: )
> > >>
> > >> 	- it's not truly a rolling window, so you either have to
> clamp the
> > >accelerated reporting until you have enough baseline to be useful
> > >(that is, you don't want to report "OMG! 100% loss!"  if just the
> > >first packet is lost) or you have to put up with noisy reports
> > >>
> > >> 	- I think the collector gets a report every second, even if
> that
> > >> report is
> > >empty
> > >>
> > >>    Maybe a way to look at this would be to have the concepts of:
> > >>
> > >> 	- never-ending measurements, which run continuously
> > >>
> > >> and
> > >>
> > >> 	- async report channels, where you could express simple
> conditions
> > >> to
> > >control whether or not a report occurs
> > >>
> > >> or maybe async report channels are really some sort of async
> > >> measurement
> > >schedule:
> > >>
> > >> 	Run the test named "DNS Badness" continuously; report using
> the
> > >channel "Collector C" whenever loss is more than 1% or the latency
> is
> > >more than 200ms or the jitter is more than 200ms over the last 300
> seconds.
> > >>
> > >>    I could be convinced that I'm thinking too hard about this or
> > >> that my
> > >perception of the measurement model as viewing Measurement Tasks as
> > >discrete things with discrete start times and durations is simply
> incorrect.
> > >>
> > >>    Did that make anything more clear?
> > >>
> > >>    Also, to address some of what you and Trevor were discussing: I
> > >> don't know
> > >that I necessarily regard these async reports as logging or status.
> > >I think they're closer to "normal" reports, they just happen
> > >asynchronously.  I could be convinced to treat this as logging or
> status, too, if that'd be helpful somehow.
> > >>
> > >> 	-Steve
> > >>
> > >> On Mon, Nov 11, 2013 at 09:42:18PM +0100, Juergen Schoenwaelder
> wrote:
> > >> > I hear two suggestions here:
> > >> >
> > >> > a) Give reporting channels their own independent schedule (right
> now
> > >> >    they are essentially tied to the scheduled task).
> > >> >
> > >> > b) Provide a means for post-processing raw data in order to
> derive
> > >> >    certain statistics before the result is shipped via a report
> > >> >    channel.
> > >> >
> > >> > Is this a fair summary of the request?
> > >> _______________________________________________
> > >> lmap mailing list
> > >> [email-masked]
> > >> https://www.ietf.org/mailman/listinfo/lmap
> > >
> > >--
> > >Juergen Schoenwaelder           Jacobs University Bremen gGmbH
> > >Phone: +49 421 200 3587         Campus Ring 1, 28759 Bremen, Germany
> > >Fax:   +49 421 200 3103         <http://www.jacobs-university.de/>
> > >_______________________________________________
> > >lmap mailing list
> > >[email-masked]
> > >https://www.ietf.org/mailman/listinfo/lmap
> 
> --
> Juergen Schoenwaelder           Jacobs University Bremen gGmbH
> Phone: +49 421 200 3587         Campus Ring 1, 28759 Bremen, Germany
> Fax:   +49 421 200 3103         <http://www.jacobs-university.de/>
> _______________________________________________
> lmap mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/lmap

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Thu, 28 Nov 2013 08:00:41 +0100

Hi,

I prefer to leave these things out of LMAP 1.0 but to think about how
such raw result processing steps can be fitted into LMAP at a later
point in time. There are several deployed systems that do not do this
and hence it seems such raw data processing functions do not seem to
be essential to get LMAP 1.0 out of the door.

/js