Subject: [lmap] LMAP Result Correlation

From: Ron Stana <[email-masked]>
Date: Tue, 12 Jan 2016 17:02:36 +0000

Alissa, LMAP Team,
Regarding the September 2015 version of A Framework for Large-Scale Measurement of Broadband Performance [LMAP], I am trying to determine how multiple result sets for a given schedule instance/execution can be correlated.
I think my questions are best illustrated using a couple scenarios:
Scenario #1:

1.       Schedule A is configured to execute Monday@ 9:00am for 1 hour

2.       Schedule B is configured to execute Tuesday @ 9:00am for 1 hour

3.       Both schedules reference the same task and use the same task configurations thus they are measuring the same network measurement but at different times.

4.       Intermediate results are sent to the collector periodically while the schedule executes and then final results are sent once it completes.

5.       What in the results can be used to know which sets of results belong to a single execution of the same schedule instance (example: Schedule A)?  This is my interpretation of LMAP; item #3 seems to address the need.
Item 1: The ietf-lmap-report path report/input/date is the time each result set was sent to the collector so that will be unique for each set of results and so cannot be used.
Item 2: The report/input/task/name will be the same across the results from the different schedules since they are both executing the same task so that cannot be used.

Item 3: The report/input/task/row/start value would be the same across results from a given schedule instance (ie. the Monday run) and yet unique across schedule instances (ie Mon vs Tues run) so it could be used.

6.       What in the results can be used to know which sets of results belong to all executions of the same schedule instance (example: Schedule A)?
Scenario #2:
While Item #3 above can address part of Scenario #1, it does not address this additional scenario:

1.       Schedule A is performing a Performance Monitoring task (ex. TWAMP) and so is configured for immediate and startup execution/events so it should always be running.

2.       The task is sending results to the collector periodically during execution.

3.       The device hosting the MA is rebooted so the MA restarts the schedule/task execution.  Thus the report/input/task/row/start value would be different in the results sent after the restart.

4.       Since the same schedule/task definition is producing the results before and after the MA restart, they logically belong to the same schedule execution.

5.       What can be used to group the results from before and after the restart together?
Adding the schedule name to report/input/task would help but since the schedule name can be reused, it does not infer a specific schedule instance.
Adding something like an 'initial schedule start' to report/input/task, where it is the date/time the schedule started the very first time (thus not changing across MA restarts), seems to provide the needed association between the result sets.  This could be abstracted to a 'schedule id' since it could alternatively be a unique id assigned to each schedule by the Controller.  While a custom value like this could be placed in report/input/task/tag or report/input/task/option I wanted to know if a common approach existed or was planned.
Thank you,
Ron

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Wed, 27 Jan 2016 15:04:29 +0100

> 6.       What in the results can be used to know which sets of results belong to all executions of the same schedule instance (example: Schedule A)?

I think you can configure two different tasks with different tags and
since the tags are reported and the task name, you can distinguish
things. Whether this is practical or whether it should also be
possible to set tags on an action is perhaps something to discuss.

> 5.       What can be used to group the results from before and after the restart together?

I think the mechanism to group measurements belonging logically
together tags. Looking at time stamps to group things is likely
brittle.

> Adding the schedule name to report/input/task would help but since the schedule name can be reused, it does not infer a specific schedule instance.

While having the schedule name may help in some situations, it won't
in others. I think we should rather use tags that have no other
meaning. This allows for example to rename schedules etc. In fact, I
would prefer not to expose the details how something got scheduled to
the collector. It should not matter much to the collector whether I
use multiple schedules bound to the same event with one action each or
a single schedule bound to the event with multiple concurrent actions.

> Adding something like an 'initial schedule start' to report/input/task, where it is the date/time the schedule started the very first time (thus not changing across MA restarts), seems to provide the needed association between the result sets.  This could be abstracted to a 'schedule id' since it could alternatively be a unique id assigned to each schedule by the Controller.

I would rather not put a requirement into the data model that an lmap
implementation has to manage persistent state.

> While a custom value like this could be placed in report/input/task/tag or report/input/task/option I wanted to know if a common approach existed or was planned.

I think the tag was designed for this purpose, namely to be able to
correlate results that logically belong together.

/js

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1 | 28759 Bremen | Germany
Fax:   +49 421 200 3103         <http://www.jacobs-university.de/>

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Thu, 28 Jan 2016 09:29:05 +0100

Tim,

how does this observation related to the discussion? The YANG model
includes the task name and the metric URIs in the report. This is
inline with ma-report-task-obj in the information model.

The place where the report YANG data model differs from the
information model is the ma-report-task-cycle-id: the YANG data model
allows for a set of tags instead of a single cycle-id.

/js

On Wed, Jan 27, 2016 at 10:50:12PM +0000, Carey, Timothy (Nokia - US) wrote:
> Juergen,
> 
> In the TR-069 data model we report the name of the task (not registry entry) that was used in the report information.  This is part of the ma-report-task-obj....
> 
> BR,
> Tim
> -----Original Message-----
> From: Juergen Schoenwaelder [mailto:[email-masked]] 
> Sent: Wednesday, January 27, 2016 8:04 AM
> To: Ron Stana
> Cc: [email-masked]; [email-masked]
> Subject: Re: [lmap] LMAP Result Correlation
> 
> I think you can configure two different tasks with different tags and since the tags are reported and the task name, you can distinguish things. Whether this is practical or whether it should also be possible to set tags on an action is perhaps something to discuss.
> 
> > Scenario #2:
> I think the mechanism to group measurements belonging logically together tags. Looking at time stamps to group things is likely brittle.
> 
> While having the schedule name may help in some situations, it won't in others. I think we should rather use tags that have no other meaning. This allows for example to rename schedules etc. In fact, I would prefer not to expose the details how something got scheduled to the collector. It should not matter much to the collector whether I use multiple schedules bound to the same event with one action each or a single schedule bound to the event with multiple concurrent actions.
> 
> I would rather not put a requirement into the data model that an lmap implementation has to manage persistent state.
> 
> I think the tag was designed for this purpose, namely to be able to correlate results that logically belong together.
> 
> /js
> 
> --

---

From: "Carey, Timothy (Nokia - US)" <[email-masked]>
Date: Wed, 27 Jan 2016 22:50:12 +0000

Juergen,

BR,
Tim
-----Original Message-----
From: Juergen Schoenwaelder [mailto:[email-masked]] 
Sent: Wednesday, January 27, 2016 8:04 AM
To: Ron Stana
Cc: [email-masked]; [email-masked]
Subject: Re: [lmap] LMAP Result Correlation

/js

---

From: "Carey, Timothy (Nokia - US)" <[email-masked]>
Date: Thu, 28 Jan 2016 17:10:59 +0000

Juergen,

You are correct; I missed the point. Sorry

I would simply add the correlation tag to the task-action as an option. These get reported in the report so you can use that. I wouldn't think we would need to specify this as it is probably something SPs would control.

BR,
Tim

-----Original Message-----
From: EXT Juergen Schoenwaelder [mailto:[email-masked]] 
Sent: Thursday, January 28, 2016 2:29 AM
To: Carey, Timothy (Nokia - US)
Cc: Ron Stana; [email-masked]; [email-masked]
Subject: Re: [lmap] LMAP Result Correlation

Tim,

how does this observation related to the discussion? The YANG model includes the task name and the metric URIs in the report. This is inline with ma-report-task-obj in the information model.

The place where the report YANG data model differs from the information model is the ma-report-task-cycle-id: the YANG data model allows for a set of tags instead of a single cycle-id.

/js

On Wed, Jan 27, 2016 at 10:50:12PM +0000, Carey, Timothy (Nokia - US) wrote:
> From: Juergen Schoenwaelder 
> [mailto:[email-masked]]

---

From: Ron Stana <[email-masked]>
Date: Sun, 31 Jan 2016 15:36:15 -0500

Juergen, Tim,

The intention is to be able to identify a set of results in the collector
for each schedule. My understanding is the correlation tag would therefore
need to be provided with the schedule rather than the task.

This is my interpretation of LMAP, please correct me if it is incorrect:
1) tasks are predefined by the MA (example, ping, TWAMP,Y.1564, etc)
2) therefore, if a configuration cannot be sent to the MA containing a task
with a task name that is unknown to the MA
3) the 'options' object sent within a task should be treated as a way to
change the default parameters for the task from that point forward
4) a schedule is required to start an instance of a task
5) the 'options' object sent within the schedule is a way to override the
task parameters for just that schedule

Thus, if we have a ping test that as 2 parameters (target IP and number of
pings), and a user wants to ping two destinations daily with the same
number of pings, LMAP supports this scenario with the following
configuration:
1) the 'options' for the Ping task could be used to set the number of pings.
2) the 'options' for Schedule A could set the target IP to y.  This
schedule will run daily.
3) the 'options' for Schedule B could set the target IP to z.  This
schedule will run daily at a time that does not conflict with Schedule A.

Alternatively the Ping task would not have to be reconfigured at all if
Schedules A and B additionally specified the number of pings in their
'options'.

If this understanding is correct, then to retrieve the results from the
collector for either Schedule A or Schedule B seems to require unique tag
to be specified with the schedule instead of the task.  The 'options'
object for the schedule could be used, however it would be more symmetrical
to allow a tag for the schedule since the value would be sent to the
collector in the 'tag' object of the report.

Ron

On Thu, Jan 28, 2016 at 12:10 PM, Carey, Timothy (Nokia - US) <
[email-masked]> wrote:

> I would simply add the correlation tag to the task-action as an option.
> These get reported in the report so you can use that. I wouldn't think we
> would need to specify this as it is probably something SPs would control.
>
> BR,
> Tim
>
> includes the task name and the metric URIs in the report. This is inline
> with ma-report-task-obj in the information model.
>
> The place where the report YANG data model differs from the information
> model is the ma-report-task-cycle-id: the YANG data model allows for a set
> of tags instead of a single cycle-id.
>
> /js
>
> > In the TR-069 data model we report the name of the task (not registry
> entry) that was used in the report information.  This is part of the
> ma-report-task-obj....
> >
> > BR,
> > Tim
> > > Regarding the September 2015 version of A Framework for Large-Scale
> Measurement of Broadband Performance [LMAP], I am trying to determine how
> multiple result sets for a given schedule instance/execution can be
> correlated.
> > > 3.       Both schedules reference the same task and use the same task
> configurations thus they are measuring the same network measurement but at
> different times.
> > >
> > > 4.       Intermediate results are sent to the collector periodically
> while the schedule executes and then final results are sent once it
> completes.
> > >
> > > 5.       What in the results can be used to know which sets of results
> belong to a single execution of the same schedule instance (example:
> Schedule A)?  This is my interpretation of LMAP; item #3 seems to address
> the need.
> > > Item 1: The ietf-lmap-report path report/input/date is the time each
> result set was sent to the collector so that will be unique for each set of
> results and so cannot be used.
> > > Item 2: The report/input/task/name will be the same across the results
> from the different schedules since they are both executing the same task so
> that cannot be used.
> > >
> > > Item 3: The report/input/task/row/start value would be the same across
> results from a given schedule instance (ie. the Monday run) and yet unique
> across schedule instances (ie Mon vs Tues run) so it could be used.
> > >
> > >
> > > 6.       What in the results can be used to know which sets of results
> belong to all executions of the same schedule instance (example: Schedule
> A)?
> >
> since the tags are reported and the task name, you can distinguish things.
> Whether this is practical or whether it should also be possible to set tags
> on an action is perhaps something to discuss.
> >
> > > Scenario #2:
> > > 1.       Schedule A is performing a Performance Monitoring task (ex.
> TWAMP) and so is configured for immediate and startup execution/events so
> it should always be running.
> > >
> > > 3.       The device hosting the MA is rebooted so the MA restarts the
> schedule/task execution.  Thus the report/input/task/row/start value would
> be different in the results sent after the restart.
> > >
> > > 4.       Since the same schedule/task definition is producing the
> results before and after the MA restart, they logically belong to the same
> schedule execution.
> > >
> > I think the mechanism to group measurements belonging logically together
> tags. Looking at time stamps to group things is likely brittle.
> >
> > > Adding the schedule name to report/input/task would help but since the
> schedule name can be reused, it does not infer a specific schedule instance.
> >
> > While having the schedule name may help in some situations, it won't in
> others. I think we should rather use tags that have no other meaning. This
> allows for example to rename schedules etc. In fact, I would prefer not to
> expose the details how something got scheduled to the collector. It should
> not matter much to the collector whether I use multiple schedules bound to
> the same event with one action each or a single schedule bound to the event
> with multiple concurrent actions.
> >
> > > Adding something like an 'initial schedule start' to
> report/input/task, where it is the date/time the schedule started the very
> first time (thus not changing across MA restarts), seems to provide the
> needed association between the result sets.  This could be abstracted to a
> 'schedule id' since it could alternatively be a unique id assigned to each
> schedule by the Controller.
> >
> > > While a custom value like this could be placed in
> report/input/task/tag or report/input/task/option I wanted to know if a
> common approach existed or was planned.
> >