Subject: Re: [lmap] LMAP Result Correlation

From: Juergen Schoenwaelder <[email-masked]>
Date: Mon, 1 Feb 2016 09:23:22 +0100

Tim,

I am not sure I understand your suggestion. What is 'task-action'? What
are 'SPs'? Are you suggesting

a) to turn ma-task-cycle-id and ma-report-task-cycle-id into a tag list
   instead of a string (like in the YANG data model) and/or

b) to add ma-action-cycle-id (or an equivalent list) to ma-action-obj?

I think both make sense. The ma-report-task-cycle-id would then report
the set union of ma-task-cycle-id and ma-action-cycle-id.

/js

On Thu, Jan 28, 2016 at 05:10:59PM +0000, Carey, Timothy (Nokia - US) wrote:
> Juergen,
> 
> You are correct; I missed the point. Sorry
> 
> I would simply add the correlation tag to the task-action as an option. These get reported in the report so you can use that. I wouldn't think we would need to specify this as it is probably something SPs would control.
> 
> BR,
> Tim
> 
> -----Original Message-----
> From: EXT Juergen Schoenwaelder [mailto:[email-masked]] 
> Sent: Thursday, January 28, 2016 2:29 AM
> To: Carey, Timothy (Nokia - US)
> Cc: Ron Stana; [email-masked]; [email-masked]
> Subject: Re: [lmap] LMAP Result Correlation
> 
> Tim,
> 
> how does this observation related to the discussion? The YANG model includes the task name and the metric URIs in the report. This is inline with ma-report-task-obj in the information model.
> 
> The place where the report YANG data model differs from the information model is the ma-report-task-cycle-id: the YANG data model allows for a set of tags instead of a single cycle-id.
> 
> /js
> 
> On Wed, Jan 27, 2016 at 10:50:12PM +0000, Carey, Timothy (Nokia - US) wrote:
> > Juergen,
> > 
> > In the TR-069 data model we report the name of the task (not registry entry) that was used in the report information.  This is part of the ma-report-task-obj....
> > 
> > BR,
> > Tim
> > -----Original Message-----
> > From: Juergen Schoenwaelder 
> > [mailto:[email-masked]]
> > Sent: Wednesday, January 27, 2016 8:04 AM
> > To: Ron Stana
> > Cc: [email-masked]; [email-masked]
> > Subject: Re: [lmap] LMAP Result Correlation
> > 
> > On Tue, Jan 12, 2016 at 05:02:36PM +0000, Ron Stana wrote:
> > > Alissa, LMAP Team,
> > > Regarding the September 2015 version of A Framework for Large-Scale Measurement of Broadband Performance [LMAP], I am trying to determine how multiple result sets for a given schedule instance/execution can be correlated.
> > > I think my questions are best illustrated using a couple scenarios:
> > > Scenario #1:
> > > 
> > > 1.       Schedule A is configured to execute Monday@ 9:00am for 1 hour
> > > 
> > > 2.       Schedule B is configured to execute Tuesday @ 9:00am for 1 hour
> > > 
> > > 3.       Both schedules reference the same task and use the same task configurations thus they are measuring the same network measurement but at different times.
> > > 
> > > 4.       Intermediate results are sent to the collector periodically while the schedule executes and then final results are sent once it completes.
> > > 
> > > 5.       What in the results can be used to know which sets of results belong to a single execution of the same schedule instance (example: Schedule A)?  This is my interpretation of LMAP; item #3 seems to address the need.
> > > Item 1: The ietf-lmap-report path report/input/date is the time each result set was sent to the collector so that will be unique for each set of results and so cannot be used.
> > > Item 2: The report/input/task/name will be the same across the results from the different schedules since they are both executing the same task so that cannot be used.
> > > 
> > > Item 3: The report/input/task/row/start value would be the same across results from a given schedule instance (ie. the Monday run) and yet unique across schedule instances (ie Mon vs Tues run) so it could be used.
> > > 
> > >
> > > 6.       What in the results can be used to know which sets of results belong to all executions of the same schedule instance (example: Schedule A)?
> > 
> > I think you can configure two different tasks with different tags and since the tags are reported and the task name, you can distinguish things. Whether this is practical or whether it should also be possible to set tags on an action is perhaps something to discuss.
> > 
> > > Scenario #2:
> > > While Item #3 above can address part of Scenario #1, it does not address this additional scenario:
> > > 
> > > 1.       Schedule A is performing a Performance Monitoring task (ex. TWAMP) and so is configured for immediate and startup execution/events so it should always be running.
> > > 
> > > 2.       The task is sending results to the collector periodically during execution.
> > > 
> > > 3.       The device hosting the MA is rebooted so the MA restarts the schedule/task execution.  Thus the report/input/task/row/start value would be different in the results sent after the restart.
> > > 
> > > 4.       Since the same schedule/task definition is producing the results before and after the MA restart, they logically belong to the same schedule execution.
> > > 
> > > 5.       What can be used to group the results from before and after the restart together?
> > 
> > I think the mechanism to group measurements belonging logically together tags. Looking at time stamps to group things is likely brittle.
> > 
> > > Adding the schedule name to report/input/task would help but since the schedule name can be reused, it does not infer a specific schedule instance.
> > 
> > While having the schedule name may help in some situations, it won't in others. I think we should rather use tags that have no other meaning. This allows for example to rename schedules etc. In fact, I would prefer not to expose the details how something got scheduled to the collector. It should not matter much to the collector whether I use multiple schedules bound to the same event with one action each or a single schedule bound to the event with multiple concurrent actions.
> > 
> > > Adding something like an 'initial schedule start' to report/input/task, where it is the date/time the schedule started the very first time (thus not changing across MA restarts), seems to provide the needed association between the result sets.  This could be abstracted to a 'schedule id' since it could alternatively be a unique id assigned to each schedule by the Controller.
> > 
> > I would rather not put a requirement into the data model that an lmap implementation has to manage persistent state.
> > 
> > > While a custom value like this could be placed in report/input/task/tag or report/input/task/option I wanted to know if a common approach existed or was planned.
> > 
> > I think the tag was designed for this purpose, namely to be able to correlate results that logically belong together.
> > 
> > /js
> > 
> > -- 
> > Juergen Schoenwaelder           Jacobs University Bremen gGmbH
> > Phone: +49 421 200 3587         Campus Ring 1 | 28759 Bremen | Germany
> > Fax:   +49 421 200 3103         <http://www.jacobs-university.de/>
> > 
> > 
> 
> -- 
> Juergen Schoenwaelder           Jacobs University Bremen gGmbH
> Phone: +49 421 200 3587         Campus Ring 1 | 28759 Bremen | Germany
> Fax:   +49 421 200 3103         <http://www.jacobs-university.de/>

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1 | 28759 Bremen | Germany
Fax:   +49 421 200 3103         <http://www.jacobs-university.de/>

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Mon, 1 Feb 2016 09:27:46 +0100

On Sun, Jan 31, 2016 at 03:36:15PM -0500, Ron Stana wrote:
> Juergen, Tim,
> 
> The intention is to be able to identify a set of results in the collector
> for each schedule. My understanding is the correlation tag would therefore
> need to be provided with the schedule rather than the task.
> 
> This is my interpretation of LMAP, please correct me if it is incorrect:
> 1) tasks are predefined by the MA (example, ping, TWAMP,Y.1564, etc)
> 2) therefore, if a configuration cannot be sent to the MA containing a task
> with a task name that is unknown to the MA
> 3) the 'options' object sent within a task should be treated as a way to
> change the default parameters for the task from that point forward
> 4) a schedule is required to start an instance of a task
> 5) the 'options' object sent within the schedule is a way to override the
> task parameters for just that schedule
> 
> Thus, if we have a ping test that as 2 parameters (target IP and number of
> pings), and a user wants to ping two destinations daily with the same
> number of pings, LMAP supports this scenario with the following
> configuration:
> 1) the 'options' for the Ping task could be used to set the number of pings.
> 2) the 'options' for Schedule A could set the target IP to y.  This
> schedule will run daily.
> 3) the 'options' for Schedule B could set the target IP to z.  This
> schedule will run daily at a time that does not conflict with Schedule A.
> 
> Alternatively the Ping task would not have to be reconfigured at all if
> Schedules A and B additionally specified the number of pings in their
> 'options'.

This is correct.

> If this understanding is correct, then to retrieve the results from the
> collector for either Schedule A or Schedule B seems to require unique tag
> to be specified with the schedule instead of the task.  The 'options'
> object for the schedule could be used, however it would be more symmetrical
> to allow a tag for the schedule since the value would be sent to the
> collector in the 'tag' object of the report.

I agree that we should allow the definition of additional tags in the
action configuration. Tim might have suggested this as well but I was
not sure, see my other email.

/js

---

From: "Carey, Timothy (Nokia - US)" <[email-masked]>
Date: Mon, 1 Feb 2016 18:47:49 +0000

Juergen and Ron,

My comments in line <TAC>

-----Original Message-----
From: EXT Juergen Schoenwaelder [mailto:[email-masked]] 
Sent: Monday, February 01, 2016 2:28 AM
To: Ron Stana
Cc: Carey, Timothy (Nokia - US); [email-masked]; [email-masked]
Subject: Re: [lmap] LMAP Result Correlation

On Sun, Jan 31, 2016 at 03:36:15PM -0500, Ron Stana wrote:
> Juergen, Tim,
> 
> The intention is to be able to identify a set of results in the 
> collector for each schedule. My understanding is the correlation tag 
> would therefore need to be provided with the schedule rather than the task.
> 
> 2) therefore, if a configuration cannot be sent to the MA containing a 
> task with a task name that is unknown to the MA
> 3) the 'options' object sent within a task should be treated as a way 
> to change the default parameters for the task from that point forward
> 5) the 'options' object sent within the schedule is a way to override 
> the task parameters for just that schedule
> 
> Thus, if we have a ping test that as 2 parameters (target IP and 
> number of pings), and a user wants to ping two destinations daily with 
> the same number of pings, LMAP supports this scenario with the 
> following
> configuration:
> Alternatively the Ping task would not have to be reconfigured at all 
> if Schedules A and B additionally specified the number of pings in 
> their 'options'.

This is correct.
<TAC> This is correct if you use the schedule's action (ma-action-obj). A schedule by itself doesn't have options - just the scheduled action. 

> If this understanding is correct, then to retrieve the results from 
> the collector for either Schedule A or Schedule B seems to require 
> unique tag to be specified with the schedule instead of the task.  The 'options'
> object for the schedule could be used, however it would be more 
> symmetrical to allow a tag for the schedule since the value would be 
> sent to the collector in the 'tag' object of the report.

I agree that we should allow the definition of additional tags in the action configuration. Tim might have suggested this as well but I was not sure, see my other email.

<TAC> Again it is the schedule action that would have the option - I am looking at draft-07 of the information framework section 3.7.2.
The report has the scheduled action options in the ma-report-task-object section 3.6.2 so I don't understand the comment for the need in the schedule...

/js

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Mon, 1 Feb 2016 20:36:36 +0100

Tim,

we are talking about tags, not options. A tag is the YANG
representation of what is called the cycle-id in the information
model. In contrast to the information model, the YANG model allows a
set of tags and not just a single one.

  The ma-task-obj has a ma-task-cycle-id of type string in the
  information model. In the YANG data model, we have instead a list of
  tags (and one of the tags can carry a cycle identifier. This allows
  us to have multiple tags for a given task.

  The ma-report-task-obj has a matching ma-report-task-cycle-id in the
  information model. In the YANG data model, we have again a list of
  tags.

  In order to distinguish reports from a given task kicked by different
  actions, the proposal was made to have the ability to define a tag or
  cycle-id in addition as part of the ma-action-obj.

I suggest to align the information model here with the YANG data
model.

/js

---

From: "Carey, Timothy (Nokia - US)" <[email-masked]>
Date: Tue, 2 Feb 2016 12:59:08 +0000

Juergen,

Oh - I am sorry then - I missed that this point. 
I wouldn't have an issue with a correlation point on the scheduled action - that is where the options are.

BR,
Tim

-----Original Message-----
From: EXT Juergen Schoenwaelder [mailto:[email-masked]] 
Sent: Monday, February 01, 2016 1:37 PM
To: Carey, Timothy (Nokia - US)
Cc: Ron Stana; [email-masked]; [email-masked]
Subject: Re: [lmap] LMAP Result Correlation

Tim,

we are talking about tags, not options. A tag is the YANG representation of what is called the cycle-id in the information model. In contrast to the information model, the YANG model allows a set of tags and not just a single one.

I suggest to align the information model here with the YANG data model.

/js