Subject: [lmap] Support for multiple report table formats from one task

From: Ron Stana <[email-masked]>
Date: Fri, 12 Feb 2016 17:31:02 -0500

LMAP Team,

My understanding of the LMAP proposal is ietf-lmap-report:report/task/name
is supposed to contain one of the values defined in
ietf-lmap-control:lmap/tasks/task/name.

If this is correct, and since each task instance within a report can only
contain one table definition (one header + its row data) then how does a
task send multiple tables of results that contain different table formats?

Use Cases:
1) Y.1564 task has results for the Service Configuration and the Service
Performance data, and each table has a different set of columns.
2) An RFC-2544 task would need a different table for the Throughput,
Latency, Frame Loss and Back to Back result sets.
3) Any task may want to send a minimal result set while it is executing
that is very different from the final result set when it has completed.

The ietf-lmap-report:report/task/tag could be used to differentiate the
result sets sent by the same task. However this is not very efficient since
this approach requires the MA to send each report to the collector
separately because each report can only contain one set of results per task
name.

Any guidance or clarification on the LMAP proposal would be appreciated,

Ron

---

From: "Carey, Timothy (Nokia - US)" <[email-masked]>
Date: Sun, 14 Feb 2016 16:11:51 +0000

Ron,

The task has multiple registry entries – I would assume your example is that this task
would have 3 registry entries.

The columns are for the task and would have to have a union of all the metrics for registry entries. – Is this what you were asking?

It seems that we lose the context of the registry entry to column though…

BR,
Tim

From: Ron Stana [mailto:[email-masked]]
Sent: Friday, February 12, 2016 4:31 PM
To: [email-masked]
Cc: [email-masked]
Subject: [lmap] Support for multiple report table formats from one task

LMAP Team,

My understanding of the LMAP proposal is ietf-lmap-report:report/task/name is supposed to contain one of the values defined in ietf-lmap-control:lmap/tasks/task/name.

If this is correct, and since each task instance within a report can only contain one table definition (one header + its row data) then how does a task send multiple tables of results that contain different table formats?

Use Cases:
1) Y.1564 task has results for the Service Configuration and the Service Performance data, and each table has a different set of columns.
2) An RFC-2544 task would need a different table for the Throughput, Latency, Frame Loss and Back to Back result sets.
3) Any task may want to send a minimal result set while it is executing that is very different from the final result set when it has completed.

The ietf-lmap-report:report/task/tag could be used to differentiate the result sets sent by the same task. However this is not very efficient since this approach requires the MA to send each report to the collector separately because each report can only contain one set of results per task name.

Ron

---

From: Ron Stana <[email-masked]>
Date: Mon, 15 Feb 2016 12:40:20 -0500

Tim,

While multiple registry entries could be used to define each of the report
types (table formats), this would require those registry entries to
replicate the inputs ('fixed parameters' in registry terms) and any outputs
that are common across the different types.  It would be better to have one
registry entry that defined the inputs to the task once, all possible
outputs once and then allowed the outputs to be grouped into different
report types. The report type is just some string value. When the tables
were sent from the MA to the Collector they would then just need to include
the report type.

Example:
1) Task measures bandwidth for one or more streams
2) One registry entry is defined for the task. It includes:
    a) the numerous inputs that define the stream's packet format
    b) an output definition for the 'last one second bandwidth'
    c) an output definition for the 'average bandwidth since task start'
3) The 'last one second bandwidth' is only useful while the task is running
so it is assigned to the report type 'running'.  The description of the
output
4) The 'average bandwidth since task start' is useful while the task is
running and as a final value when the task completes so it is assigned to
the report types 'running' and 'final'.
5) When the reports are sent from the MA, they indicate the report type.
This is where I suggested the 'tag' object in the report could be used.

In the end I am asking about these three issues:
1) How does a task properly define multiple report types (tables)?
2) How does the MA identify the report type being sent?
Even if #1 is addressed using multiple registry entries, and #2 is
addressed by including a reference to the registry entry in the
'metric/uri' object of the task within the report, there is still this
issue:
3) How does the MA send multiple report types at the same time? My
understanding of the current report definition is that it limits each task
to having one table definition per POST.  The MA could send each table
definition to the collector separately but allowing each task to send an
array of table definitions would be more efficient.

You brought up a fourth issue which is how the data in each table is mapped
to the registry.  I have been making the assumption this was going to be
accomplished by setting the column header value in the report to one of the
summary/name values defined in the registry for the metric.

Ron

On Sun, Feb 14, 2016 at 11:11 AM, Carey, Timothy (Nokia - US) <
[email-masked]> wrote:

> The columns are for the task and would have to have a union of all the
> metrics for registry entries. – Is this what you were asking?
>
>
>
> *From:* Ron Stana [mailto:[email-masked]]
> *Sent:* Friday, February 12, 2016 4:31 PM
> *To:* [email-masked]
> *Cc:* [email-masked]
> *Subject:* [lmap] Support for multiple report table formats from one task
>
>
>
> LMAP Team,
>
>
>

---

From: "Carey, Timothy (Nokia - US)" <[email-masked]>
Date: Mon, 15 Feb 2016 22:31:08 +0000

Ron,

I would talk to the IPPM team then – its really how they define the metrics that make up the registry.

As for a report per task constraint – I don’t know if saving inputs and outputs as you suggest warrant breaking that constraint – why not just have 2 tasks and duplicate the registry entry.

I like the when the task is complete – generate the results for reporting and then report results based on the schedule to the collector. I know in our BBF implementation this works quite well.

Otherwise you get into the nasty problem of a state machine that correlates running the tests with reporting results in the context of the overarching scheduled task (You call it running and final but there is a context to those states for that particular task).

BR,
Tim

From: Ron Stana [mailto:[email-masked]]
Sent: Monday, February 15, 2016 5:40 PM
To: Carey, Timothy (Nokia - US)
Cc: [email-masked]; [email-masked]
Subject: Re: [lmap] Support for multiple report table formats from one task

Tim,

While multiple registry entries could be used to define each of the report types (table formats), this would require those registry entries to replicate the inputs ('fixed parameters' in registry terms) and any outputs that are common across the different types.  It would be better to have one registry entry that defined the inputs to the task once, all possible outputs once and then allowed the outputs to be grouped into different report types. The report type is just some string value. When the tables were sent from the MA to the Collector they would then just need to include the report type.

Example:
1) Task measures bandwidth for one or more streams
2) One registry entry is defined for the task. It includes:
    a) the numerous inputs that define the stream's packet format
    b) an output definition for the 'last one second bandwidth'
    c) an output definition for the 'average bandwidth since task start'
3) The 'last one second bandwidth' is only useful while the task is running so it is assigned to the report type 'running'.  The description of the output
4) The 'average bandwidth since task start' is useful while the task is running and as a final value when the task completes so it is assigned to the report types 'running' and 'final'.
5) When the reports are sent from the MA, they indicate the report type. This is where I suggested the 'tag' object in the report could be used.

In the end I am asking about these three issues:
1) How does a task properly define multiple report types (tables)?
2) How does the MA identify the report type being sent?
Even if #1 is addressed using multiple registry entries, and #2 is addressed by including a reference to the registry entry in the 'metric/uri' object of the task within the report, there is still this issue:
3) How does the MA send multiple report types at the same time? My understanding of the current report definition is that it limits each task to having one table definition per POST.  The MA could send each table definition to the collector separately but allowing each task to send an array of table definitions would be more efficient.

You brought up a fourth issue which is how the data in each table is mapped to the registry.  I have been making the assumption this was going to be accomplished by setting the column header value in the report to one of the summary/name values defined in the registry for the metric.

Ron

On Sun, Feb 14, 2016 at 11:11 AM, Carey, Timothy (Nokia - US) <[email-masked]<mailto:[email-masked]>> wrote:
Ron,

BR,
Tim

From: Ron Stana [mailto:[email-masked]<mailto:[email-masked]>]
Sent: Friday, February 12, 2016 4:31 PM
To: [email-masked]<mailto:[email-masked]>
Cc: [email-masked]<mailto:[email-masked]>
Subject: [lmap] Support for multiple report table formats from one task

LMAP Team,

Use Cases:

Ron

---

From: "Carey, Timothy (Nokia - US)" <[email-masked]>
Date: Tue, 16 Feb 2016 06:14:20 +0000

Ron,

I was thinking about this some more last night.

Are you asking to categorize the reports by registry entry? - which I think you call report type?  If so I don’t see an issue with that – we can use the registry entry as part of the report results…

BR,
Tim

From: Carey, Timothy (Nokia - US)
Sent: Monday, February 15, 2016 10:31 PM
To: 'Ron Stana'
Cc: [email-masked]; [email-masked]
Subject: RE: [lmap] Support for multiple report table formats from one task

Ron,

BR,
Tim

Cc: [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>

Tim,

Example:

Ron

BR,
Tim

LMAP Team,

Use Cases:

Ron

---

From: Ron Stana <[email-masked]>
Date: Tue, 16 Feb 2016 09:46:23 -0500

Tim,

I am trying to determine how one task can produce results that require
multiple table formats (what I called report types) and send them to the
collector at the same time.  Most tasks we have will require this type of
functionality.

This is my understanding of what we have discussed so far:
1) The registry would be used to define the different table formats...for
the purpose of this discussion let's go with your suggestion that there
would be one entry for each table format.
2) The multiple registry entries are then assigned to a single task using
multiple instances of the the lmap/tasks/task/metric object.
3) When the task goes to send these different tables of results to the
Collector, it identifies the table that is being sent using a single
instance of the the report/task/metric object.

If we agree thus far then the part that remains is how to send multiple
tables to the collector at one time.  The current report model only allows
each task to have one entry so the MA would have to POST a separate report
for each table.  I am asking if the tables could all be sent together.

Ron

On Tue, Feb 16, 2016 at 1:14 AM, Carey, Timothy (Nokia - US) <
[email-masked]> wrote:

> Are you asking to categorize the reports by registry entry? - which I
> think you call report type?  If so I don’t see an issue with that – we can
> use the registry entry as part of the report results…
>
>
>
> BR,
>
> Tim
>
>
>
> *From:* Carey, Timothy (Nokia - US)
> *Sent:* Monday, February 15, 2016 10:31 PM
> *To:* 'Ron Stana'
> *Cc:* [email-masked]; [email-masked]
> *Subject:* RE: [lmap] Support for multiple report table formats from one
> task
>
>
>
> Ron,
>
>
>
> As for a report per task constraint – I don’t know if saving inputs and
> outputs as you suggest warrant breaking that constraint – why not just have
> 2 tasks and duplicate the registry entry.
>
>
>
> I like the when the task is complete – generate the results for reporting
> and then report results based on the schedule to the collector. I know in
> our BBF implementation this works quite well.
>
>
>
> Otherwise you get into the nasty problem of a state machine that
> correlates running the tests with reporting results in the context of the
> overarching scheduled task (You call it running and final but there is a
> context to those states for that particular task).
>
>
>
> BR,
>
> Tim
>
>
>
>
>
> *From:* Ron Stana [mailto:[email-masked]
> <[email-masked]>]
> *Sent:* Monday, February 15, 2016 5:40 PM
> *To:* Carey, Timothy (Nokia - US)
> *Cc:* [email-masked]; [email-masked]
> *Subject:* Re: [lmap] Support for multiple report table formats from one
> task
>
>
>
> Tim,
>
>
>
> 3) The 'last one second bandwidth' is only useful while the task is
> running so it is assigned to the report type 'running'.  The description of
> the output
>
> You brought up a fourth issue which is how the data in each table is
> mapped to the registry.  I have been making the assumption this was going
> to be accomplished by setting the column header value in the report to one
> of the summary/name values defined in the registry for the metric.
>
>
>
> Ron
>
>
>

---

From: "Carey, Timothy (Nokia - US)" <[email-masked]>
Date: Tue, 16 Feb 2016 14:54:38 +0000

Inline <TAC>

From: EXT Ron Stana [mailto:[email-masked]]
Sent: Tuesday, February 16, 2016 2:46 PM
To: Carey, Timothy (Nokia - US)
Cc: [email-masked]; [email-masked]
Subject: Re: [lmap] Support for multiple report table formats from one task

Tim,

I am trying to determine how one task can produce results that require multiple table formats (what I called report types) and send them to the collector at the same time.  Most tasks we have will require this type of functionality.

1)      The registry would be used to define the different table formats...for the purpose of this discussion let's go with your suggestion that there would be one entry for each table format.
<TAC> Agreed

2)      The multiple registry entries are then assigned to a single task using multiple instances of the the lmap/tasks/task/metric object.
<TAC> Agreed
3) When the task goes to send these different tables of results to the Collector, it identifies the table that is being sent using a single instance of the the report/task/metric object.
<TAC> Agreed

If we agree thus far then the part that remains is how to send multiple tables to the collector at one time.  The current report model only allows each task to have one entry so the MA would have to POST a separate report for each table.  I am asking if the tables could all be sent together.
<TAC> Correct – the information model for the report object would need to be modified and reorganized tofurther qualify the results by the metric (registry entry) in addition to the task name.
Ron

On Tue, Feb 16, 2016 at 1:14 AM, Carey, Timothy (Nokia - US) <[email-masked]<mailto:[email-masked]>> wrote:
Ron,

BR,
Tim

Ron,

BR,
Tim

Tim,

Example:

Ron

BR,
Tim

LMAP Team,

Use Cases:

Ron

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Mon, 22 Feb 2016 09:56:11 +0100

Ron,

the YANG data model already has a tag to distinguish multiple
'tables'. But even with that tag in place, I think the reporting model
is not working well yet. I agree that there is room for improvement.

One of the odd things is also that results are produced by actions,
not by claim to report by tasks. This alone causes trouble because
actions may use options that impact the result format. Originally, we
did not have the notion of actions and somehow the result model simply
lags behind.

I need to think more about how to do this properly. I do not have a
solution proposal yet but I agree that the reporting model needs some
more work (once we have the configuration/instruction model stable).

/js

> _______________________________________________
> lmap mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/lmap

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1 | 28759 Bremen | Germany
Fax:   +49 421 200 3103         <http://www.jacobs-university.de/>