Subject: [lmap] proposed information model / data model changes

From: Juergen Schoenwaelder <[email-masked]>
Date: Thu, 16 Apr 2015 21:33:53 +0200

Hi,

we are working on an implementation of the data model and we found a
couple of things non-intuitive or more difficult to implement than
needed or complex to use for certain use cases. After discussing
things for a while internally, we consolidated our thoughts in the
following list of proposed changes. Note that some of the listed
changes 1)-6) interact. I would be nice if could get feedback quickly
so we can get our code do the 'right' thing.

1) Execution model for actions: A schedule has a list of task with
   parameters (we call them actions). The info model currently says
   that the tasks are executed sequentially. Or in Unix shell syntax,
   the actions a_1 to a_n are executed as follows:

      a_1; a_2; ...; a_n

   There is no data passed between the actions in the above example
   (unless you configure specific action destinations options). We
   believe there are other execution models that should be supported,
   in particular the piped model where data flows from one task to the
   next task:

      a_1 | a_2 | ... | a_n

   The prime use case is to have a measurement task feeding results
   directly into a reporting task. This is needed to support simple
   ad-hoc measurements that report data immediately (the help-desk use
   case). Right now, it is unclear and/or cumbersome to support this
   use case (in which two schedule objects are involved).

   The third possible execution model is that all tasks are started
   concurrently, in Unix shell syntax:

      a_1 & a_2 & ... & a_n &

   This may appear a bit less useful to have but there are situations
   where this is actually useful (e.g., if you want to start N long
   running passive monitoring tasks from a schedule bound to a startup
   timing object; right now you have to create N different schedules
   (because schedules are executed in parallel).

   Our proposal is thus to change the information model and the data
   model to support all three types of invocations. If done properly,
   it may even be possible to mix things, e.g.

     a_1 & a_2 | a_3; a_4

   (Start a_1 but do not wait for completion, run a_2 and feed the
   output into a_3, once a_3 is done, run a_4.)

   To support this, it is necessary to add another element to
   ma-action-obj indicating whether the action is executed
   sequentially, concurrently, or as part of a pipe.

2) Remove the notion of multiple task outputs. Compensate that by
   having a tag in every result row and allow filter tasks.

   During implementation, we found it complicated to deal with
   multiple task outputs. It is also unclear how one discovers the
   outputs supported by a given task. We find the model used by at
   least one large scale measurement platform way simpler to work
   with. On that platform, every result row carries a tag that
   identifies the type of a result row. A measurement task that can
   measure different metrics simply creates differently tagged result
   rows. Since we have moved to a generic model where tasks can also
   act as filters, it seems easier to have only a single output for
   task and to use filters to filter output that is not needed in a
   task processing chain. In other words, instead of having an action
   a_1 in schedule s_1 feedings different output to other actions in
   other schedules

     s_1: a_1 -o_2-> s_2.a_2, -o_3-> s_3.a_2

   we would do

     s_1: a_1 -> s_2.a_1, -> s_3.a_1

     s_2: a_1 | a_2

     s_3: a_1 | a_2

   where s_2.a_1 and s_3.a_1 are acting as filters. We found that the
   proposed model simplifies implementation and at the same time adds
   flexibility (since filtering can be more flexible than hard wired
   task outputs). Of course, this requires to have a mechanism in
   place to pipeline data.

3) Remove the feature to forward task output to a specific action
   within a schedule; instead output is directed to another schedule.

   Right now, data is moved from a source task in an action list of a
   schedule to a receiving task in an action list of another schedule.
   It is unclear why this is useful; we had trouble finding convincing
   use cases. It seems that it is sufficient to forward to the first
   task of a different schedule only. This simplifies the data (and
   information model). To continue the previous example, we propose to
   reduce things to this:

     s_1: a_1 -> s_2, -> s_3

     s_2: a_1 | a_2

     s_3: a_1 | a_2

   The proposed changes 2) and 3) together allow to eliminate
   ma-action-dest-obj and to replace it with a list of
   ma-action-dest-schedule-name values in the ma-action-obj.

4) Timing should be generalized to events.

   There was a comment at the last IETF meetings that IEEE likes to
   bind the invocation of measurements to events such as reaching a
   certain location. Other people commented that they like to bind the
   execution of measurements to the status of certain interfaces
   (e.g., run the connectivity test when the WAN interface changes
   into the 'up' state).

   To accommodate future extension in this direction, we propose to
   change the information model and the data model such that schedules
   are a list of actions bound to an event. Timing objects like
   periodic and calendar or one-off timings simply generate
   events. The existing startup and immediate timing objects
   effectively are not timing objects but events. This is mostly an
   editorial change but it supports future extensibility for other
   event sources.

5) Separate the reporting of meta-information (possibly verbose since
   it includes all options) from the reporting of results to make
   result reports more efficient.

   Right now, the information model mandates that every report
   contains the complete task configuration so that results can be
   interpreted correctly. This, however, introduces significant
   overhead. Assuming that task configurations change rarely compared
   to the frequency with which we ship result records, we propose to
   decouple things. This allows to send meta-information at a low
   frequency or only when it changes. (This separation has already
   proven useful in IPFIX where templates are send less frequently
   than data records.)

   This change is relatively trivial to implement; simply reorganize
   the ma-report-obj and ma-report-task-obj.

6) Change timers to use meaningful units rather than milliseconds
   everywhere.

   Right now, all timers or timeouts use the same unit (milliseconds).
   This is, however, not a suitable unit for some of the timers or
   timeouts (e.g., ma-controller-lost-timeout). We propose to use
   units that a meaningful for the specific information model element
   instead of using the highest resolution possible everywhere.

   This change is relatively trivial to implement.

X) The structural changes would be roughly:

     object {
         string                ma-action-name;
         string                ma-action-task-name;
+        enumeration           ma-action-exec-type;
        [ma-option-obj         ma-action-task-options<0..*>];
-       [ma-action-dest-obj    ma-action-destinations<0..*>;]	
+       [string                ma-action-sendto-schedule<0..*>;]
     } ma-action-obj;

-    object {
-        string    ma-action-dest-schedule-name;
-        string    ma-action-dest-action-name;
-       [int       ma-action-dest-action-outputs<0..*>;]
-    } ma-action-dest-obj;

     object {
         datetime            ma-report-date;
        [uuid                ma-report-agent-id;]
        [string              ma-report-group-id;]
        [ma-report-task-obj  ma-report-tasks<0..*>];
+       [ma-report-row-obj   ma-report-data<0..*>];
     } ma-report-obj;

     object {
         string              ma-report-task-name;
        [uri                 ma-report-task-registry-entry;]
        [ma-option-obj       ma-report-scheduled-task-options<0..*>];
        [string              ma-report-task-cycle-id;]
         string              ma-report-task-column-labels<0..*>;
-        ma-report-row-obj   ma-report-task-rows<0..*>;
     } ma-report-task-obj;

     object {
         datetime            ma-report-result-start-time;
        [datetime            ma-report-result-end-time;]
         string              ma-report-result-conflicts<0..*>;
+	 string              ma-report-result-tag;
         data                ma-report-result-values<0..*>;
     } ma-report-row-obj;

/js

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1 | 28759 Bremen | Germany
Fax:   +49 421 200 3103         <http://www.jacobs-university.de/>

---

From: "MORTON, ALFRED C (AL)" <[email-masked]>
Date: Sun, 19 Apr 2015 18:44:54 -0400

Hi Juergen,

you wrote:
...

   we would do

     s_2: a_1 | a_2

     s_3: a_1 | a_2

...

Last week, I started to see a related issue with another part of 
supporting picture for LMAP. It's in the way we have the 
performance metrics registry is organized.

The info model says, in section 3:

   3.  Task Configurations.  A set of Task Configurations is used to
       configure the Tasks that are run by the MA.  This includes _the_
       _registry_entry for the Task and any configuration parameters.

I take this to mean that each task can have one registry entry. 
We have organized the performance metrics registry so that each
entry for *active* metrics and methods will describe the packet stream
to be sent and a *single* form of output result. 
We are also likely to have some sets of registry entries that use 
identical stream characteristics, but produce different results.
Results like mean delay, 95th %-time of delay variation, loss ratio,
and others could all use the same stream. In fact, we want to avoid
having the MA interpret that each task/registry entry with the 
*same* stream specification would each produce its own stream
when all we want is four or five different metrics measured at the
same time.

So, I see the problem shaping-up differently, if I understood
your scenario. Rather than tasks with multiple outputs, I imagined
multiple tasks/registry entries that have a single output, 
but ideally depend on a single measurement resource, the 
packet stream.

In the output, I think we both want:

ID  Time   Stream   Result tag   Result  
01  01:00  100pps   Loss_Ratio   0.001
01  01:00  100pps   Mean_delay   0.015
01  01:00  100pps   PDV_95%      0.030
01  ...

where "ID" groups these measurements and results.
If the task configuration provided a common ID,
we would be able to combine measurements on a single
stream when desired, or measure on separate streams
if two tasks had different IDs.

OTOH, if we can somehow combine tasks to share resources
already, then there would likely be multiple outputs,
and I support your proposal.

regards,
Al

________________________________________
From: lmap [[email-masked]] On Behalf Of Juergen Schoenwaelder [[email-masked]]
Sent: Thursday, April 16, 2015 3:33 PM
To: [email-masked]
Subject: [lmap] proposed information model / data model changes

Hi,

      a_1; a_2; ...; a_n

      a_1 | a_2 | ... | a_n

      a_1 & a_2 & ... & a_n &

     a_1 & a_2 | a_3; a_4

   we would do

     s_2: a_1 | a_2

     s_3: a_1 | a_2

     s_1: a_1 -> s_2, -> s_3

     s_2: a_1 | a_2

     s_3: a_1 | a_2

     object {
         string                ma-action-name;

-    object {

     object {
         datetime            ma-report-date;

     object {

     object {

/js

_______________________________________________
lmap mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/lmap

---

From: <[email-masked]>
Date: Wed, 22 Apr 2015 10:11:10 +0000

1) Execution model: Sounds like a nice idea if people don't find it too confusing and it's easy to implement (in code, not in the information model). 

2) Task outputs: I agree multiple task outputs have always been quite difficult. The bit of the picture that is missing is that tasks could have configuration parameters to indicate which tags they use/filter.

3) Output to Schedule: Generally I think this can work - e.g. having a post-processing or reporting task on a delayed schedule. If we really think this makes things easier then OK.

Overall I can see that the new suggestions 1-3 can work. I guess I'm unclear whether they a) make things easier to understand and b) are actually easy to implement.

4) Timing as events: OK. I think this is just semantics and it probably does makes more sense to think of things like 'startup' as an event rather than a timing.

5) Reporting of metadata: There is an incorrect assumption that all the task configurations have to be reported. It is explained that this is a choice of the reporting task and can be configured with an option on more complex reporting tasks. So a reporting task can already do what you suggest: i.e. send metadata only when it changes. I'm not sure I want to lose the ability to include the metadata in the same report as the results - this makes it easy for a Collector to de-normalise settings into the result data if required rather than having to look up the current applied metadata in order to be able to perform such enhancement.

6) different timing units: no strong opinion. I don't see it really improves things much to have coarse units (e.g. seconds) though.

Trevor.

-----Original Message-----
From: lmap [mailto:[email-masked]] On Behalf Of Juergen Schoenwaelder
Sent: 16 April 2015 20:34
To: [email-masked]
Subject: [lmap] proposed information model / data model changes

Hi,

      a_1; a_2; ...; a_n

      a_1 | a_2 | ... | a_n

      a_1 & a_2 & ... & a_n &

     a_1 & a_2 | a_3; a_4

   we would do

     s_2: a_1 | a_2

     s_3: a_1 | a_2

     s_1: a_1 -> s_2, -> s_3

     s_2: a_1 | a_2

     s_3: a_1 | a_2

     object {
         string                ma-action-name;

-    object {

     object {
         datetime            ma-report-date;

     object {

     object {

/js

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Thu, 30 Apr 2015 11:39:39 +0200

> 5) Reporting of metadata: There is an incorrect assumption that all the task configurations have to be reported. It is explained that this is a choice of the reporting task and can be configured wi…

Yes, I checked the text again and this was a misunderstanding on our
side. That said, it remains unclear how this is configured. The text
is simply saying referring to "setting a configurable parameter in the
Task Configuration" and I guess our confusion comes from the fact that
there is no such parameter defined in the information model.

Anyway, looking at the object definitions, it seems that ma-report-obj
includes an optional sequence of ma-report-task-obj and
ma-report-task-obj in turn includes a mandatory sequence of
ma-report-row-obj (but the sequence is allowed to be empty).  Would it
not make more sense to change ma-report-task-obj as follows

     object {
         string              ma-report-task-name;
        [uri                 ma-report-task-registry-entry;]
        [ma-option-obj       ma-report-scheduled-task-options<0..*>];
        [string              ma-report-task-cycle-id;]
        [string              ma-report-task-column-labels<0..*>;]
        [ma-report-row-obj   ma-report-task-rows<0..*>;]
     } ma-report-task-obj;

to mark ma-report-task-column-labels and ma-report-task-rows more
clearly as optional?

> 6) different timing units: no strong opinion. I don't see it really
> improves things much to have coarse units (e.g. seconds) though.

It likely does not make much sense to configure, for example,
ma-controller-lost-timeout in milliseconds granularity but right now
you require an implementation to implement a milliseconds granularity
(or to cheat by simply rounding things up to something like seconds
granularity).  In other words, the model allows likely not useful
configurations (or even harmful configuration errors) and the model
raise the implementation requirements to support somewhat questionable
and potentially harmful settings.

/js