Subject: [lmap] LMAP: Defining the Schedule's events

From: "Carey, Timothy (Nokia - US)" <[email-masked]>
Date: Fri, 8 Apr 2016 12:52:00 +0000

Team,

In the IETF#95 LMAP session we had a discussion on the need for the new parameters for the ma-schedule-end and ma-schedule-duration parameters, given that the ma-schedule-start (which is typed as an event)

We said we would show that these parameters were not needed using an example - so here goes.

Lets assume we have a schedule made up of 2 actions (A1, A2) in pipeline mode. The ma-schedule-start would has a periodic event:

Start April 1, 2016 at 02:00:00

End May 1, 2016 at 02:00:00

Interval 86,400 (daily)

Randomness 3600 (within the hour)

So the schedule will reoccur every day starting on April 1 beginning at 2:00pm. (T0) The starting period will be random between 2:00am and 3:00am. (T1) This definition is sufficient for invoking A1 (T1) and is what Al considers as Blue times.

As A1 performs a test A1 places bits on the wire (T2) and collects results (T3).
When A1 completes (T4), A2 is invoked (T5) which places bits on a wire for A2 (T2) and collects results (T3)). Finally the results from the schedule are reported (T6).

T2&T3: Al considers this time to be part of the definition of the metric and are reported as part of the result content (not as an actual parameter). This was his Red time

T4&T5: Are the ma-report-[start|end]-time
T6: Is the ma-report-date

As such - As long as an Event has a defined occurrence (T1=T0 + randomness) all that is necessary is a single parameter to define the occurrence.

Questions?

BR,
Tim

---

From: Ron Stana <[email-masked]>
Date: Sun, 17 Apr 2016 12:34:31 -0400

Tim,

It appears you are making an assumption that each action will end on its
own (T4).  While that is true for turn-up tests like RFC-2544 and Y.1564,
long term performance monitoring tests like TWAMP and Y.1731 are expected
to produce results 24/7 and so do not have a defined end time.  They are
configured with a rate to send packets, the MA measures FD, FDV and FLR and
then the MA periodically aggregates these stats and sends them to the
collector.  The aggregation rate is typically 1, 5 or 15 minutes so reports
contain rolling start/end times as the test executes.

While what I describe above is the typical use case, there are other
scenarios where these same tests can be configured to emulate short random
bursts. In this scenario the tests need to be configured with an initial
start time, an interval to restart the test, a random start offset and a
duration.  For example, start at 1:00 today, restart every 5 minutes
thereafter, each run should be offset by a random value within 4 minutes
and each run should send the data flows for 1 minute.  I think the only
part of this LMAP did not define was the last parameter - the test
duration. While each test could offer test-specific parameters to define
its duration, I thought the proposal to add the 'end' object was to provide
a common approach for this.

Ron

On Fri, Apr 8, 2016 at 8:52 AM, Carey, Timothy (Nokia - US) <
[email-masked]> wrote:

> Team,
>
>
>
> In the IETF#95 LMAP session we had a discussion on the need for the new
> parameters for the ma-schedule-end and ma-schedule-duration parameters,
> given that the ma-schedule-start (which is typed as an event)
>
>
>
> We said we would show that these parameters were not needed using an
> example – so here goes.
>
>
>
> Lets assume we have a schedule made up of 2 actions (A1, A2) in pipeline
> mode. The ma-schedule-start would has a periodic event:
>
> So the schedule will reoccur every day starting on April 1 beginning at
> 2:00pm. (T0) The starting period will be random between 2:00am and 3:00am.
> (T1) This definition is sufficient for invoking A1 (T1) and is what Al
> considers as Blue times.
>
>
>
> When A1 completes (T4), A2 is invoked (T5) which places bits on a wire for
> A2 (T2) and collects results (T3)). Finally the results from the schedule
> are reported (T6).
>
>
>
> T2&T3: Al considers this time to be part of the definition of the metric
> and are reported as part of the result content (not as an actual
> parameter). This was his Red time
>
>
>
> As such – As long as an Event has a defined occurrence (T1=T0 +
> randomness) all that is necessary is a single parameter to define the
> occurrence.
>
>
>
> Questions?
>
>
>
> BR,
>
> Tim
>
> _______________________________________________
> lmap mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/lmap
>
>

---

From: "Carey, Timothy (Nokia - US)" <[email-masked]>
Date: Mon, 18 Apr 2016 01:19:20 +0000

Ron,

Thanks for looking at this.
When I discussed this with Al indicated that these would be part of the metric/action definition.

But if we would want to define a non-metric specific duration attribute of an individual action within the context of the overall schedule (I believe this is what you are requesting?) then we should assign that duration that results in (T4) to the action. Actually you have the same problem with T2 – within the context action – this offset if necessary is different from the schedule start randomness (once the process starts how long to wait before the bits hit the wire).  Maybe that is what we are missing – A timer for an action within the context of the schedule…

Comments?

From: EXT Ron Stana [mailto:[email-masked]]
Sent: Sunday, April 17, 2016 6:35 PM
To: Carey, Timothy (Nokia - US)
Cc: [email-masked]
Subject: Re: [lmap] LMAP: Defining the Schedule's events

Tim,

It appears you are making an assumption that each action will end on its own (T4).  While that is true for turn-up tests like RFC-2544 and Y.1564, long term performance monitoring tests like TWAMP and Y.1731 are expected to produce results 24/7 and so do not have a defined end time.  They are configured with a rate to send packets, the MA measures FD, FDV and FLR and then the MA periodically aggregates these stats and sends them to the collector.  The aggregation rate is typically 1, 5 or 15 minutes so reports contain rolling start/end times as the test executes.

While what I describe above is the typical use case, there are other scenarios where these same tests can be configured to emulate short random bursts. In this scenario the tests need to be configured with an initial start time, an interval to restart the test, a random start offset and a duration.  For example, start at 1:00 today, restart every 5 minutes thereafter, each run should be offset by a random value within 4 minutes and each run should send the data flows for 1 minute.  I think the only part of this LMAP did not define was the last parameter - the test duration. While each test could offer test-specific parameters to define its duration, I thought the proposal to add the 'end' object was to provide a common approach for this.

Ron

On Fri, Apr 8, 2016 at 8:52 AM, Carey, Timothy (Nokia - US) <[email-masked]<mailto:[email-masked]>> wrote:
Team,

We said we would show that these parameters were not needed using an example – so here goes.

Interval 86,400 (daily)

As such – As long as an Event has a defined occurrence (T1=T0 + randomness) all that is necessary is a single parameter to define the occurrence.

Questions?

BR,
Tim

_______________________________________________
lmap mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/lmap

---

From: Ron Stana <[email-masked]>
Date: Mon, 18 Apr 2016 09:40:18 -0400

Tim,

The exact definition of the metric is still unclear to me since I have not
seen an example of a test with average complexity using metrics with LMAP
from beginning to end.  By average complexity I mean a test that allows
independent configuration of multiple simultaneous data flows and produces
multiple report formats during the test duration. The tests I mentioned
previously (RFC-2544, Y.1564, Y.1731 and TWAMP) minimally require this type
of support and are the main tests used in Ethernet service testing today.
None-the-less, my understanding of the metric is that it defines the format
of the packet stream and the methodology of the transmit frequency. These
are two examples from
https://tools.ietf.org/html/draft-ietf-ippm-initial-registry-00

Act_IP_UDP_Poisson_UDP-Payload-250_One-way_Delay_Std_Dev

Act_IP_UDP_Periodic-var_UDP-Payload-142_One-way_Delay_Mean

It does not appear to define the time period over which the
measurement was taken.  This is good because for most tests the
measurement period is a user defined input parameter.  Performance
monitoring tests will report the metric repeatedly during the test
execution.  Referring to the example in my prior email, a TWAMP test
would produce a group of metrics for each data stream every 1, 5 or 15
minutes depending on the test configuration. It is possible the test
could be configured to produce multiple reports - for example at a
short interval for troubleshooting in addition to one of the intervals
previously mentioned for general service monitoring. Thus, the TWAMP
test produces instances of metrics for an unspecified amount of time -
typically until a user manually stops the test.

In summary, from my perspective, the duration of the test is outside
the scope of the metric definition....but maybe I just don't
understand the metric definition well enough.

Ron

On Sun, Apr 17, 2016 at 9:19 PM, Carey, Timothy (Nokia - US) <
[email-masked]> wrote:

> But if we would want to define a non-metric specific duration attribute of
> an individual action within the context of the overall schedule (I believe
> this is what you are requesting?) then we should assign that duration that
> results in (T4) to the action. Actually you have the same problem with T2 –
> within the context action – this offset if necessary is different from the
> schedule start randomness (once the process starts how long to wait before
> the bits hit the wire).  Maybe that is what we are missing – A timer for an
> action within the context of the schedule…
>
>
>
> Comments?
>
>
>
> *From:* EXT Ron Stana [mailto:[email-masked]]
> *Sent:* Sunday, April 17, 2016 6:35 PM
> *To:* Carey, Timothy (Nokia - US)
> *Cc:* [email-masked]
> *Subject:* Re: [lmap] LMAP: Defining the Schedule's events
>
>
>
> Tim,
>
>
>

---

From: "MORTON, ALFRED C (AL)" <[email-masked]>
Date: Mon, 18 Apr 2016 11:19:28 -0400

Hi Ron and Tim,

trying to make a few minutes for this...

The Registry Entries* for the Metrics include Run-time Parameters
for the Start time and End Time, but one of them could be interpreted
as a Duration if specified correctly (the Start time must be all zeros,
then the Stop time is interpreted as a duration, AND the actual start
and stop time are controlled through “other means”). See

https://tools.ietf.org/html/draft-ietf-ippm-initial-registry-00#section-4.3.5

“Other means” would be the Schedule object and the Event object.
As I understand it both schedule and event objects have start time
and stop time specifications available, and Event has Duration specification
available which would be useful for recurring events (periodic or calendar).

Eventually, the MA has to report the results of the metric/measurement.
That’s when the actual (RED) start and end times when packets were actually
flowing between measurement points are reported, according to the Registry Entry.
See:

https://tools.ietf.org/html/draft-ietf-ippm-initial-registry-00#section-4.4.2

As I said at the meeting, it would be good to look at some examples

which attempt to schedule the “Act_IP_UDP_Round-trip_Delay_Poisson_95th-percentile”

registry entry, as a Periodic event, a one-time event, and a continuous

event as Ron is suggesting.

regards,

Al

* Note that the Registry Entry for a Metric includes the Metric Definition,
the method of measurement, and Run-time parameters.

From: Ron Stana [mailto:[email-masked]]
Sent: Monday, April 18, 2016 9:40 AM
To: Carey, Timothy (Nokia - US)
Cc: [email-masked]; MORTON, ALFRED C (AL)
Subject: Re: [lmap] LMAP: Defining the Schedule's events

Tim,

The exact definition of the metric is still unclear to me since I have not seen an example of a test with average complexity using metrics with LMAP from beginning to end.  By average complexity I mean a test that allows independent configuration of multiple simultaneous data flows and produces multiple report formats during the test duration. The tests I mentioned previously (RFC-2544, Y.1564, Y.1731 and TWAMP) minimally require this type of support and are the main tests used in Ethernet service testing today.  None-the-less, my understanding of the metric is that it defines the format of the packet stream and the methodology of the transmit frequency. These are two examples from https://tools.ietf.org/html/draft-ietf-ippm-initial-registry-00

It does not appear to define the time period over which the measurement was taken.  This is good because for most tests the measurement period is a user defined input parameter.  Performance monitoring tests will report the metric repeatedly during the test execution.  Referring to the example in my prior email, a TWAMP test would produce a group of metrics for each data stream every 1, 5 or 15 minutes depending on the test configuration. It is possible the test could be configured to produce multiple reports - for example at a short interval for troubleshooting in addition to one of the intervals previously mentioned for general service monitoring. Thus, the TWAMP test produces instances of metrics for an unspecified amount of time - typically until a user manually stops the test.

In summary, from my perspective, the duration of the test is outside the scope of the metric definition....but maybe I just don't understand the metric definition well enough.

Ron

On Sun, Apr 17, 2016 at 9:19 PM, Carey, Timothy (Nokia - US) <[email-masked]<mailto:[email-masked]>> wrote:
Ron,

Comments?

From: EXT Ron Stana [mailto:[email-masked]<mailto:[email-masked]>]
Sent: Sunday, April 17, 2016 6:35 PM
To: Carey, Timothy (Nokia - US)
Cc: [email-masked]<mailto:[email-masked]>
Subject: Re: [lmap] LMAP: Defining the Schedule's events

Tim,

Ron

Interval 86,400 (daily)

Questions?

BR,
Tim

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Tue, 19 Apr 2016 11:19:15 +0200

Yes, this is the reason I added this.

/js

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1 | 28759 Bremen | Germany
Fax:   +49 421 200 3103         <http://www.jacobs-university.de/>

---

From: "Carey, Timothy (Nokia - US)" <[email-masked]>
Date: Tue, 19 Apr 2016 12:01:25 +0000

Al,
Inline <TAC>

From: EXT MORTON, ALFRED C (AL) [mailto:[email-masked]]
Sent: Monday, April 18, 2016 5:19 PM
To: Ron Stana; Carey, Timothy (Nokia - US)
Cc: [email-masked]
Subject: RE: [lmap] LMAP: Defining the Schedule's events

Hi Ron and Tim,

<TAC> The schedule’s timer is sufficient to state when the initial action(s) are invoked. I say actions because the execution mode for the schedule’s action’s could be parallel. In the case when the schedule’s execution mode is sequential then it would be sufficient to state when the first action is invoked.
What the schedule’s timer doesn’t do is specify the duration of the individual action(s) or when the second action is executed in relation to the completion of the first action. What we would need to do is either: Pass the duration of action based on the options (T0 = zero, Tf=some duration) or we could put a timer on the action – which would provide the external control that would allow the start to be better controlled in the context of the overall schedule.

<TAC> These seem to be the actual – not configured start and end times
Interesting I think there might be a problem in the draft with Tf as it would be “end” of the interval not the “start” of the interval?
   T0 the start of a measurement interval, (format "date-and-time" as
      specified in Section 5.6 of [RFC3339]<https://tools.ietf.org/html/rfc3339#section-5.6>, see also Section 3 of<https://tools.ietf.org/html/rfc6991#section-3>
      [RFC6991]<https://tools.ietf.org/html/rfc6991#section-3>).  The UTC Time Zone is required by Section 6.1 of<https://tools.ietf.org/html/rfc2330#section-6.1>
      [RFC2330]<https://tools.ietf.org/html/rfc2330#section-6.1>.

   Tf the start of a measurement interval, (format "date-and-time" as
      specified in Section 5.6 of [RFC3339]<https://tools.ietf.org/html/rfc3339#section-5.6>, see also Section 3 of<https://tools.ietf.org/html/rfc6991#section-3>
      [RFC6991]<https://tools.ietf.org/html/rfc6991#section-3>).  The UTC Time Zone is required by Section 6.1 of<https://tools.ietf.org/html/rfc2330#section-6.1>
      [RFC2330]<https://tools.ietf.org/html/rfc2330#section-6.1>.

regards,

Al

Cc: [email-masked]<mailto:[email-masked]>; MORTON, ALFRED C (AL)

Tim,

Ron

Comments?

Tim,

Ron

Interval 86,400 (daily)

Questions?

BR,
Tim

---

From: "MORTON, ALFRED C (AL)" <[email-masked]>
Date: Tue, 19 Apr 2016 08:24:45 -0400

Yes, in Tf, s/start/end/. 
Thanks for the catch,
Al
________________________________________
From: Carey, Timothy (Nokia - US) [[email-masked]]
Sent: Tuesday, April 19, 2016 8:01 AM
To: MORTON, ALFRED C (AL); Ron Stana
Cc: [email-masked]
Subject: RE: [lmap] LMAP: Defining the Schedule's events

Al,
Inline <TAC>

Hi Ron and Tim,

regards,

Al

Tim,

Ron

Comments?

Tim,

Ron

Interval 86,400 (daily)

Questions?

BR,
Tim

---

From: "Carey, Timothy (Nokia - US)" <[email-masked]>
Date: Wed, 27 Apr 2016 15:09:01 +0000

Team,

This might have been lost in the email threads so I am reviving the discussion.

From: Carey, Timothy (Nokia - US)
Sent: Tuesday, April 19, 2016 7:01 AM
To: 'EXT MORTON, ALFRED C (AL)'; Ron Stana
Cc: [email-masked]
Subject: RE: [lmap] LMAP: Defining the Schedule's events

Al,
Inline <TAC>

Hi Ron and Tim,

regards,

Al

Tim,

Ron

Comments?

Tim,

Ron

Interval 86,400 (daily)

Questions?

BR,
Tim