Subject: Re: [lmap] I-D Action: draft-ietf-lmap-information-model-12.txt

From: "MORTON, ALFRED C (AL)" <[email-masked]>
Date: Wed, 16 Nov 2016 02:38:03 +0000

Hi Juergen,

In the latest Info Model...
> -----Original Message-----
...
> 
> There's also a htmlized version available at:
> https://tools.ietf.org/html/draft-ietf-lmap-information-model-12
> 
[ACM] 
The metric registry has become generic:

3.10.  Common Objects: Registry Information

   Tasks and actions can be associated with entries in a registry.  A
   registry object refers to an entry in a registry (identified by a
   URI) and it may define a set of roles.

3.10.1.  Definition of ma-registry-obj

     object {
         uri                 ma-registry-uri;
        [string              ma-registry-role<0..*>;]
     } ma-registry-obj;

   The ma-registry-obj refers to an entry of a registry and it defines
   the associated role(s).  The ma-registry-obj consists of the
   following elements:

   ma-registry-uri:          A URI identifying an entry in a registry.

   ma-registry-role:         An optional and possibly empty unordered
                             set of roles for the identified registry
                             entry.
-=-=-=-=-=-=-=-=-

It's not clear to me how the generic object works with 
respect to the performance metric registry. It was clear when the
metric aspect was prominent (ma-metric-registry-obj).
This may be because the idea of an LMAP task registry
has been mentioned, but I haven't seen any details about 
the LMAP task registry and maybe an example would help. 
It would be good to see how the LMAP Task Registry and 
IANA Performance metrics registry are used together.

Since I'm mostly interested in using LMAP for control 
and reporting of registered performance metrics, the
composition of these measurement tasks should be as
straightforward as possible.

A follow-up question is how this will change the data model,
which still appears to contain specific references to metrics.

regards,
Al

---

From: <[email-masked]>
Date: Wed, 16 Nov 2016 09:07:16 +0000

Sorry for having been quiet on this recently. Thanks for all the work on this Juergen. 

I think it would be good to have a section that provides an example - to act as some kind of link from the architecture doc to the info model. The info model is now more generic (eg no specific measurement tasks) - so I think it would be useful to show a simple example with measurement & reporting.  

Thanks,
phil

-----Original Message-----
From: lmap [mailto:[email-masked]] On Behalf Of MORTON, ALFRED C (AL)
Sent: 16 November 2016 02:38
To: [email-masked]
Subject: Re: [lmap] I-D Action: draft-ietf-lmap-information-model-12.txt

Hi Juergen,

...
[ACM]

     object {
         uri                 ma-registry-uri;

It's not clear to me how the generic object works with respect to the performance metric registry. It was clear when the metric aspect was prominent (ma-metric-registry-obj).
This may be because the idea of an LMAP task registry has been mentioned, but I haven't seen any details about the LMAP task registry and maybe an example would help. 
It would be good to see how the LMAP Task Registry and IANA Performance metrics registry are used together.

Since I'm mostly interested in using LMAP for control and reporting of registered performance metrics, the composition of these measurement tasks should be as straightforward as possible.

A follow-up question is how this will change the data model, which still appears to contain specific references to metrics.

regards,
Al

_______________________________________________
lmap mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/lmap

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Wed, 16 Nov 2016 10:39:09 +0100

Al,

since the edit is essentially just a name change, why do you think
this makes things unclear? We just do not constrain the URI to point
to a metrics registry. The motivation is that there are LMAP tasks
that do not do measurements and thus do not require metric
definitions.

/js

PS: I personally believe that an LMAP task registry would be a very
    good idea but I do not want to hold off the information model any
    longer.

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1 | 28759 Bremen | Germany
Fax:   +49 421 200 3103         <http://www.jacobs-university.de/>

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Wed, 16 Nov 2016 10:42:16 +0100

Phil,

I am not sure which example you find missing. There is an example for
an execution in section 4. There is no concrete serialization example
because the information model is abstract and does not define any
serialization rules. A concrete example (in XML serialization) can be
found in the YANG data model - and the grand plan was to have the JSON
serialization of the same model in the RESTCONF document.

/js

---

From: <[email-masked]>
Date: Wed, 16 Nov 2016 15:42:51 +0000

The Section 4 example is about timing of events & schedules - pipelining /serialise etc - which is useful.

I meant something like: take some variant of the example in RFC7594 and explain how to use the information model
<<   The Controller instructs one or more MAs and communicates the set of
   Measurement Tasks an MA should perform and when.  For example, it may
   instruct an MA at a home gateway: "Measure the 'UDP latency' with
   www.example.org; repeat every hour at xx.05".  The Controller also
   manages an MA by instructing it on how to report the Measurement
   Results, for example: "Report results once a day in a batch at 4am".

Strictly speaking this is not needed, but I think it would be useful, as the information model is now quite abstract (eg measurement tasks and registries no longer immediately obvious - as they're just tasks & registires). If other people think this is pointless, then I'm ok. 

-----Original Message-----
From: Juergen Schoenwaelder [mailto:[email-masked]] 
Sent: 16 November 2016 09:42
To: Eardley,PL,Philip,TUB8 R <[email-masked]>
Cc: [email-masked]; [email-masked]
Subject: Re: [lmap] I-D Action: draft-ietf-lmap-information-model-12.txt

Phil,

I am not sure which example you find missing. There is an example for an execution in section 4. There is no concrete serialization example because the information model is abstract and does not define any serialization rules. A concrete example (in XML serialization) can be found in the YANG data model - and the grand plan was to have the JSON serialization of the same model in the RESTCONF document.

/js

---

From: "Bugenhagen, Michael K" <[email-masked]>
Date: Wed, 16 Nov 2016 15:56:43 +0000

Phil, Juergen,

I apologize ahead of time for the long-winded retrospect… but here’s something I’ve run across you may be able to consider in your framework.

        I reviewed the draft, and one thing that seems to be missing here is any “NAK” from the test server when it’s busy or overloading.

Given the MA’s can automatically delay testing due to “cross traffic” there should be some CAC state tracking so Op’s knows what is happening when tests don’t get run because the test server was busy.    This is kind of controller 101 for capacity planning, and ops’ traffic engineering for the testing, and of course logging.

    I’m not sure there is a “test server behavior” that tracks this in the information model, but all the current carrier grade systems (systems that telecoms use) have them.
It would be “nice” if the MA knew why a test server ignored a test (I’m assuming they only allow so many connections so a failed / timed out test can occur), but I’m not sure if there are any provisions for this.

   In the past I have seen “test software” (MA’s ) get turned up without telling the “op’s teams” who maintain the servers – which results in “HOT” servers that give bad results because they don’t have any CAC, or throttling on how many tests are concurrently run, and ended up as a congestion point.   – this fails the ISO9k & 7K testing standards for the most part.

      Given the criticality of this, it’s a best practice to communicate the test server NAK / Busy, .. to the MA’s so the system is more full proof.
   Especially when the customer, and the test server may be two different unaffiliated parties.

So – test system best practices would say -… make sure you have a CAC on the test servers and communicate that to the MA’s so they know what’s going on.

Best,
Mike

On 11/16/16, 9:42 AM, "lmap on behalf of [email-masked]" <[email-masked] on behalf of [email-masked]> wrote:

    -----Original Message-----
    From: Juergen Schoenwaelder [mailto:[email-masked]]
    Sent: 16 November 2016 09:42
    To: Eardley,PL,Philip,TUB8 R <[email-masked]>
    Cc: [email-masked]; [email-masked]
    Subject: Re: [lmap] I-D Action: draft-ietf-lmap-information-model-12.txt

    Phil,

    /js

This communication is the property of CenturyLink and may contain confidential or privileged information. Unauthorized use of this communication is strictly prohibited and may be unlawful. If you have received this communication in error, please immediately notify the sender by reply e-mail and destroy all copies of the communication and any attachments.

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Wed, 16 Nov 2016 17:15:39 +0100

Dear Michael,

what is a 'test server' - the 'thing' a measurement task on a
measurement agent runs a test against? If so, this is outside the
scope of the information model. Even if the 'test server' would be run
under the control of an LMAP measurement agent, it would be a task,
i.e., the CAC (Call Admission Control?) would have to be implemented
as part of the measurement task, not in the LMAP scheduler.

The only thing that _could_ be relevant for the LMAP information model
that I see in your message is throttling, i.e., suspending schedules
if there are too many failures.

/js

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Wed, 16 Nov 2016 17:20:54 +0100

> Strictly speaking this is not needed, but I think it would be useful, as the information model is now quite abstract (eg measurement tasks and registries no longer immediately obvious - as they're…

My preference would have been to make the document even more concise.
The reason is that maintaining consistency is a pain. And I believe we
actually have a reasonable balance between explanatory text and
concrete definitions. And the document does not exist in isolation.

/js

---

From: "Bugenhagen, Michael K" <[email-masked]>
Date: Wed, 16 Nov 2016 16:32:02 +0000

Juergen,

   In TWAMP this was handled via a “setting” of how many sessions / streams the MA could support.
       In a LMAP type system that problem is more double due to the probability of over running a dedicated test server complex.

        My concern is that Large scale projects have encountered this occurrence -  Maybe it’s the fault of the test server manager not setting a threshold as was created in TWAMP to ensure the situation doesn’t occur.  But it is real, and has occurred so I thought it was worth pointing out.

Given the MA framework as described the “control” should follow the TWAMP CAC / Session method to make sure test servers don’t get overloaded, with some utilization / call blocking stat’s, which are normal for system capacity management.

   If it is out of scope, some informative text may be appropriate to alert potential users of the spec of the issue…

         If nothing else, maybe a big red warning label?   ☺

Regards,
Mike

On 11/16/16, 10:15 AM, "lmap on behalf of Juergen Schoenwaelder" <[email-masked] on behalf of [email-masked]> wrote:

    Dear Michael,

    /js

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Wed, 16 Nov 2016 18:50:58 +0100

Mike,

at this level of abstraction, it sounds more like material that should
perhaps more deeply discussed in the framework (RFC 7594). The LMAP
information model is all about describing how the internals of the
LMAP measurement agents work. I am not sure this is the proper place
to state that measurement protocols should have proper rate limiting
and overload controls.

I searched for 'limit' and 'load' and 'rate' in RFC 7594 and there is
some discussion but mostly limited to possible load issues on the
collector, load issues on the path, and that suppression may be used
to handle overload situations.

/js

---

From: "Bugenhagen, Michael K" <[email-masked]>
Date: Wed, 16 Nov 2016 18:30:28 +0000

I agree.

     My thought here was that a test server is a MA….  To some extent.

So, the TWAMP method might apply globally – but I haven’t been able to keep up with the LMAP work so I thought I owed the group the input on “hey we’ve seen this in real testing” type inputs.

That way the group has a chance to react if something needs to be fixed…
Not sure if there is anything to fix, but at least I communicated it.

Wish I was at the meeting with you guys!!

Best regards,
Mike

On 11/16/16, 11:50 AM, "Juergen Schoenwaelder" <[email-masked]> wrote:

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Wed, 16 Nov 2016 23:39:30 +0100

On Wed, Nov 16, 2016 at 08:55:35PM +0000, MORTON, ALFRED C (AL) wrote:
> 
> Help us understand how you think LMAP will work now,
>

Al,

it works as good or as bad as before - the URI can still point to the
metrics registry. If you know how the metrics registry works, things
will continue to function.

The point is that there are tasks that do not collect metrics. (I
thought we discussed this and settled this before.)

/js

---

From: "MORTON, ALFRED C (AL)" <[email-masked]>
Date: Wed, 16 Nov 2016 20:55:35 +0000

> Sent: Wednesday, November 16, 2016 11:21 AM
> To: [email-masked]
> Cc: MORTON, ALFRED C (AL); [email-masked]
> > The Section 4 example is about timing of events & schedules -
> pipelining /serialise etc - which is useful.
> >
> > I meant something like: take some variant of the example in RFC7594
> and explain how to use the information model
> > Strictly speaking this is not needed, but I think it would be useful,
> as the information model is now quite abstract (eg measurement tasks and
> registries no longer immediately obvious - as they're just tasks &
> registires). If other people think this is pointless, then I'm ok.
> >
> 
[ACM] 

Juergen,
You have described a Task registry on several occasions, but we have no
details in writing or examples in the Framework RFC because this registry
concept is new.

Now that the information model has been generalized to allow for a Task 
registry, I want to understand how you intend to use that registry
and how it interacts with the Metrics registry (whose role we have 
understood for years, but now potentially changes, I cannot tell how).

Phil and I have both asked for examples so that we can understand 
the implications of the changes you've executed. Examples have been
the stepping stones to reach greater clarity in much of our work in
LMAP over the last year, and I provided quite a few examples for the 
metrics registry and Cycle Number discussions.