Subject: [lmap] my comments on draft-akhter-lmap-framework-00

From: "Romascanu, Dan (Dan)" <[email-masked]>
Date: Wed, 24 Jul 2013 11:28:15 +0000

Please find below a set of comments after my first reading of draft-akhter-lmap-framework-00. These are made from the perspective of contributor, and take into account the fact that this is a 00 individual submission, so idnits was not run, spelling and references were not checked, etc. 

1. The name of the I-D speaks about 'Inventory' which is a little bit confusing or mis-reading. Actually the authors rather seem to have described some applications and protocols that fit into the LMAP architecture, so it's rather an incomplete 'taxonomy'. See more when talking about the (lack of) requirements. 

2. The previous contributions on LMAP framework included in draft-eardley-lmap-framework were completely ignored, some material is duplicated (in content, not in text), but the eardley I-D is not even included in the list of references. As the two I-Ds (and other relevant contributions) will need to be consolidated into one single WG I-D pretty soon I will refer in the following to some of the similarities and some of the differences. 

3. The order of Sections 1.2 and 1.3 should be reversed. 

4. Figure 1 is almost identical to Figure 1 in draft-eardley with two differences. The first is that the entity named Measurement Peer in draft-eardley is called here Remote Measurement Test Target and seems to have a much richer functionality and variants as illustrated in Sections 3.2 and Figure 2. The option of one measurement agent communicating with multiple Remote Measurement Agents needs to bear a warning concerning scalability. 

5. The second significant difference in the two diagrams is the communication between the Controller and the Collector. Later in the draft this is described as optional and motivated by the need to optimize for Collector Test Configuration synchronization (so that each MA needs not report its configuration to controllers). This addition to the basic architecture needs to be carefully discussed by the WG. 

6. One crisp limitation that is being introduced by draft-eardley is about each MA being configured only by one Controller (motivated among other by the need of simplifying authentication of Controllers with the MA). This limitation is not mentioned in this I-D, at least not explicitly. Is this an editorial miss, or the authors take a different stand on this issue? 

7. Sections 3,4,5 are to some extent written with one set of protocols already in mind to answer LMAP architectural requirements (OWAMP and TWAMP for active monitoring, PSAMP for passive monitoring, IPFIX for reporting protocol, IPFIX mediators to solve the scalability issues). Although this may be a reasonable set of solutions, it seems a little bit too early and too detailed at this phase. Missing here are the basic requirements for the Control protocol, Report protocol, and associated data models. I would prefer to see this being written first. 

8. The Security Consideration section needs to include not only information about authentication, but also about the risks of modification of information (data integrity).

Thanks and Regards,

Dan
(as contributor)

---

From: <[email-masked]>
Date: Thu, 25 Jul 2013 14:52:12 +0100

> 5. The second significant difference in the two diagrams is the
> communication between the Controller and the Collector. Later in the
> draft this is described as optional and motivated by the need to
> optimize for Collector Test Configuration synchronization (so that each
> MA needs not report its configuration to controllers). This addition to
> the basic architecture needs to be carefully discussed by the WG.

Expanding on this point.

There are ways we can reduce the overhead.

Firstly, a Measurement Agent will send an occasional report to its Collector (say once a day). So this report may include lots of 'repeat measurements' - the same Measurement Method made with the same parameters, every 10 minutes (say). When these results are reported, you only need to list the Measurement Method and parameters once - plus the individual results and their times. So the overhead isn't huge.
This makes an assumption about encoding of information in the report (eg json could do this). If I remember, this issue crops up in the http protocol draft.   

Secondly, if this isn't enough, the measurement system could have a registry - so the report would just include a reference to a registry entry.  This is somewhat similar to your idea of a controller-collector interface. Personally I prefer the former (as the latter needs to worry about syncronisation). But, more important, I don't think the wg should think about either idea in the initial work.

Best wishes
phil

---

From: Paul Aitken <[email-masked]>
Date: Thu, 25 Jul 2013 15:07:54 +0100

Philip,

> Firstly, a Measurement Agent will send an occasional report to its Collector (say once a day). So this report may include lots of 'repeat measurements' - the same Measurement Method made with the s…

Isn't huge wrt a single MA. However, if 1,000's of MAs have been 
commanded to perform the same test, the impact on the collector is much 
higher: it has to examine all the methods and parameters to determine 
that they all performed the same test.

> Secondly, if this isn't enough, the measurement system could have a registry - so the report would just include a reference to a registry entry.  This is somewhat similar to your idea of a controller-collector interface.

What we wrote in draft-aakhter-lmap framework: as an optimisation, the 
controller treats the collector as an additional MA, thereby informing 
the collector once of the method and params to be used, presumably using 
some ID. Then the MA's need only report that same ID, so the collector 
overhead is much reduced.

P.

>   Personally I prefer the former (as the latter needs to worry about syncronisation). But, more important, I don't think the wg should think about either idea in the initial work.
>
> Best wishes
> phil
> _______________________________________________
> lmap mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/lmap

---

From: "MORTON JR., ALFRED C (AL)" <[email-masked]>
Date: Thu, 25 Jul 2013 10:24:06 -0400

Paul,

> overhead is much reduced.

how does this optimization match the flexibility to use different measurement 
methods on the fly, as Phil highlighted:
<< One proposal for making use of both active and passive measurement is
   to allow the Measurement Agent to make local decisions on which
   technique to use to deliver a particular metric-- as long as the
   specific method is included in the report.>>

I think we agree that passive and active methods are sufficiently different
that they require categorization, possibly as far as summarizing 
results from each method separately. MA local decisions certainly 
require indicating the measurement method with each value, 
hopefully through some shorthand like the proposed registry.

It seems we can have either the optimizations with static test Instructions,
or MA's making autonomous decisions.

Al

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Paul Aitken
> Sent: Thursday, July 25, 2013 10:08 AM
> To: [email-masked]
> Cc: [email-masked]; [email-masked]
> Subject: Re: [lmap] my comments on draft-akhter-lmap-framework-00
> 
> Philip,
> 
> > Firstly, a Measurement Agent will send an occasional report to its
> Collector (say once a day). So this report may include lots of 'repeat
> measurements' - the same Measurement Method made with the same parameters,
> every 10 minutes (say). When these results are reported, you only need to
> list the Measurement Method and parameters once - plus the individual
> results and their times. So the overhead isn't huge.
> 
> > This makes an assumption about encoding of information in the report (eg
> json could do this). If I remember, this issue crops up in the http
> protocol draft.
> >
> > Secondly, if this isn't enough, the measurement system could have a
> registry - so the report would just include a reference to a registry
> entry.  This is somewhat similar to your idea of a controller-collector
> interface.
> 
> >   Personally I prefer the former (as the latter needs to worry about
> syncronisation). But, more important, I don't think the wg should think
> about either idea in the initial work.
> >
> > Best wishes
> > phil

---

From: Paul Aitken <[email-masked]>
Date: Thu, 25 Jul 2013 21:49:15 +0100

Al,

>     specific method is included in the report.>>

If it's necessary for the MA to report the measurement techniques used, 
then these should be reported in the test results.

It's like another test primitive: select an appropriate technique, and 
include it in the test report.

> or MA's making autonomous decisions.

So, the proposal is that each MA reports: { which tests were run, which 
test parameters were used, what the test results were }.

The selected technique could be reported as a test parameter, or as a 
test result (ie, as if the technique selection was a test in its own right).

If the selected technique is reported as a test parameter, then there 
may be many variations in the { tests, parameters, results } tuple which 
reduce optimisation opportunities since the only part which is certainly 
constant is the { tests }.

Whereas, if the technique is reported as a test result in its own right 
then the { tests, parameters } part of the tuple is a constant across 
all the MAs, which provides a great optimisation opportunity: each MA 
does not have to report this information over and over again, and the 
collector does not have to store N unique instances of this tuple, just one.

So I propose that the test parameters reports what the MA was commanded 
to do by the controller, rather than MA-specific details.

eg, what I'm proposing looks like:

     Controller to MA:

         id = 1239
         test = perform a DNS lookup
         param = lookup "test.net"
         param = use whatever DNS server you think is best; include it 
in the report.
         param = report the lookup time

     MA to collector:

         result = 1239: 5ms from dns.near.me

Since the (test+params) is constant, each MA need only report the ID and 
results.

The collector understands the ID-to-(tests+params) mapping because a 
copy of the original controller message is also sent to the collector as 
well as to each MA in the test.

Therefore the amount of duplication is easily reduced, while still 
allowing each MA flexibility to choose techniques and report local settings.

P.

---

From: Paul Aitken <[email-masked]>
Date: Thu, 25 Jul 2013 22:24:14 +0100

Dan, Thanks for your feedback. Please find some replies inline:

> 1. The name of the I-D speaks about 'Inventory' which is a little bit confusing or mis-reading. Actually the authors rather seem to have described some applications and protocols that fit into the…

We started by making an inventory of the existing/missing IETF blocks, 
which evolved towards a framework.

> 2. The previous contributions on LMAP framework included in draft-eardley-lmap-framework were completely ignored, some material is duplicated (in content, not in text), but the eardley I-D is not e…

Sorry for not including draft-eardley-lmap-framework. We were debating 
whether to include it or not; we're happy to add it, though we'll be 
working with the other authors to merge the drafts anyway.

> 3. The order of Sections 1.2 and 1.3 should be reversed.

OK.

> 4. Figure 1 is almost identical to Figure 1 in draft-eardley with two differences. The first is that the entity named Measurement Peer in draft-eardley is called here Remote Measurement Test Target…

Yes, we agree about scalability. We'll add some use cases.

> 5. The second significant difference in the two diagrams is the communication between the Controller and the Collector. Later in the draft this is described as optional and motivated by the need to…

Controller to Collector communication is optional; we see this as an 
optimisation for large-scale tests. it could be up to the controller 
based on the number of MAs involved.
If the majority of the tests are single MAs, then this optimization 
works against. However, if there are more than 2 or 3 MAs, the 
optimisation works. We can discuss.

> 6. One crisp limitation that is being introduced by draft-eardley is about each MA being configured only by one Controller (motivated among other by the need of simplifying authentication of Contro…

Multiple controllers are generally only for the case of redundancy 
failover; usually an MA has a relationship with only one controller at 
any time.
However, since all controllers are part of the same administrative 
domain which is centrally controlled, each can be understood as a 
different controller instance.
So an MA with multiple controllers is simply communicating with multiple 
controller instances - so allowing multiple controllers should not be 
problematic, since they should not issue contradictory instructions.

Multiple distinct controllers under different administrative control 
aren't within charter.

> 7. Sections 3,4,5 are to some extent written with one set of protocols already in mind to answer LMAP architectural requirements (OWAMP and TWAMP for active monitoring, PSAMP for passive monitoring…

We started by talking about firewalls, NAT etc. We can add the basic 
requirements; there's already an analysis of OWAMP and TWAMP. We'll 
separate this out so there's clear separation between the requirements 
of those protocols and the analysis of the individual protocols.

> 8. The Security Consideration section needs to include not only information about authentication, but also about the risks of modification of information (data integrity).

We'll note that it's insecure in this particular way. Note that the 
charter says that gaming is outside the scope.

Aamer + Paul.

---

From: <[email-masked]>
Date: Fri, 26 Jul 2013 10:20:11 +0100

There'd be some complexities to handle, for instance when the test details change to get the results synchronised with the information you're sending from the controller. 
Anyway, I like the idea of keeping the reporting stateless. 

I'm not convinced the gain in de-duplication is worthwhile (is the overhead really that big?), it would be good to have some numbers here on typical bytes of a report.

> From: Paul Aitken [mailto:[email-masked]]
> Sent: 25 July 2013 21:49
> To: MORTON JR., ALFRED C (AL)
> Cc: Eardley,PL,Philip,TUB8 R; [email-masked]; [email-masked]
> >> What we wrote in draft-aakhter-lmap framework: as an optimisation,
> >> the controller treats the collector as an additional MA, thereby
> >> informing the collector once of the method and params to be used,
> >> presumably using some ID. Then the MA's need only report that same
> >> ID, so the collector overhead is much reduced.
> > how does this optimization match the flexibility to use different
> > measurement methods on the fly, as Phil highlighted:

---

From: Sam Crawford <[email-masked]>
Date: Fri, 26 Jul 2013 10:47:33 +0100

I also very much like the idea of keeping reporting stateless. It removes
the need for the client to know about the internal state of the collector
when reporting data. It's how we do it at the moment, and its simplicity
has been very beneficial a few times.

Regarding overhead: To give you a very rough idea, each run of one of our
metrics on our probes generates about 70-100 bytes of report data. Around
30-50% of that details the input parameters. These numbers vary by metric
of course, and are without any compression or de-duplication. It's a small
amount and the added overhead has never caused us concern.

Thanks,

Sam

On 26 July 2013 10:20, <[email-masked]> wrote:

> There'd be some complexities to handle, for instance when the test details
> change to get the results synchronised with the information you're sending
> from the controller.
> I'm not convinced the gain in de-duplication is worthwhile (is the
> overhead really that big?), it would be good to have some numbers here on
> typical bytes of a report.
>
>
>

-- 
----
Sam Crawford
SamKnows Limited

Switchboard: +44 (0) 20 3111 4330
DDI: +44 (0) 20 3111 4332

This email is sent for and on behalf of SamKnows Limited.

This email and any attachments are confidential, legally privileged and
protected by copyright. If you are not the intended recipient dissemination
or copying of this email is prohibited. If you have received this in error,
please notify the sender by replying by email and then delete the email
completely from your system.

SamKnows Limited, Registered Number: 06510477, Registered Office: Hill
House, 1 Little New Street, London, EC4A 3TR.  Registered in England and
Wales. Trade Mark 2507103

---

From: Brian Trammell <[email-masked]>
Date: Fri, 26 Jul 2013 13:51:30 +0200

Hi, all,

Trying to catch up with these threads before Tuesday morning; but wanted to chime in here:

+1 to stateless reporting; I'd go so far as to say that control should be "as stateless as possible" -- it should be possible to determine whether a task completed at an MA or not, but this doesn't necessarily need to be handled as an error condition at the control level, especially in the case that the MA is on a device where the customer controls the power and/or whether the app is running: it's simply something that the collector needs to know about to support analysis of the results. MAs may "lose sync" for any number of reasons, so collectors and controllers will also have to be robust against MAs that are running out-of-date instructions; this is also not necessarily an error condition.

More later, best regards,

Brian

On 26 Jul 2013, at 11:47, Sam Crawford <[email-masked]> wrote:

> I also very much like the idea of keeping reporting stateless. It removes the need for the client to know about the internal state of the collector when reporting data. It's how we do it at the moment, and its simplicity has been very beneficial a few times.
> 
> Regarding overhead: To give you a very rough idea, each run of one of our metrics on our probes generates about 70-100 bytes of report data. Around 30-50% of that details the input parameters. These numbers vary by metric of course, and are without any compression or de-duplication. It's a small amount and the added overhead has never caused us concern.
> 
> Thanks,
> 
> Sam
> 
> 
> 
> This email and any attachments are confidential, legally privileged and protected by copyright. If you are not the intended recipient dissemination or copying of this email is prohibited. If you have received this in error, please notify the sender by replying by email and then delete the email completely from your system. 
> 
> SamKnows Limited, Registered Number: 06510477, Registered Office: Hill House, 1 Little New Street, London, EC4A 3TR.  Registered in England and Wales. Trade Mark 2507103

---

From: "MORTON JR., ALFRED C (AL)" <[email-masked]>
Date: Fri, 26 Jul 2013 08:43:59 -0400

If the collector has state that is essential to interpreting the results,
then the state must be appended when moving the results to the analysis phase
(and the information compression is lost)
or the analysis system must also have the same state, if I've understood the proposal.

If we compress the information using IDs or handles with definitions that are widely accessible,
then every entity that gets the data can use and expand the results with confidence,
even those outside the LMAP system functions. We had some related discussion of handles
several months back:  http://www.ietf.org/mail-archive/web/lmap/current/msg00266.html

so, leaning toward stateless,
Al

> From: Brian Trammell [mailto:[email-masked]]
> Sent: Friday, July 26, 2013 7:52 AM
> To: Sam Crawford
> Cc: EARDLEY, Phil; [email-masked]; [email-masked]; [email-masked];
> MORTON JR., ALFRED C (AL)
> +1 to stateless reporting; I'd go so far as to say that control should be
> "as stateless as possible" -- it should be possible to determine whether a
> task completed at an MA or not, but this doesn't necessarily need to be
> handled as an error condition at the control level, especially in the case
> that the MA is on a device where the customer controls the power and/or
> whether the app is running: it's simply something that the collector needs
> to know about to support analysis of the results. MAs may "lose sync" for
> any number of reasons, so collectors and controllers will also have to be
> robust against MAs that are running out-of-date instructions; this is also
> not necessarily an error condition.
> 
> > I also very much like the idea of keeping reporting stateless. It
> removes the need for the client to know about the internal state of the
> collector when reporting data. It's how we do it at the moment, and its
> simplicity has been very beneficial a few times.
> >
> > Regarding overhead: To give you a very rough idea, each run of one of
> our metrics on our probes generates about 70-100 bytes of report data.
> Around 30-50% of that details the input parameters. These numbers vary by
> metric of course, and are without any compression or de-duplication. It's
> a small amount and the added overhead has never caused us concern.
> >
> > Thanks,
> >
> > Sam
> >
> >
> >
> > There'd be some complexities to handle, for instance when the test
> details change to get the results synchronised with the information you're
> sending from the controller.
> > > > << One proposal for making use of both active and passive
> measurement
> > > is
> > > collector does not have to store N unique instances of this tuple,
> just
> > > one.
> > >
> protected by copyright. If you are not the intended recipient
> dissemination or copying of this email is prohibited. If you have received
> this in error, please notify the sender by replying by email and then
> delete the email completely from your system.
> >

---

From: Paul Aitken <[email-masked]>
Date: Fri, 26 Jul 2013 14:02:52 +0100

Brian,

> +1 to stateless reporting; I'd go so far as to say that control should be "as stateless as possible" -- it should be possible to determine whether a task completed at an MA or not, but this doesn't necessarily need to be handled as an error condition at the control level

Aamer and I were discussing this yesterday:

1. if a controller commands an MA to perform a test which it cannot do, 
then the MA could immediately indicate this (eg, by some NACK in the 
control protocol) so the controller knows and is enabled to take 
appropriate action, whatever that may be (report error, revise the test, 
command another test, ...; it's out of scope).

2. if the MA accepts the command but is later unable to perform the test 
as requested (eg, lack of resource, insufficient traffic, backed off for 
too long, conflict), then there's no longer any back-channel to report 
this to the controller. All the MA can do is indicate the failure to the 
collector.

3. If the MA is unable to perform the test (or at least, is unable to 
report the results) then the controller receives nothing and doesn't 
even know that the MA had been requested to perform some test, but 
failed to do so.

If we don't close this loop then the controller will be blind to the 
fact that the requested test didn't happen.

So how do we close the loop? Does the collector inform the controller, 
or does the controller check on the collector to ensure that the results 
arrived?

P.

> , especially in the case that the MA is on a device where the customer controls the power and/or whether the app is running: it's simply something that the collector needs to know about to support analysis of the results. MAs may "lose sync" for any number of reasons, so collectors and controllers will also have to be robust against MAs that are running out-of-date instructions; this is also not necessarily an error condition.
>

---

From: <[email-masked]>
Date: Fri, 26 Jul 2013 14:16:07 +0100

> > 6. One crisp limitation that is being introduced by draft-eardley is
> about each MA being configured only by one Controller (motivated among
> other by the need of simplifying authentication of Controllers with the
> MA). This limitation is not mentioned in this I-D, at least not
> explicitly. Is this an editorial miss, or the authors take a different
> stand on this issue?
> 

The constraint proposed in eardley-lmap-framework is that a MA gets instructions from one Controller at any point in time.
If the operator of the measurement chooses to implement the Controller function in a distributed way then it can. But I think this should be left as internals to how the Controller is implemented - ie we shouldn't worry about it when designing the Control Protocol. 

At some point we do have to think about what happens if the Controller fails, and to what extent the WG solves the issue. I don't think that point is now. 

Best wishes
phil

---

From: Brian Trammell <[email-masked]>
Date: Fri, 26 Jul 2013 15:24:59 +0200

hi Paul, all,

comments and questions inline,

On 26 Jul 2013, at 15:02, Paul Aitken <[email-masked]> wrote:

> 1. if a controller commands an MA to perform a test which it cannot do, then the MA could immediately indicate this (eg, by some NACK in the control protocol) so the controller knows and is enabled to take appropriate action, whatever that may be (report error, revise the test, command another test, ...; it's out of scope).

Yep, but that's a different case than out-of-sync. Again, depending on the measurement being run, this may not be a real problem, and how this gets presented to the controller administrator is situation dependent, but it makes sense to tell the controller this in the control protocol.

> 2. if the MA accepts the command but is later unable to perform the test as requested (eg, lack of resource, insufficient traffic, backed off for too long, conflict), then there's no longer any back-channel to report this to the controller. All the MA can do is indicate the failure to the collector.
> 
> 3. If the MA is unable to perform the test (or at least, is unable to report the results) then the controller receives nothing and doesn't even know that the MA had been requested to perform some test, but failed to do so.
> 
> If we don't close this loop then the controller will be blind to the fact that the requested test didn't happen.

Is this a problem? Why does the controller need to know about measurement failures in cases 2 or 3? If an MA is still up and knows that it won't run a test, it can certainly send a no-report message to the collector (so it knows there was an instruction sent that was not executed) so that no-report can be factored into post-facto analysis. But once the controller sends an instruction, why is it involved in the further fate of the measurement task?

> So how do we close the loop? Does the collector inform the controller, or does the controller check on the collector to ensure that the results arrived?
> 
> 
> P.
>

---

From: Paul Aitken <[email-masked]>
Date: Fri, 26 Jul 2013 15:27:16 +0100

Brian,

> Is this a problem? Why does the controller need to know about measurement failures in cases 2 or 3?

I'm thinking of it like ICMP messages: if nobody tells the controller 
that the requested test didn't work, then it might continue to request 
the same. The administrator might go to the collector looking to analyse 
the results and be surprised not to find any.

> If an MA is still up and knows that it won't run a test, it can certainly send a no-report message to the collector (so it knows there was an instruction sent that was not executed) so that no-report can be factored into post-facto analysis.

Right, that's case 2.

What do you propose for case 3?

> But once the controller sends an instruction, why is it involved in the further fate of the measurement task?

Not so much in the fate of that specific task per se, but of the overall 
measurement. eg, depending on the failure reason, it could be that the 
test should be modified or a different test run. Or perhaps an action 
raised to upgrade the failing set of MAs.

Suppose Benoit tells me to write a report and send it to you. Normally I 
say "OK" and do it. But I might say "No", in which case we negotiate a 
different report or timeframe. Or I might say "OK" then just not do it, 
so you receive nothing. You don't even know that you didn't receive the 
report, because nobody told you to expect it - so when Benoit comes to 
discuss it with you, you're clueless. Not good.

So normally Benoit would tell you, "I've asked Paul to send you the 
report" so you'd be expecting it. Then you can check back with either of 
us if you didn't receive it.

Or, Benoit would periodically check with us to see whether I finished 
the report and whether you received it.

LMAP should work like this too.

P.

---

From: Paul Aitken <[email-masked]>
Date: Fri, 26 Jul 2013 15:38:29 +0100

Philip,

> The constraint proposed in eardley-lmap-framework is that a MA gets instructions from one Controller at any point in time.

While I know what you mean, if it's a single CPU (or at least, a single 
MA process) then it's necessarily only getting instructions from one 
Controller at a time - although there could be different controllers 
every few milliseconds...

Why do we fear multiple controllers? By the LMAP charter, they're under 
control of a single organisation, so they should not be issuing 
contradictory instructions.

> At some point we do have to think about what happens if the Controller fails, and to what extent the WG solves the issue. I don't think that point is now.

If it's not mentioned early on, then either the accepted design will 
prevent it from being added later, or it'll later be seen as a new 
requirement and rejected. Surely we've all seen this many times?

P.

---

From: <[email-masked]>
Date: Fri, 26 Jul 2013 15:43:34 +0100

My couple of contributions inline...

>Sent: 26 July 2013 15:27
>To: Brian Trammell
>Cc: [email-masked]; Eardley,PL,Philip,TUB8 R; acmorton; Sam
>Crawford; [email-masked]
>>>> +1 to stateless reporting; I'd go so far as to say that control
>>>> +should be "as stateless as possible" -- it should be possible to
>>>> +determine whether a task completed at an MA or not, but this
>>>> +doesn't necessarily need to be handled as an error condition at the
>>>> +control level
>control protocol) so the controller knows and is enabled to take appropriate
>action, whatever that may be (report error, revise the test, command
>another test, ...; it's out of scope).
>> Yep, but that's a different case than out-of-sync. Again, depending on the
>measurement being run, this may not be a real problem, and how this gets
>presented to the controller administrator is situation dependent, but it
>makes sense to tell the controller this in the control protocol.
>>
>as requested (eg, lack of resource, insufficient traffic, backed off for too long,
>conflict), then there's no longer any back-channel to report this to the
>controller. All the MA can do is indicate the failure to the collector.
>>>
>>> 3. If the MA is unable to perform the test (or at least, is unable to report
>the results) then the controller receives nothing and doesn't even know that
>the MA had been requested to perform some test, but failed to do so.
>>>
>I'm thinking of it like ICMP messages: if nobody tells the controller that the
>requested test didn't work, then it might continue to request the same. The
>administrator might go to the collector looking to analyse the results and be
>surprised not to find any.

If nobody tells the controller, then it definitely WON'T keep going to the MA as it will assume the test is running fine. In the information model we've talked about reporting back to the controller but imagined that most of what we want to do could be done asynchronously via some sort of log messages so we can easily use HTTP Rest type approaches.

The measurement itself can decide how to report  failed tests. This could simply be a failed flag, or more sophisticated data columns. This has to be left to the test since failure modes for different tests and how that should be reported will be different.

Someone going to the data should see test failure messages.

>> If an MA is still up and knows that it won't run a test, it can certainly send
>a no-report message to the collector (so it knows there was an instruction
>sent that was not executed) so that no-report can be factored into post-facto
>analysis.
>
>Right, that's case 2.
>
>measurement. eg, depending on the failure reason, it could be that the test
>should be modified or a different test run. Or perhaps an action raised to
>upgrade the failing set of MAs.
>
>
>Suppose Benoit tells me to write a report and send it to you. Normally I say
>"OK" and do it. But I might say "No", in which case we negotiate a different
>report or timeframe. Or I might say "OK" then just not do it, so you receive
>nothing. You don't even know that you didn't receive the report, because
>nobody told you to expect it - so when Benoit comes to discuss it with you,
>you're clueless. Not good.
>
>So normally Benoit would tell you, "I've asked Paul to send you the report"
>so you'd be expecting it. Then you can check back with either of us if you
>didn't receive it.
>
>Or, Benoit would periodically check with us to see whether I finished the
>report and whether you received it.
>

It's a big database with lots of data. I don't see the Collector should be trying to police the data it is receiving. It would have to know all of the schedules of different MAs and implement timeouts etc. We want to keep the Collector simple. Another motivation is if the Collector is a 3rd party who we simply want to ship data to.

---

From: Paul Aitken <[email-masked]>
Date: Fri, 26 Jul 2013 15:52:54 +0100

Trevor,

> The measurement itself can decide how to report  failed tests. This could simply be a failed flag, or more sophisticated data columns. This has to be left to the test since failure modes for differ…

Oh please, no.

This was an oversight in IPFIX: there's no official method of reporting 
that certain metrics weren't seen or couldn't be measured. So now there 
are a variety of field-specific values and implementation-specific 
methods such that collectors need a lot of specific knowledge. There 
should be a single standard mechanism which everyone uses. I have a 
draft with some proposals.

Similarly for LMAP: if the collector is to be kept simple, then there 
should be a common method of reporting test failure.

> Someone going to the data should see test failure messages.

Unless case (3) where the MA isn't able to send the test report. eg, the 
report message is lost or the MA loses power.

> It's a big database with lots of data. I don't see the Collector should be trying to police the data it is receiving. It would have to know all of the schedules of different MAs and implement timeo…

The shipped data doesn't arrive. Who's responsible for a) detecting, and 
b) addressing that?

P.

---

From: <[email-masked]>
Date: Fri, 26 Jul 2013 16:23:29 +0100

>Sent: 26 July 2013 15:53
>To: Burbridge,T,Trevor,TUB8 R
>Cc: [email-masked]; [email-masked]; Eardley,PL,Philip,TUB8
>R; [email-masked]; [email-masked]; [email-masked]
>>> From: [email-masked] [mailto:[email-masked]] On Behalf
>>> Of Paul Aitken
>>> Sent: 26 July 2013 15:27
>>> To: Brian Trammell
>>>>>> +doesn't necessarily need to be handled as an error condition at
>>>>>> +the control level
>>> appropriate action, whatever that may be (report error, revise the
>>> test, command another test, ...; it's out of scope).
>>> measurement being run, this may not be a real problem, and how this
>>> gets presented to the controller administrator is situation
>>> dependent, but it makes sense to tell the controller this in the control
>protocol.
>>> as requested (eg, lack of resource, insufficient traffic, backed off
>>> for too long, conflict), then there's no longer any back-channel to
>>> report this to the controller. All the MA can do is indicate the failure to
>the collector.
>>> the results) then the controller receives nothing and doesn't even
>>> know that the MA had been requested to perform some test, but failed
>to do so.
>>>>> If we don't close this loop then the controller will be blind to
>>>>> the fact that
>>> the requested test didn't happen.
>>> that the requested test didn't work, then it might continue to
>>> request the same. The administrator might go to the collector looking
>>> to analyse the results and be surprised not to find any.
>>
>> If nobody tells the controller, then it definitely WON'T keep going to the
>MA as it will assume the test is running fine. In the information model we've
>talked about reporting back to the controller but imagined that most of
>what we want to do could be done asynchronously via some sort of log
>messages so we can easily use HTTP Rest type approaches.
>>
>> The measurement itself can decide how to report  failed tests. This could
>simply be a failed flag, or more sophisticated data columns. This has to be
>left to the test since failure modes for different tests and how that should be
>reported will be different.
>
>Oh please, no.
>
>This was an oversight in IPFIX: there's no official method of reporting that
>certain metrics weren't seen or couldn't be measured. So now there are a
>variety of field-specific values and implementation-specific methods such
>that collectors need a lot of specific knowledge. There should be a single
>standard mechanism which everyone uses. I have a draft with some
>proposals.
>
>Similarly for LMAP: if the collector is to be kept simple, then there should be
>a common method of reporting test failure.

OK, so we mandate a success/fail column in the measurement reporting information model. I think the only think I have mandated at the moment is a timestamp.

>test
>say
>report"
>> It's a big database with lots of data. I don't see the Collector should be
>trying to police the data it is receiving. It would have to know all of the
>schedules of different MAs and implement timeouts etc. We want to keep
>the Collector simple. Another motivation is if the Collector is a 3rd party who
>we simply want to ship data to.
>

It's shipped using HTTP for example. The MA can have a retry/failover strategy.

Trevor.

---

From: <[email-masked]>
Date: Fri, 26 Jul 2013 16:52:26 +0100

> Sent: 26 July 2013 15:38
> To: Eardley,PL,Philip,TUB8 R
> > The constraint proposed in eardley-lmap-framework is that a MA gets
> instructions from one Controller at any point in time.
> 

I think there's less chance of going wrong 
It's more obvious what to do in nasty cases like juergen's examples
It keeps the MA simpler - easier for the MA to handle if it does (reply to instruction with reject code)
I don't see that multiple controllers gains you anything.

> 
> 
> > If the operator of the measurement chooses to implement the
> Controller function in a distributed way then it can. But I think this
> should be left as internals to how the Controller is implemented - ie
> we shouldn't worry about it when designing the Control Protocol.
> >
> > At some point we do have to think about what happens if the
> Controller fails, and to what extent the WG solves the issue. I don't
> think that point is now.
> 

As trevor points out to me, the information model i-d already provides resilience. The bootstrap process informs the MA about its Controller, this information is a FQDN, therefore we get resilience from dns. Similarly for collector. Not sure we need to do anything more complicated. Maybe something like, if MA finishes all its instruction, then check in with controller??