Subject: [lmap] Defining Status and Logging (Capabilities and Failure)

From: <[email-masked]>
Date: Wed, 27 Nov 2013 15:11:55 +0000

I agree with Juergen that we should be more consistent between the framework & information model drafts.

Currently the framework has (from MA to Controller):
* ACKs of various msgs
* List of Measurement Methods (typically in response to a Capability request)
* Failure report 

Looking at Info model i-d:
* Status Information - includes List of Measurement Methods and other items like the interface speed and type and the link and IP address of the device with the MA (btw, I think the section could be clearer - hopefully I got it correct)
* Logging Information - notify successful configuration of eg a reporting channel; notify of a failure: the MA cannot action the Instruction (eg Task is missing a mandatory parameter); Task couldn't be executed (eg unexpectedly MA has no spare CPU); Collector not responding

The wording is a bit confusing, eg logging information is said to include status.

Firstly, I don't think the logging information should notify successful configuration of eg a reporting channel. The framework currently says:
>>   There is no need for the MA to confirm to the Controller that it has
   understood and acted on the Instruction, since the Controller knows
   the capabilities of the MA.  However, the Control Protocol must
   support robust error reporting by the MA, to provide the Controller
   with sufficiently detailed reasons for any failures.

Secondly, I suggest adding two definitions:

Capabilities Information: 
The list of the Measurement Methods that the MA can perform, plus information about the device hosting the MA (for example its interface type and speed and its IP address). 

Failure Information:
Information about the MA's failure to action or execute an Instruction, whether concerning Measurement Tasks or Reporting. 

Also the Control Protocol definition would need tweaking to:
Control Protocol: The protocol delivering Instruction(s) from a Controller to a Measurement Agent. It also delivers Failure Information and Capabilities information from the Measurement Agent to the Controller.

I prefer the term 'capabilities information' to 'status information' - personally I think 'status' suggests 'is the MA on or off?'.
Typically this msg is sent in response to a request from the Controller, but it might also be sent when the MA becomes capable of a new MA, the rate of the interface changes or the MA re-boots.

I prefer the term 'failure information' to 'status' - with the elimination of notifying successful configuration, it now only includes info about failures. I'd also be ok with 'error'.
This msg is (only) sent asynchronously by the MA. 

I'll also expand the framework's discussion of failures - at the moment it only mentions Measurement Tasks - Reporting can also fail.

Nit: in the framework i-d, currently the figure has "Failure report: (reason)" - which I think is a typo for "[reason]" - as the nomenclature is (optional) and [potentially repeated]

Comments?
Thanks
Phil

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Juergen Schoenwaelder
> Sent: 12 November 2013 11:20
> To: Burbridge,T,Trevor,TUB8 R
> Cc: [email-masked]; [email-masked]
> Subject: Re: [lmap] Splitting measurement tasks into measurement tasks
> plus reporting tasks?
> 
> On Tue, Nov 12, 2013 at 11:12:53AM +0000, [email-masked]
> wrote:
> >
> > In the information model the report channel delivers information to a
> specified URL. I don't see why that can't be the Controller. Or if we
> want we can talk about the Controller having a Collector interface we
> can do that as well. This is all terminology.
> >
> > As I said we need to tidy up some of the naming and definitions, but
> technically I don't see any problems - we designed it to work this way.
> >
> 
> For me, this is not at all clear. In fact, the latest info model also
> has a status channel and a logging channel. It is great if we can ship
> stuff anywhere - but I think we would be even better off if we have a
> clear view who should be the receiver / consumer of status and logging
> information. I think this is actually an architectural question to
> answer.
> 
> /js
> 
> --
> Juergen Schoenwaelder           Jacobs University Bremen gGmbH
> Phone: +49 421 200 3587         Campus Ring 1, 28759 Bremen, Germany
> Fax:   +49 421 200 3103         <http://www.jacobs-university.de/>
> _______________________________________________
> lmap mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/lmap

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Thu, 28 Nov 2013 17:47:57 +0100

Phil,

I agree with many things you wrote but I am still unclear about
'status' information. For me, capabilities are relatively static,
e.g., I do not expect that the list of measurement methods changes
highly dynamically. Other information like say the IP addresses of an
interface are what management people meanwhile call operational state
and this can change regularly (e.g., because DHCP leases timeout). So
lumping all this together into 'capabilities' may not be right.

That said, I personally believe the status information is right now
too detailed in the information model. For some of the relevant status
information we do have existing data models and I think we should
promote their reuse rather than defining new lists of objects in the
LMAP information model. And I think we need to focus on the
_essential_ information a _controller_ needs to generate proper
instructions.

For data collection, I can imagine to install a measurement test that
reports say the local interface configuration and the IP layer network
settings. This way, LMAP does not need a special mechanism to send
operational state for data collection purposes.

Perhaps we should change the category to "Capability and Status
Information" or we split things such that "Capability Information" is
really just capability information and "Status Information" is really
just the _essential_ operational state need by the controller to
generate Instructions.

/js

---

From: <[email-masked]>
Date: Fri, 29 Nov 2013 10:56:05 +0000

Capabilities vs operational info seems a reasonable split to me.

Yes, it's always possible to report info to the Collector and from there (via an out-of-scope mechanism) to the Controller. The example we gave was the results of one test leading to some follow-up tests. Maybe this type of interaction should be assumed where we have to analyse the results before deciding what Task to do next - if you're trying to diagnose a fault. 
Whereas if you're trying to decide how big a file to use for your download speed test, that depends on operational state (the interface speed)

The info model has lots of optional stuff concerning operational state. Who decides what's sent? Does the Controller's Request-for-operational-info include the list of things it wants to learn? Or not - and then does the MA reply with all Operational-info? Everything that's changed since it last sent? Any operational-info at its choice?
Do we allow extensibility? Ie can the operator of the measurement system include its own additional operational-info fields? If yes, how?

My guesses would be:
Request-for-operational-info(list) or (all)
Can define own operator /vendor specific fields

I also think it's worth adding to the info model i-d reasons why the Information Model divided up at all. I guess it's something like:

1. different information transferred between different elements, eg the Instruction information is transmitted from the Controller to the MA, the Reporting from the MA to the Collector, pre-configuration information ...
2. between the same functions, but at different timescales and at different times. Eg for the sub-parts of the Instruction, the Measurement Tasks & reporting channels would be updated infrequently; the Measurement Schedule quite often, and suppression ad hoc when needed...
3. possibly the protocol can be different /optimised for each case - or re-use of a data model, as you mention
4. anything else?

phil
> From: Juergen Schoenwaelder [mailto:j.schoenwaelder@jacobs-
> university.de]
> Sent: 28 November 2013 16:48
> To: Eardley,PL,Philip,TUB8 R
> Cc: Burbridge,T,Trevor,TUB8 R; [email-masked]; [email-masked]
> Subject: Re: Defining Status and Logging (Capabilities and Failure)
> 
> Phil,
> 
> 'status' information. For me, capabilities are relatively static, e.g.,
> I do not expect that the list of measurement methods changes highly
> dynamically. Other information like say the IP addresses of an
> That said, I personally believe the status information is right now too
> detailed in the information model. For some of the relevant status
> LMAP information model. And I think we need to focus on the _essential_
> information a _controller_ needs to generate proper instructions.
> 
> > * Status Information - includes List of Measurement Methods and other
> > items like the interface speed and type and the link and IP address
> of
> > the device with the MA (btw, I think the section could be clearer -
> > hopefully I got it correct)
> > * Logging Information - notify successful configuration of eg a
> > reporting channel; notify of a failure: the MA cannot action the
> > Instruction (eg Task is missing a mandatory parameter); Task couldn't
> > be executed (eg unexpectedly MA has no spare CPU); Collector not
> > responding
> >
> > Firstly, I don't think the logging information should notify
> successful configuration of eg a reporting channel. The framework
> currently says:
> > The list of the Measurement Methods that the MA can perform, plus
> information about the device hosting the MA (for example its interface
> type and speed and its IP address).
> >
> > Failure Information:
> > Information about the MA's failure to action or execute an
> Instruction, whether concerning Measurement Tasks or Reporting.
> >
> > Control Protocol: The protocol delivering Instruction(s) from a
> Controller to a Measurement Agent. It also delivers Failure Information
> and Capabilities information from the Measurement Agent to the
> Controller.
> >
> >
> > I prefer the term 'capabilities information' to 'status information'
> - personally I think 'status' suggests 'is the MA on or off?'.
> > Typically this msg is sent in response to a request from the
> Controller, but it might also be sent when the MA becomes capable of a
> new MA, the rate of the interface changes or the MA re-boots.
> >
> > I prefer the term 'failure information' to 'status' - with the
> elimination of notifying successful configuration, it now only includes
> info about failures. I'd also be ok with 'error'.
> > I'll also expand the framework's discussion of failures - at the
> moment it only mentions Measurement Tasks - Reporting can also fail.
> >
> > Nit: in the framework i-d, currently the figure has "Failure report:
> > (reason)" - which I think is a typo for "[reason]" - as the
> > nomenclature is (optional) and [potentially repeated]
> >
> > Comments?
> > Thanks
> > Phil
> >
> > > From: [email-masked] [mailto:[email-masked]] On
> Behalf
> > > Of Juergen Schoenwaelder
> > > Subject: Re: [lmap] Splitting measurement tasks into measurement
> > > tasks plus reporting tasks?
> > >
> > > specified URL. I don't see why that can't be the Controller. Or if
> > > we want we can talk about the Controller having a Collector
> > > interface we can do that as well. This is all terminology.
> > > >
> > > For me, this is not at all clear. In fact, the latest info model
> > > also has a status channel and a logging channel. It is great if we
> > > can ship stuff anywhere - but I think we would be even better off
> if
> > > we have a clear view who should be the receiver / consumer of
> status
> > > and logging information. I think this is actually an architectural
> > > question to answer.
> > >
> > > /js
> > >
> > > --

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Fri, 29 Nov 2013 13:01:19 +0100

> Whereas if you're trying to decide how big a file to use for your download speed test, that depends on operational state (the interface speed)

You can design this into the test itself by making the test adapt to
the interface speed and perhaps also at the statistical properties of
the download, e.g. you request a big file but abort the transfer once
you have sufficient data received. The more self-contained tests can
be, the less the controller needs to understand about every test,
which leads to a great simplification overall.
 
> Can define own operator /vendor specific fields

I think it is fair to assume that there is a mechanism that allows a
controller to select the operational state it is interested
in. Whether there is an "everything since last time" filter, I do not
know - this requires some state to be kept on the device. And yes, I
believe operational state needs to be extensible and hence I suggested
to actually reduce stuff we currently have in the information model so
that we do not spend lots of time discussing what operational state
might all be useful (since this depends to a large extend on the tests
you are running).

/js

---

From: Steven Miller <[email-masked]>
Date: Sat, 30 Nov 2013 17:22:00 -0500

FWIW I like the idea that feeding operational state back is just 
another measurement task.  The stuff in the current I-D does seem like a 
long list, though as I read it with an eye to things I thought could 
easily be thrown out, I found myself thinking that most or all of that 
is stuff I'd like to have available if I was trying to figure out after 
the fact why something had started performing poorly.  For example, 
knowing the gateway that was in use at a time more-or-less around when a 
given measurement started reporting poor results might let me 
cross-correlate that back to a device for which I have some other 
failure information, a la "oh, that was this router, yeah, there was a 
device-down alert for that at the same time, that explains it."

     -Steve