Subject: Re: [Roll] knowing which multiple metrics matter: MRHOF related questions

From: Ralph Droms <[email-masked]>
Date: Fri, 1 Jun 2012 15:47:16 -0400

On May 25, 2012, at 12:31 PM 5/25/12, Pascal Thubert (pthubert) wrote:

> Hi Michael:
> 
> I think RPL does not want to take party there. The OF is a piece of logic to tie metrics and policies together. 
> As such, there could be multiple metrics as long as there is good logic to tie them in. for instance one would look at optimizing metric A within contraints as expressed by metric B and the OF model will allow that.
> 
> OTOH, it a flows requires a certain optimization (say per one metric) and another requires something different, then certainly you want two instances.
> 
> So ... it depends!
> 
> Pascal
> 
> 
> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of Michael Richardson
> Sent: vendredi 25 mai 2012 16:54
> To: [email-masked]
> Subject: [Roll] knowing which multiple metrics matter: MRHOF related questions
> 
> 
> Ralph asked some questions a few days ago.
> His originally DISCUSS is at:
>    http://datatracker.ietf.org/doc/draft-ietf-roll-minrank-hysteresis-of/ballot/
> 
> This was my reply.    I am particularly interested in replies from
> Pascal, Anders and Mukul about my assertion about how we would never pick RPL instances by metrics; that they would in fact be seperate RPL Instance numbers and DODAG values, and that these things would provisioned by the network installer.
> 
> ====
> 
> I'm going to reply to your comments in a different order than you asked them, because I think question #3 is most important, and the rest fall out of it.
> 
>>>>>> "Ralph" == Ralph Droms <[email-masked]> writes:
>    Ralph> 3. Based on (1) and (2), would configuration and selection
>    Ralph> issues be ameliorated if the five candidate selected metrics
>    Ralph> were each asssigned a separate objective function?  Use of a
>    Ralph> separate OF code point for each of the five possible selected
>    Ralph> metrics would allow multiple RPL instances.
> 
> I think that it's important to understand that ROLL has a whole palette of things that need to be provisioned by the "network operator".
> 
> In contrast to the situation of ISPs and customers, where the ISP is the network operator, ROLL networks are more like highly orchestrated
> Enterprises: "all your host belong to us"
> 
> so, when we write something like:
>    "The metric chosen by the network operator to use for path
>    selection."
> in section 2, we really mean:
>    "The metric chosen by the network operator and provisioned into
>    the node when the firmware was flashed to use for path selection."

OK, so I get that the model is that MRHOF is a toolbox, where the specific tools are chosen when the device is flashed.  In that case, I think some additional review might be needed to ensure that the MRHOF spec is internally consistent with that point of view.

As one example, I think the "selected metric" should be called out explicitly somewhere in section 5 and included in the list of configured parameters in section 6.1.

As another example, I read this text from section 3:

   Rank is undefined for these node/link metrics: node state and
   attributes, throughput, and link color.  If the rank is undefined,
   the node must join one of the neighbors as a RPL Leaf node according
   to [RFC6550].

to mean that if the selected metric is one of the metrics for which rank is undefined, the node joins as a lead node.  But, how can that happen if the selected metric is chosen by the network operator?

> Ralph> 1.  Why is one objective function defined for several potential 
> Ralph> metrics?  The details of MRHOF seem to preclude the establishment 
> Ralph> of several RPL instances in an LLN, each of which uses MRHOF with 
> Ralph> a different selected metric.
> 
> If one had many different RPL Instances, then we would have different
> RPL Instance numbers in the RPL header.   There can be many different
> DODAG ("destinations") created within that instance.  The instances share a common set of (provisioned) parameters.

How does a node know which RPL Instance number maps to which selected metric?

One way would be to specify that a node advertise ONLY the selected metric for the RPL Instance in a DIO.

> (To put it into DHCP terms: if we have multiple DHCP servers on a link,  then one would expect them to all offer IP addresses in the same subnet.
> If one wanted to have addresses in different subnets, and let the host  pick between them, then, one would need different layer-2s: different  VLANs or ESSIDs, or... )
> 
> If you feel that RPL is rather schizo about provisioning vs configuration, then I agree.  It's not always clear to me why some things are advertised while others are provisioned.
> 
> In BGPv4, we calculate metrics by adding AS entries in the path.
> (It's always additive), and we add at least one AS entry to the path.
> One can AS-stuff and add more, but proper operation of BGP does not depend upon the exact algorithm used.
> 
> Finally, my impression is that how the various metrics are used (singly, or in some combination) to calculate Rank Increase is a question of further research, experimentation, and trade secret.

Well, OK, but for the purposes of the MRHOF spec, it seems pretty clear to me there is one selected metric that is used across all nodes as a strictly additive path cost computation.

> So long as the Rank increases, and a node does not flap between parents, the exact details do not matter.  Each node can do it's parent selection based upon the available metrics on it's own.  It advertises the metrics it has.
> 
> I hope the authors will correct me if I'm completely off here.

I read the spec to require a single selected metric across the entire RPL Instance.

- Ralph

> 
> 
> 
> 
> 
> _______________________________________________
> Roll mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/roll

---

From: Omprakash Gnawali <[email-masked]>
Date: Sat, 2 Jun 2012 03:45:10 -0500

My understanding is the metric to be used by MRHOF is to be carried in
the DAG metric container in DIOs. If an instance is using a given
metric, that metric will be in the metric container.

- om_p

---

From: JP Vasseur <[email-masked]>
Date: Sat, 2 Jun 2012 14:35:24 +0200

Agree with Pascal - for information there was a discussion about 2 years on the subject matter and there was a consensus
not to try to either mix metrics/constraints (which IMO is the right thing to do even if it leads to sub-optimality or disconnected
areas) or try to make use of default metrics.

On May 28, 2012, at 12:04 PM, Pascal Thubert (pthubert) wrote:

> Hello Micheal
> 
>>>>>> "Pascal" == Pascal Thubert <(pthubert)" <[email-masked]>> writes:
>    Pascal> I think RPL does not want to take party there. The OF is a
>    Pascal> piece of logic to tie metrics and policies together.  
> 
> My question is:
> 
> - do the nodes of a DODAG have to use the same metrics to pick a parent,
>  (and if so, how do they agree)
>  I think that they do not, so long as they use an algorithm (such as
>  MRHOF) which has certain properties.
> 
> [Pascal]  The current spec requires that all players of an instance play by the same rule, that is same OF, in order to guarantee the expected result. 
> The goal there was deeper than the OF, like same OF with same parameterization. MRHOF is generic and allows various incarnations, depending in the metrics; all those incarnations are to be seen as different OFs WRT to RPL 's uniqueness rule.
> 
> Note that this rule is not for looplessness, the Rank would assure that anyway.
> 
> 
> - if we had multiple RPL instances in an LLN, using different metrics,
>  then we would have multiple RPL Instances and DODAGs.  The different
>  set of metrics would not co-exist in the same RPL Instance.
> 
> [Pascal] I think we agree. An instance can use multiple metrics bound by some logic. But those metrics and that logic must be identical throughout the instance. 
> A different (set of) metric means a different OF even if MRHOF is behind both.
> 
> Cheers,
> 
> Pascal
> 
> -- 
> Michael Richardson <[email-masked]>, Sandelman Software Works 
> IETF ROLL WG co-chair.    http://datatracker.ietf.org/wg/roll/charter/
>

---

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Mon, 4 Jun 2012 06:38:22 +0000

Hi:

I agree with Ralph that there is a choice to be made whether this OF is one OF or a generic OF that is instantiated by metric (a toolbox).

If it is a specific OF, it is fair to require from IANA a specific OF number (1 in the IANA section) but then we can expect that all implementations of this OF will behave the same.

Here, the behavior will vary quite dramatically with some implementation decisions (see the unresolved issue in the attached mail) and metric choices (Ralph's point here). 

As it goes, we could figure that MrHof is not one but many OFs.  If that's so then the OF number for the local variation should be assigned consistently for a deployment as opposed to globally significant. There could in fact be multiple MrHof in a same deployment.

My point is somewhat analogous to the UDP ports. Should we have a range for IANA reserved well known OFs, and then a range that would be assigned on demand? 

Cheers,

Pascal

-----Original Message-----
From: Ralph Droms [mailto:[email-masked]] 
Sent: vendredi 1 juin 2012 21:47
To: Michael Richardson
Cc: [email-masked]; Pascal Thubert (pthubert)
Subject: Re: [Roll] knowing which multiple metrics matter: MRHOF related questions

On May 25, 2012, at 12:31 PM 5/25/12, Pascal Thubert (pthubert) wrote:

> In contrast to the situation of ISPs and customers, where the ISP is 
> the network operator, ROLL networks are more like highly orchestrated

> Ralph> metrics?  The details of MRHOF seem to preclude the 
> Ralph> establishment of several RPL instances in an LLN, each of which 
> Ralph> uses MRHOF with a different selected metric.
> 

> If one wanted to have addresses in different subnets, and let the host  
> pick between them, then, one would need different layer-2s: different  
> VLANs or ESSIDs, or... )
> 

- Ralph

Subject: Re: [Roll] New Version Notification-draft-ietf-roll-minrank-hysteresis-of-07.txt
From: "Pascal Thubert (pthubert)" <[email-masked]>
To: Philip Levis <[email-masked]>
Cc: "[email-masked]" <[email-masked]>
Date: Tue, 10 Apr 2012 06:10:29 +0000

Phil:

It's not.
If I switch a device with another from a different vendor I should
expect the new device to interact properly with existing devices and I
should expect globally a similar behavior in my network.
That's what standards are for.

RPL 3.7.1 gives strong directions to avoid greediness. You may allow to
stray from that but that must be under control of the admin and you must
give strong recommendations for the default behavior.

Cheers,

Pascal

-----Original Message-----
From: Philip Levis [mailto:[email-masked]]
Sent: lundi 9 avril 2012 17:46
To: Pascal Thubert (pthubert)
Cc: [email-masked]
Subject: Re: [Roll] New Version Notification
-draft-ietf-roll-minrank-hysteresis-of-07.txt

On Apr 8, 2012, at 10:00 AM, Pascal Thubert (pthubert) wrote:

> Phil:
>
> It is an implementation decision as long as it does not lead to
> interop issue or unwanted behavior, which is probably the case here.
> RFC 6550 demands that a node can't stray from the best Rank as
> computed amongst parents by more than MaxRankIncrease.
> This 3 clauses do not seem to capture this that we are discussing do
> not seem to capture this.
> With (my reading of) the text as it stands, it seems that the
> implementation has all freedom to select parents and then is given
> rules to compute a Rank.
> The resulting Rank could be anything if the parent set as no
constraint.
>
>
> In particular  " The largest  calculated Rank among paths through the
> parent set, minus MaxRankIncrease" must be less that the Rank obtained

> from the preferred parent, not the final computed Rank.
>
>
> The node should:
> 1) Compute the Ranks form its parent set.
> 2) Determine a preferred parent as resulting with the lowest Rank
> (using the computation described in the previous paragraph)
> 3) Determine which parents generate a Rank that is between that lowest

> Rank and MaxRankIncrease
> 4) Pick a set of parents between those
>
> What do you think?

This is an implementation decision. I can imagine other algorithms,
e.g., where if the best Rank and next best Rank differ by more than
MaxRankIncrease, the node chooses to advertise a worse Rank than the
preferred parent provides.

Phil

---

From: Don Sturek <[email-masked]>
Date: Mon, 04 Jun 2012 06:20:42 -0700

Hi Pascal,

Having a range of reserved well known OF's from IANA makes sense.   This
is the minimum requirement to make MrHOF work.

If we do intend to support flexibility in deployment (eg industry groups
which may define their own OF's) then we would want a way to define on
demand OF's.

Don

On 6/3/12 11:38 PM, "Pascal Thubert (pthubert)" <[email-masked]> wrote:

>Hi:
>
>I agree with Ralph that there is a choice to be made whether this OF is
>one OF or a generic OF that is instantiated by metric (a toolbox).
>
>If it is a specific OF, it is fair to require from IANA a specific OF
>number (1 in the IANA section) but then we can expect that all
>implementations of this OF will behave the same.
>
>Here, the behavior will vary quite dramatically with some implementation
>decisions (see the unresolved issue in the attached mail) and metric
>choices (Ralph's point here).
>
>As it goes, we could figure that MrHof is not one but many OFs.  If
>that's so then the OF number for the local variation should be assigned
>consistently for a deployment as opposed to globally significant. There
>could in fact be multiple MrHof in a same deployment.
>
>My point is somewhat analogous to the UDP ports. Should we have a range
>for IANA reserved well known OFs, and then a range that would be assigned
>on demand? 
>
>Cheers,
>
>Pascal
>
>> I think RPL does not want to take party there. The OF is a piece of
>>logic to tie metrics and policies together.
>> As such, there could be multiple metrics as long as there is good logic
>>to tie them in. for instance one would look at optimizing metric A
>>within contraints as expressed by metric B and the OF model will allow
>>that.
>> 
>> OTOH, it a flows requires a certain optimization (say per one metric)
>>and another requires something different, then certainly you want two
>>instances.
>> 
>> So ... it depends!
>> 
>> Pascal
>> 
>> 
>> Pascal, Anders and Mukul about my assertion about how we would never
>>pick RPL instances by metrics; that they would in fact be seperate RPL
>>Instance numbers and DODAG values, and that these things would
>>provisioned by the network installer.
>> 
>> ====
>> 
>> I'm going to reply to your comments in a different order than you asked
>>them, because I think question #3 is most important, and the rest fall
>>out of it.
>> 
>> I think that it's important to understand that ROLL has a whole palette
>>of things that need to be provisioned by the "network operator".
>> 
>OK, so I get that the model is that MRHOF is a toolbox, where the
>specific tools are chosen when the device is flashed.  In that case, I
>think some additional review might be needed to ensure that the MRHOF
>spec is internally consistent with that point of view.
>
>As one example, I think the "selected metric" should be called out
>explicitly somewhere in section 5 and included in the list of configured
>parameters in section 6.1.
>
>to mean that if the selected metric is one of the metrics for which rank
>is undefined, the node joins as a lead node.  But, how can that happen if
>the selected metric is chosen by the network operator?
>
>> DODAG ("destinations") created within that instance.  The instances
>>share a common set of (provisioned) parameters.
>
>> (To put it into DHCP terms: if we have multiple DHCP servers on a link,
>> then one would expect them to all offer IP addresses in the same subnet.
>> If you feel that RPL is rather schizo about provisioning vs
>>configuration, then I agree.  It's not always clear to me why some
>>things are advertised while others are provisioned.
>> 
>> Finally, my impression is that how the various metrics are used
>>(singly, or in some combination) to calculate Rank Increase is a
>>question of further research, experimentation, and trade secret.
>
>Well, OK, but for the purposes of the MRHOF spec, it seems pretty clear
>to me there is one selected metric that is used across all nodes as a
>strictly additive path cost computation.
>
>> So long as the Rank increases, and a node does not flap between
>>parents, the exact details do not matter.  Each node can do it's parent
>>selection based upon the available metrics on it's own.  It advertises
>>the metrics it has.
>>

---

From: Ralph Droms <[email-masked]>
Date: Mon, 4 Jun 2012 12:42:14 -0400

> [Pascal]  The current spec requires that all players of an instance play by the same rule, that is same OF, in order to guarantee the expected result. 

Well ... given the current definitions of OF, I would say that even specifying the same OF across a RPL instance does not guarantee a consistent result.  OF0 allows pretty much any mechanism at all - within some constraints on the range of the computed rank - so there are no guarantees about how different nodes will compute their ranks under OF0.  If different nodes use a different metric under MRHOF, there will be no guarantees of consistency for MRHOF.

> The goal there was deeper than the OF, like same OF with same parameterization. MRHOF is generic and allows various incarnations, depending in the metrics; all those incarnations are to be seen as…

So, MRHOF+ETX is a different OF than MRHOF+link-quality?

> Note that this rule is not for looplessness, the Rank would assure that anyway.

OK.

> A different (set of) metric means a different OF even if MRHOF is behind both.

I'm not sure how to reconcile your first two sentences with the MRHOF spec.  As I read the MRHOF spec, any individual node uses a single metric to compute its rank.  I infer that all the nodes in a RPL instance (using MRHOF) would use the same metric.

I understand that other OFs might use a rank computation that combines several metrics.  Do I have it right that MRHOF uses a single metric?

- Ralph

---

From: Ralph Droms <[email-masked]>
Date: Mon, 4 Jun 2012 13:11:56 -0400

> Here, the behavior will vary quite dramatically with some implementation decisions (see the unresolved issue in the attached mail) and metric choices (Ralph's point here). 

Yes, I'm curious about the expected behavior - using a single OF number with multiple selected metrics will yield very different behaviors for a single OF.  My understanding of the purpose for OFs is to have predictable behavior in any RPL instance specifying the use of that OF.  In that case

> My point is somewhat analogous to the UDP ports. Should we have a range for IANA reserved well known OFs, and then a range that would be assigned on demand? 

The details of the IANA assignment can be settled after the fundamental issue about the relationship between MRHOF and the selected metric is decided.  At this point, MRHOF is defined to work with exactly 5 metrics.  It might be easiest to simply assign separate OF number to those 5 examples - e.g., OF1: MRHOF/ETX; OF2: MRHOF/latency - and assigning other OF numbers as other metrics are defined in the future.

As an aside, the hysteresis function described in MRHOF may be useful advice for other OFs, even those that combine multiple metrics into a single path computation.

- Ralph

> <Mail Attachment.eml>

---

From: Mukul Goyal <[email-masked]>
Date: Tue, 5 Jun 2012 01:33:32 -0500 (CDT)

Hi Pascal

>An instance can use multiple metrics bound by some logic. But those metrics and that logic must be identical throughout the instance. A different (set of) metric means a different OF even if MRHOF is behind both.

Could you please explain. This is not how I understand the relationship among an RPL Instance, an OF and metrics to be. As per my understanding, there is one OF associated with an RPL Instance and the metrics in use are chosen by the root. Accordingly, an RPL Instance _can_ have different DODAGIDs using different sets of metrics.

Thanks
Mukul 

----- Original Message -----
From: "Pascal Thubert (pthubert)" <[email-masked]>
To: "Michael Richardson" <[email-masked]>
Cc: [email-masked]
Sent: Monday, May 28, 2012 2:04:59 PM
Subject: Re: [Roll] knowing which multiple metrics matter: MRHOF related questions

Hello Micheal

My question is:

Cheers,

Pascal

---

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Tue, 5 Jun 2012 06:52:09 +0000

Hi Ralph:

Well, no. Whether an instance is a single DODAG (one root) or multiple, the instance has to be homogeneous. 
This is because nodes will move from one DODAG to the next when the environment requires the change, and the rules must not change. 

Cheers,

Pascal

-----Original Message-----
From: Mukul Goyal [mailto:[email-masked]] 
Sent: mardi 5 juin 2012 08:34
To: Pascal Thubert (pthubert)
Cc: [email-masked]; Michael Richardson
Subject: Re: [Roll] knowing which multiple metrics matter: MRHOF related questions

Hi Pascal

Thanks
Mukul 

----- Original Message -----
From: "Pascal Thubert (pthubert)" <[email-masked]>
To: "Michael Richardson" <[email-masked]>
Cc: [email-masked]
Sent: Monday, May 28, 2012 2:04:59 PM
Subject: Re: [Roll] knowing which multiple metrics matter: MRHOF related questions

Hello Micheal

My question is:

Cheers,

Pascal

---

From: Mukul Goyal <[email-masked]>
Date: Tue, 5 Jun 2012 02:11:45 -0500 (CDT)

Hi Pascal

So, what you are suggesting is that an RPLInstanceID is associated with (OF, metrics) pair and not just OF? This would not be in accordance with RC 6550, which associates an RPLInstanceID with an OF. My understanding is:

1) An RPLInstanceID is associated with an OF.
2) (RPLInstanceID, DODAGID) pair is associated with (OF, metrics) pair.

Thanks
Mukul

----- Original Message -----
From: "Pascal Thubert (pthubert)" <[email-masked]>
To: "Mukul Goyal" <[email-masked]>
Cc: [email-masked], "Michael Richardson" <[email-masked]>
Sent: Tuesday, June 5, 2012 1:52:09 AM
Subject: RE: [Roll] knowing which multiple metrics matter: MRHOF related questions

Hi Ralph:

Cheers,

Pascal

-----Original Message-----
From: Mukul Goyal [mailto:[email-masked]] 
Sent: mardi 5 juin 2012 08:34
To: Pascal Thubert (pthubert)
Cc: [email-masked]; Michael Richardson
Subject: Re: [Roll] knowing which multiple metrics matter: MRHOF related questions

Hi Pascal

Thanks
Mukul 

----- Original Message -----
From: "Pascal Thubert (pthubert)" <[email-masked]>
To: "Michael Richardson" <[email-masked]>
Cc: [email-masked]
Sent: Monday, May 28, 2012 2:04:59 PM
Subject: Re: [Roll] knowing which multiple metrics matter: MRHOF related questions

Hello Micheal

My question is:

Cheers,

Pascal

---

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Tue, 5 Jun 2012 07:55:04 +0000

Hello Mukul:

I'm curious: where did you get the second point from?

No, you can't switch metric as you switch DODAGID within an instance. Because nodes can move (jump is the word in the spec) from a DODAG to the next, and need to compare apple to apple when they do so...

Cheers,

Pascal

-----Original Message-----
From: Mukul Goyal [mailto:[email-masked]] 
Sent: mardi 5 juin 2012 09:12
To: Pascal Thubert (pthubert)
Cc: [email-masked]; Michael Richardson
Subject: Re: [Roll] knowing which multiple metrics matter: MRHOF related questions

Hi Pascal

Thanks
Mukul

----- Original Message -----
From: "Pascal Thubert (pthubert)" <[email-masked]>
To: "Mukul Goyal" <[email-masked]>
Cc: [email-masked], "Michael Richardson" <[email-masked]>
Sent: Tuesday, June 5, 2012 1:52:09 AM
Subject: RE: [Roll] knowing which multiple metrics matter: MRHOF related questions

Hi Ralph:

Cheers,

Pascal

-----Original Message-----
From: Mukul Goyal [mailto:[email-masked]] 
Sent: mardi 5 juin 2012 08:34
To: Pascal Thubert (pthubert)
Cc: [email-masked]; Michael Richardson
Subject: Re: [Roll] knowing which multiple metrics matter: MRHOF related questions

Hi Pascal

Thanks
Mukul 

----- Original Message -----
From: "Pascal Thubert (pthubert)" <[email-masked]>
To: "Michael Richardson" <[email-masked]>
Cc: [email-masked]
Sent: Monday, May 28, 2012 2:04:59 PM
Subject: Re: [Roll] knowing which multiple metrics matter: MRHOF related questions

Hello Micheal

My question is:

Cheers,

Pascal

---

From: Mukul Goyal <[email-masked]>
Date: Tue, 5 Jun 2012 03:09:49 -0500 (CDT)

Hi Pascal

>I'm curious: where did you get the second point from?

The mapping from (RPLInstanceID, DODAGID) pair to (OF, metrics) pair comes from my understanding that the root (the provider of DODAGID) decides what metrics to use.

>No, you can't switch metric as you switch DODAGID within an instance. Because nodes can move (jump is the word in the spec) from a DODAG to the next, and need to compare apple to apple when they do…

That makes sense. I stand corrected.

Cheers
Mukul

-----Original Message-----
From: Mukul Goyal [mailto:[email-masked]] 
Sent: mardi 5 juin 2012 09:12
To: Pascal Thubert (pthubert)
Cc: [email-masked]; Michael Richardson
Subject: Re: [Roll] knowing which multiple metrics matter: MRHOF related questions

Hi Pascal

Thanks
Mukul

----- Original Message -----
From: "Pascal Thubert (pthubert)" <[email-masked]>
To: "Mukul Goyal" <[email-masked]>
Cc: [email-masked], "Michael Richardson" <[email-masked]>
Sent: Tuesday, June 5, 2012 1:52:09 AM
Subject: RE: [Roll] knowing which multiple metrics matter: MRHOF related questions

Hi Ralph:

Cheers,

Pascal

-----Original Message-----
From: Mukul Goyal [mailto:[email-masked]] 
Sent: mardi 5 juin 2012 08:34
To: Pascal Thubert (pthubert)
Cc: [email-masked]; Michael Richardson
Subject: Re: [Roll] knowing which multiple metrics matter: MRHOF related questions

Hi Pascal

Thanks
Mukul 

----- Original Message -----
From: "Pascal Thubert (pthubert)" <[email-masked]>
To: "Michael Richardson" <[email-masked]>
Cc: [email-masked]
Sent: Monday, May 28, 2012 2:04:59 PM
Subject: Re: [Roll] knowing which multiple metrics matter: MRHOF related questions

Hello Micheal

My question is:

Cheers,

Pascal