Subject: Re: [Roll] I-D Action: draft-thubert-roll-flow-label-00.txt

From: Brian E Carpenter <[email-masked]>
Date: Fri, 04 May 2012 12:58:46 +0100

Hi,

IMHO there are problems with this proposal.

One is that it doesn't, as far as I can see, demonstrate that
the overhead of rewriting the flow label on exit from an RPL
domain is significantly less than the overhead of dealing
with the RPL hop-by-hop option at the edge of the domain.

Then, this phrase:

>    Sadly, the Option must be placed in
>    a Hop-by-Hop option that must be inserted or removed as the packet
>    crosses the border of the RPL domain. 

bothers me, because (and I have looked recently) there is nothing
in the IPv6 specs that allows for an extension header being
inserted (or removed) en route. The text continues

>    ...This operation may involve an
>    extra encapsulation that is detrimental to the network operation, in
>    particular with regards to bandwidth and battery constraints.

Why "may"? If the packet is inbound to RPL, RFC 6553 makes it clear
that it must be encapsulated; if the packet is outbound, why would
it be encapsulated at all? This case is covered in the RFC too:

>    3.  Datagrams destined to nodes outside the RPL Instance are dropped
>        if the outermost IPv6 header contains a RPL Option not generated
>        by the RPL router forwarding the datagram.

Then, the draft says:

>    This document specifies how the Flow Label can be reused within the
>    RPL domain as a replacement to the RPL option.  The use of the Flow
>    Label within a RPL domain is an instance of the stateful scenarios
>    decribed in [RFC6437]...

In fact RFC 6437 carefully says very little about stateful scenarios; it
does not describe them, but it does constrain them. Here is the entire
section on this topic:

> 4.  Flow State Establishment Requirements
> 
>    A node that sets the flow label MAY also take part in a flow state
>    establishment method that results in assigning specific treatments to
>    specific flows, possibly including signaling.  Any such method MUST
>    NOT disturb nodes taking part in the stateless scenario just
>    described.  Thus, any node that sets flow label values according to a
>    stateful scheme MUST choose labels that conform to Section 3 of this
>    specification.  Further details are not discussed in this document.

The problem is that section 3 intentionally does not consider flow label
values in which any of the bits have semantic significance, and this was
the consensus in the 6MAN WG after a great deal of discussion. However,
the present proposal assigns semantics to various bits in the flow
label, destroying its desired property of belonging to a statistically
uniform distribution.

The issue is clearly stated in RFC 6437:

>    Once set to a non-zero value, the Flow Label is expected to be
>    delivered unchanged to the destination node(s).  A forwarding node
>    MUST either leave a non-zero flow label value unchanged or change it
>    only for compelling operational security reasons as described in
>    Section 6.1.

Regards
   Brian Carpenter

(please keep me on CC as I am not on the ROLL list)

---

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Mon, 7 May 2012 14:59:08 +0200

Hi Brian:

 >One is that it doesn't, as far as I can see, demonstrate that the overhead of rewriting the flow label on exit from an RPL domain is significantly less than the overhead of dealing with the RPL hop-by-hop option at the edge of the domain.

[Pascal] The overhead is mainly for packets coming from the outside into the RPL domain. Such packets need extra encapsulation that's detrimental for batteries and may cause a small packet to grow above the MAC Max Frame, leading to L2 or 6LoWPAN fragmentation.

>    crosses the border of the RPL domain. 

bothers me, because (and I have looked recently) there is nothing in the IPv6 specs that allows for an extension header being inserted (or removed) en route. The text continues

[Pascal] I'll reword to indicate that the operation of inserting / removing the HbH header comes with 6in6.
 RFC 6553 says: 
"
   When the router is the source of the original packet and the
   destination is known to be within the same RPL Instance, the router
   SHOULD include the RPL Option directly within the original packet.
   Otherwise, routers MUST use IPv6-in-IPv6 tunneling [RFC2473] and
   place the RPL Option in the tunnel header.  Using IPv6-in-IPv6
   tunneling ensures that the delivered datagram remains unmodified and
   that ICMPv6 errors generated by a RPL Option are sent back to the
   router that generated the RPL Option."

>    particular with regards to bandwidth and battery constraints.

Why "may"? If the packet is inbound to RPL, RFC 6553 makes it clear that it must be encapsulated; if the packet is outbound, why would it be encapsulated at all? This case is covered in the RFC too:

[Pascal] I wrote "may" from the fact that the packet may be issued and consumed within the RPL domain in which case the option is conserved all the way and there is no 6in6 encapsulation.
But I agree the operation above does come with 6in6.

>    Label within a RPL domain is an instance of the stateful scenarios

In fact RFC 6437 carefully says very little about stateful scenarios; it does not describe them, but it does constrain them. Here is the entire section on this topic:

[Pascal] what about the word "introduced", or "discussed"?

>The problem is that section 3 intentionally does not consider flow label values in which any of the bits have semantic significance, and this was the consensus in the 6MAN WG after a great deal of discussion. However, the  present proposal assigns semantics to various bits in the flow label, destroying its desired property of belonging to a statistically uniform distribution.

[Pascal] That is true within the edge network that is the RPL domain. But outside the RPL domain the desired properties are conserved. The rationale for the statistically uniform distribution does not necessarily bring a lot of value within the LLN. On the other hand, the 6in6 encaps and HbH header are hugely detrimental to the operations and will result is shorter battery life and reduced applicability. I participated as a coeditor to the ISA100.11a standard that actually uses IPv6 in the 6LoWPAN HC form. I've seen how the adoption of IPv6 was at stake for a single additional octet in a header, and this actually led to changes at the IETF and the new 6LoWPAN HC. Indeed, unless you have a power source, every octet that you have to place it in every frame or packet in the LLN counts dearly.

>    only for compelling operational security reasons as described in

[Pascal]  We have a compelling operational reason for sure, because the power consumption is the number one critical constraint that drives most of the physical design choices in a LLN where a battery life must often expressed in years.  I'm unclear, though, why RFC 6437 went as far as indicating that the compelling reason should be qualified as security. I take that as an indication of criticality as opposed to a tentative to predict future needs. It makes sense to me to add in the draft that "the source in the LLN SHOULD set the Flow label to zero and MUST NOT expect that the flow label will be conserved end to end if it knows by policy that the LLN will use this draft". Would that be sufficient to alleviate this concern?

And yes, I will certainly keep you CCed, and CC you if other threads pop up on the subject in ROLL, if you do not mind;

Pascal

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 08 May 2012 10:52:48 +0100

On 2012-05-07 13:59, Pascal Thubert (pthubert) wrote:
> Hi Brian:
> 
>> IMHO there are problems with this proposal. One is that it doesn't,
>> as far as I can see, demonstrate that the overhead of rewriting the
>> flow label on exit from an RPL domain is significantly less than
>> the overhead of dealing with the RPL hop-by-hop option at the edge
>> of the domain.
> 
> [Pascal] The overhead is mainly for packets coming from the outside
> into the RPL domain. Such packets need extra encapsulation that's
> detrimental for batteries and may cause a small packet to grow above
> the MAC Max Frame, leading to L2 or 6LoWPAN fragmentation.

If you can explain this in a few words in the text, it will be fine.

> 
> Then, this phrase:
> 
>> Sadly, the Option must be placed in a Hop-by-Hop option that must
>> be inserted or removed as the packet crosses the border of the RPL
>> domain.
> 
> bothers me, because (and I have looked recently) there is nothing in
> the IPv6 specs that allows for an extension header being inserted (or
> removed) en route. The text continues
> 
> [Pascal] I'll reword to indicate that the operation of inserting /
> removing the HbH header comes with 6in6. RFC 6553 says: " When the
> router is the source of the original packet and the destination is
> known to be within the same RPL Instance, the router SHOULD include
> the RPL Option directly within the original packet. Otherwise,
> routers MUST use IPv6-in-IPv6 tunneling [RFC2473] and place the RPL
> Option in the tunnel header.  Using IPv6-in-IPv6 tunneling ensures
> that the delivered datagram remains unmodified and that ICMPv6 errors
> generated by a RPL Option are sent back to the router that generated
> the RPL Option."

Yes, I had to read quite deeply into 6553 to understand this. I think
your draft will be read by people without detailed knowledge of RPL so
it needs to be explained.

> 
>> ...This operation may involve an extra encapsulation that is
>> detrimental to the network operation, in particular with regards to
>> bandwidth and battery constraints.
> 
> that it must be encapsulated; if the packet is outbound, why would it
> be encapsulated at all? This case is covered in the RFC too:
> 
> [Pascal] I wrote "may" from the fact that the packet may be issued
> and consumed within the RPL domain in which case the option is
> conserved all the way and there is no 6in6 encapsulation. But I agree
> the operation above does come with 6in6.

OK

> 
>> 3.  Datagrams destined to nodes outside the RPL Instance are
>> dropped if the outermost IPv6 header contains a RPL Option not
>> generated by the RPL router forwarding the datagram.
> 
> Then, the draft says:
> 
>>  RPL domain as a replacement to the RPL option.  The use of the
>> Flow Label within a RPL domain is an instance of the stateful
>> scenarios decribed in [RFC6437]...
> 
> In fact RFC 6437 carefully says very little about stateful scenarios;
> it does not describe them, but it does constrain them. Here is the
> entire section on this topic:
> 

OK

>> establishment method that results in assigning specific treatments
>> to specific flows, possibly including signaling.  Any such method
>> MUST NOT disturb nodes taking part in the stateless scenario just 
>> described.  Thus, any node that sets flow label values according to
>> a stateful scheme MUST choose labels that conform to Section 3 of
>> this specification.  Further details are not discussed in this
>> document.
> 
>> The problem is that section 3 intentionally does not consider flow
>> label values in which any of the bits have semantic significance,
>> and this was the consensus in the 6MAN WG after a great deal of
>> discussion. However, the  present proposal assigns semantics to
>> various bits in the flow label, destroying its desired property of
>> belonging to a statistically uniform distribution.
> 
> [Pascal] That is true within the edge network that is the RPL domain.
> But outside the RPL domain the desired properties are conserved. The
> rationale for the statistically uniform distribution does not
> necessarily bring a lot of value within the LLN. On the other hand,
> the 6in6 encaps and HbH header are hugely detrimental to the
> operations and will result is shorter battery life and reduced
> applicability. I participated as a coeditor to the ISA100.11a
> standard that actually uses IPv6 in the 6LoWPAN HC form. I've seen
> how the adoption of IPv6 was at stake for a single additional octet
> in a header, and this actually led to changes at the IETF and the new
> 6LoWPAN HC. Indeed, unless you have a power source, every octet that
> you have to place it in every frame or packet in the LLN counts
> dearly.

I understand that, but there was a strong feeling in 6MAN that the flow
label should be an end-to-end field (even though that is unenforceable).
If you can make it clear that the flow label in your model refers only
to the label of the encapsulating packet, and that any flow label set by
the source host in the inner packet is not changed, it will be fine.
That's the same model as RFC 6438 (flow labels for ECMP/LAG tunnels).

>> MUST either leave a non-zero flow label value unchanged or change
>> it only for compelling operational security reasons as described in
>>  Section 6.1.
> 
> [Pascal]  We have a compelling operational reason for sure, because
> the power consumption is the number one critical constraint that
> drives most of the physical design choices in a LLN where a battery
> life must often expressed in years.  I'm unclear, though, why RFC
> 6437 went as far as indicating that the compelling reason should be
> qualified as security. 

Because the firewall people had already made it very clear that whatever
we wrote in an RFC, firewalls in sensitive applications would treat the
flow label as a covert channel risk. Apart from that, as noted above,
the WG wanted to insist on the end to end property of the label.

> I take that as an indication of criticality as
> opposed to a tentative to predict future needs. It makes sense to me
> to add in the draft that "the source in the LLN SHOULD set the Flow
> label to zero and MUST NOT expect that the flow label will be
> conserved end to end if it knows by policy that the LLN will use this
> draft". Would that be sufficient to alleviate this concern?

If we are talking about the encapsulating packet, sure. Is there any
reason that the label in the raw packet would be changed?

   Brian

> 
> And yes, I will certainly keep you CCed, and CC you if other threads
> pop up on the subject in ROLL, if you do not mind;
> 
> Pascal

---

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Tue, 8 May 2012 14:50:09 +0200

Hi Brian:

I'll be publishing a fix soon. That a bunch for those comments.

> If we are talking about the encapsulating packet, sure. Is there any reason that the label in the raw packet would be changed?
But that's the core problem we'll have to solve. This draft aims specifically at removing the extra encapsulation that consumes octets and thereby battery.

If we have no extra encaps then there is no outer flow label to play with; we have to use the 'end to end' flow label that is not 'end to end 'anymore but has 2 separate lives, one within the LLN and one in the Internet as we know it.

Remarks there:
- RFC 6553 is a definite no go in the fringe. When you're spending considerable efforts to save a single bit in a standard like ISA100.11a, WiHART or the equivalent IEC versions, you do not think for a minute of adding 20 bits that have to look like random. Rather, you ask whether you really want IPv6 in that world at all. Guess what, WiHART went without IP.
- The extra encaps is certainly a major concern for ZigbeeIP people as well, and the main reason why we published draft-hui-6man-rpl-headers . 
- RFC 6553 was designed with a certain mind set that's focused on the benefit for the Internet core but ignores the fringe that's building at the edge. My conclusion is that we must ensure that it is respected in the domain where it is designed to apply, and the draft tries to reach that goal. You'll note that the fringe has a potential to represent orders of magnitude more devices (dare I call those hosts or routers?) than the Internet as we know it.
- Evolution is the key for survival. I've been advocating for (more than 5) years that IPv6 can evolve and be relevant in the fringe; did that at Cisco Networkers, Emerson Exchange,  ETSO M2M, and ISA/IEC SDO meetings. Please do not prove me wrong.

Cheers,

Pascal

-----Original Message-----
From: Brian E Carpenter [mailto:[email-masked]] 
Sent: mardi 8 mai 2012 11:53
To: Pascal Thubert (pthubert)
Cc: [email-masked]
Subject: Re: I-D Action: draft-thubert-roll-flow-label-00.txt

On 2012-05-07 13:59, Pascal Thubert (pthubert) wrote:
>> flow label on exit from an RPL domain is significantly less than the 
>> overhead of dealing with the RPL hop-by-hop option at the edge of the 
>> domain.
> 

> 
> Then, this phrase:
> 
>> Sadly, the Option must be placed in a Hop-by-Hop option that must be 
>> inserted or removed as the packet crosses the border of the RPL 
>> domain.
> 
> the RPL Option directly within the original packet. Otherwise, routers 
> MUST use IPv6-in-IPv6 tunneling [RFC2473] and place the RPL Option in 
> the tunnel header.  Using IPv6-in-IPv6 tunneling ensures that the 
> delivered datagram remains unmodified and that ICMPv6 errors generated 
> by a RPL Option are sent back to the router that generated the RPL 
> Option."

Yes, I had to read quite deeply into 6553 to understand this. I think your draft will be read by people without detailed knowledge of RPL so it needs to be explained.

> [Pascal] I wrote "may" from the fact that the packet may be issued and 
> consumed within the RPL domain in which case the option is conserved 
> all the way and there is no 6in6 encapsulation. But I agree the 
> operation above does come with 6in6.

OK

>> if the outermost IPv6 header contains a RPL Option not generated by 
>> the RPL router forwarding the datagram.
> 
> Then, the draft says:
> 

OK

>> label values in which any of the bits have semantic significance, and 
>> this was the consensus in the 6MAN WG after a great deal of 
> applicability. I participated as a coeditor to the ISA100.11a standard 
> that actually uses IPv6 in the 6LoWPAN HC form. I've seen how the 
> adoption of IPv6 was at stake for a single additional octet in a 
> header, and this actually led to changes at the IETF and the new 

I understand that, but there was a strong feeling in 6MAN that the flow label should be an end-to-end field (even though that is unenforceable).
If you can make it clear that the flow label in your model refers only to the label of the encapsulating packet, and that any flow label set by the source host in the inner packet is not changed, it will be fine.
That's the same model as RFC 6438 (flow labels for ECMP/LAG tunnels).

> 6437 went as far as indicating that the compelling reason should be 

Because the firewall people had already made it very clear that whatever we wrote in an RFC, firewalls in sensitive applications would treat the flow label as a covert channel risk. Apart from that, as noted above, the WG wanted to insist on the end to end property of the label.

> I take that as an indication of criticality as opposed to a tentative 
> to predict future needs. It makes sense to me to add in the draft that 
> "the source in the LLN SHOULD set the Flow label to zero and MUST NOT 
> expect that the flow label will be conserved end to end if it knows by 
> policy that the LLN will use this draft". Would that be sufficient to 
> alleviate this concern?

If we are talking about the encapsulating packet, sure. Is there any reason that the label in the raw packet would be changed?

   Brian

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 08 May 2012 14:43:43 +0100

> But that's the core problem we'll have to solve. This draft aims specifically at removing the extra encapsulation that consumes octets and thereby battery.

OK, somehow this was still not very clear to me. I think maybe some
diagrams for the non-RPL-expert are needed.

I try to avoid being doctrinaire; all I can tell you is the models we
proposed in 6man which involved rewriting the flow label at domain
boundaries were very unpopular.

You should probably look at draft-carpenter-6man-flow-update-03
(and only the -03 version). I think that would have allowed what you
want to do, but it was firmly rejected by 6man.

Thus what I am about to say is heresy: as long as the flow labels
emerging from the RPL domain *look* as if the source computer set
a flow label per flow conformant with RFC 6437, it should be OK.

Except for this complication: in the world of server load balancing,
there is some interest in using the same flow label for multiple
transport sessions from the same client, even if the client's
IP address changes due to mobility. This is still pretty speculative,
but it really does require the source node to have control over
the final value of the flow label.

    Brian

---

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Wed, 9 May 2012 10:31:35 +0200

Hi Brian:
 
>OK, somehow this was still not very clear to me. I think maybe some diagrams for the non-RPL-expert are needed.
 >I try to avoid being doctrinaire; all I can tell you is the models we proposed in 6man which involved rewriting the flow label at domain boundaries were very unpopular.

[Pascal] This is MUCH appreciated, Brian. 

>You should probably look at draft-carpenter-6man-flow-update-03[]  (and only the -03 version). I think that would have allowed what you want to do, but it was firmly rejected by 6man.

>Thus what I am about to say is heresy: as long as the flow labels emerging from the RPL domain *look* as if the source computer set a flow label per flow conformant with RFC 6437, it should be OK.

>Except for this complication: in the world of server load balancing, there is some interest in using the same flow label for multiple transport sessions from the same client, even if the client's IP address changes due to mobility. This is still pretty speculative, but it really does require the source node to have control over the final value of the flow label.

[Pascal] I added text and published 01 per our discussions. The text indicates that the RPL option is still an option. This is true in particular when the flow label as another usage or if the part of the Rank that is meaningful to establish the path consistency cannot be compressed in one octet.

Thanks a bunch for all;

Pascal
>>> overhead of dealing with the RPL hop-by-hop option at the edge of 
>>> the domain.
>>> this specification.  Further details are not discussed in this document.
>> to predict future needs. It makes sense to me to add in the draft 
>> that "the source in the LLN SHOULD set the Flow label to zero and 
>> MUST NOT expect that the flow label will be conserved end to end if 
>> it knows by policy that the LLN will use this draft". Would that be 
>> sufficient to alleviate this concern?
>