Subject: Re: [Roll] draft-rahul-roll-rpl-observations-00 Section 2.1: Wear leveling

From: Michael Richardson <[email-masked]>
Date: Wed, 02 May 2018 18:49:18 -0400

Rahul Arvind Jadhav <[email-masked]> wrote:
    > module doing such (relatively) frequent writes). Also RPL signaling
    > does not allow multiple state information to be clubbed together and do
    > a single write(write is in terms of pages and thus not good for less
    > number of bytes). The question is, Is it possible for RPL signaling to
    > reduce persistent write at the cost of adding some signaling during
    > node boot (at-least for 6LRs and 6LNs). Node reboot will be relatively
    > rare-occurrence (or at-least much less frequent) and thus the cost of
    > additional signaling bits might be worth it (?). Also I feel new RPL
    > extensions should consider this and do not commit to persistent storage
    > more frequently.

I think that I don't remember why a node needs to save the DTSN across
reboots.  (I hate being away from code for so long that I forget these
things, I wish I could spend more time writing code)

I think that in storing mode, that a node can set it's DTSN to whatever value
is sees in the parents' DIO, and increment from there.

I'm confused as to why any node other than the root needs it to be stored in
flash?   I do not think it should be required to store info in flash.
(OSPF doesn't require this...)
This also makes it hard to replace hardware in the field, or do firmware
updates if we have to keep this kind of data around.

Can we expand 7.1 to better explain what is written, and why?

-- 
]               Never tell me the odds!                 | ipv6 mesh networks [ 
]   Michael Richardson, Sandelman Software Works        | network architect  [ 
]     [email-masked]  http://www.sandelman.ca/        |   ruby on rails    [

---

From: Rahul Jadhav <[email-masked]>
Date: Thu, 03 May 2018 00:41:21 +0000

On Thu, 3 May 2018 at 4:49 AM, Michael Richardson <[email-masked]>
wrote:

> reboots.

[RJ] dtsn is for triggering the dao from downstream nodes. If the child
nodes do not see a new dtsn from the parent then they won't send dao and
thus routing state for the (sub)childs won't be added for any child nodes
 on the restarted 6lr.
An example,.... the 6lr dtsn was 245 when it got restarted and on restart
it sets its dtsn to default 240... Now the child nodes (they have stored
245) don't respond to the DIOs because they see an older dtsn(value 240).
Thus no DAOs are triggered causing no routing state to be installed on
restarted 6lr for the (sub)childs resulting in unavailability of downstream
path to the subdodag rooted at that 6lr.

(I hate being away from code for so long that I forget these
> is sees in the parents' DIO, and increment from there.

[RJ] i do not think that a node infers or should infer its dtsn from
parents dio... Per section 7.2 of 6550, the default dtsn should be set to
240 (assuming sequence window of 16).

> flash?   I do not think it should be required to store info in flash.

[RJ] this is true only for non storing mop but for storing mop the 6lrs
needs to remember for the reason mentioned above.

> updates if we have to keep this kind of data around.

[RJ]true

> Can we expand 7.1 to better explain what is written, and why?

[RJ] sure will do.

> _______________________________________________
> Roll mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/roll
>

---

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Thu, 3 May 2018 07:47:21 +0000

Hello Rahul :

Rebooting within the lollipop is really bad luck; the whole point of the lollipop is to detect intermittent reboots and not force to store the sequence in flash.
Note that you do not need to start from default 240, you may restart from, say, 250. Or you may increase the sequence by more than one. All that to stay less in the straight piece of the lollipop.
All in all, I do not think that the DTSN has to be stored in persistent memory. Is there text that I do not remember that would indicate so?

Take care,

Pascal

From: Roll <[email-masked]> On Behalf Of Rahul Jadhav
Sent: jeudi 3 mai 2018 02:41
To: Routing Over Low power and Lossy networks <[email-masked]>
Subject: Re: [Roll] draft-rahul-roll-rpl-observations-00 Section 2.1: Wear leveling

On Thu, 3 May 2018 at 4:49 AM, Michael Richardson <[email-masked]<mailto:[email-masked]>> wrote:

[RJ] dtsn is for triggering the dao from downstream nodes. If the child nodes do not see a new dtsn from the parent then they won't send dao and thus routing state for the (sub)childs won't be added for any child nodes  on the restarted 6lr.
An example,.... the 6lr dtsn was 245 when it got restarted and on restart it sets its dtsn to default 240... Now the child nodes (they have stored 245) don't respond to the DIOs because they see an older dtsn(value 240). Thus no DAOs are triggered causing no routing state to be installed on restarted 6lr for the (sub)childs resulting in unavailability of downstream path to the subdodag rooted at that 6lr.

[RJ] i do not think that a node infers or should infer its dtsn from parents dio... Per section 7.2 of 6550, the default dtsn should be set to 240 (assuming sequence window of 16).

[RJ] this is true only for non storing mop but for storing mop the 6lrs needs to remember for the reason mentioned above.

[RJ]true

[RJ] sure will do.

--
]               Never tell me the odds!                 | ipv6 mesh networks [
]   Michael Richardson, Sandelman Software Works        | network architect  [
]     [email-masked]<mailto:[email-masked]>  http://www.sandelman.ca/        |   ruby on rails    [

_______________________________________________
Roll mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/roll

---

From: Michael Richardson <[email-masked]>
Date: Thu, 03 May 2018 09:30:28 -0400

Pascal Thubert (pthubert) <[email-masked]> wrote:
    > Rebooting within the lollipop is really bad luck; the whole point of
    > the lollipop is to detect intermittent reboots and not force to store
    > the sequence in flash. 
    > Note that you do not need to start from default 240, you may restart
    > from, say, 250. Or you may increase the sequence by more than one. All
    > that to stay less in the straight piece of the lollipop.

I don't understand why a node can't take the DTSN from it's parent as it's
base value.

---

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Thu, 3 May 2018 13:37:30 +0000

Hello Michael

There might be a confusion with the DODAG version. DTSN is sourced at each node, individually.

Take care,

Pascal

> -----Original Message-----
> From: Roll <[email-masked]> On Behalf Of Michael Richardson
> Sent: jeudi 3 mai 2018 15:30

---

From: Michael Richardson <[email-masked]>
Date: Thu, 03 May 2018 13:15:18 -0400

> There might be a confusion with the DODAG version. DTSN is sourced at

I'm not saying it has to be synchronized, but rather it can always be > than
parent. If one does that, then there is no bootstrap problem.

Isn't that the non-storing case anyway?

DODAG version would always be the same.

I looked again, and I don't see anything about DTSN being a lollipop
counter that starts at 240, and wraps at 239.  That would, I think make it
clear when a node has booted, and maybe that solves the problem.

---

From: Rahul Jadhav <[email-masked]>
Date: Fri, 04 May 2018 01:12:19 +0000

Thanks Pascal..

Yes, Rebooting within the lollipop is the problem that i have described in
the draft... It is not possible to completely avoid that scenario if
persistent storage is not used... The problem is, the nodes cannot recover
(or they recover very very late) if they get into this scenario. Choosing a
higher increment factor may reduces the probability but cannot eliminate
it.

There is no text in 6550 which says dtsn should be maintained in persistent
storage. But  then 6550 does not talk about handling reboots in general.

On Thu, 3 May 2018 at 1:47 PM, Pascal Thubert (pthubert) <[email-masked]>
wrote:

> Hello Rahul :
>
>
>
> Rebooting within the lollipop is really bad luck; the whole point of the
> lollipop is to detect intermittent reboots and not force to store the
> sequence in flash.
>
> Note that you do not need to start from default 240, you may restart from,
> say, 250. Or you may increase the sequence by more than one. All that to
> stay less in the straight piece of the lollipop.
>
> All in all, I do not think that the DTSN has to be stored in persistent
> memory. Is there text that I do not remember that would indicate so?
>
>
>
> Take care,
>
>
>
> Pascal
>
>
>
> *From:* Roll <[email-masked]> *On Behalf Of *Rahul Jadhav
> *Sent:* jeudi 3 mai 2018 02:41
>
>
> *To:* Routing Over Low power and Lossy networks <[email-masked]>
>
> *Subject:* Re: [Roll] draft-rahul-roll-rpl-observations-00 Section 2.1:
> Wear leveling
>
>
>
>
>

---

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Fri, 4 May 2018 07:17:49 +0000

True, Rahul

Maybe your doc could point the lack of reboot text as the main issue. Then point out that there is a trade off between persistent memory, size of the straight part of the lollipop and time to recover from 2 consecutive reboots. This trade off applies to all the sequence numbers in RPL... but in the end things recover anyway.

Regards,

Pascal

Le 4 mai 2018 à 03:17, Rahul Jadhav <[email-masked]<mailto:[email-masked]>> a écrit :

Thanks Pascal..

Yes, Rebooting within the lollipop is the problem that i have described in the draft... It is not possible to completely avoid that scenario if persistent storage is not used... The problem is, the nodes cannot recover (or they recover very very late) if they get into this scenario. Choosing a higher increment factor may reduces the probability but cannot eliminate it.

There is no text in 6550 which says dtsn should be maintained in persistent storage. But  then 6550 does not talk about handling reboots in general.

On Thu, 3 May 2018 at 1:47 PM, Pascal Thubert (pthubert) <[email-masked]<mailto:[email-masked]>> wrote:
Hello Rahul :

Take care,

Pascal

From: Roll <[email-masked]<mailto:[email-masked]>> On Behalf Of Rahul Jadhav
Sent: jeudi 3 mai 2018 02:41

To: Routing Over Low power and Lossy networks <[email-masked]<mailto:[email-masked]>>
Subject: Re: [Roll] draft-rahul-roll-rpl-observations-00 Section 2.1: Wear leveling

[RJ]true

[RJ] sure will do.

---

From: rabi narayan sahoo <[email-masked]>
Date: Fri, 04 May 2018 08:15:45 +0000

Hi
I agree with Rahul. I want to add few more points here.
1- If a rebooted router has to chose a higher value how it will decide
which one to choose so that it can recover it's reouting table. Choosing
higher value still has chance of failure in recovery
2- If wait for auto recovery we are not sure how long it will take
.Currently we have only one option
a- Node sends a refresh DAO
Till that time all the nodes having downward path via the rebooted node
won't be reachable unless until they switch the pp.

RFC 6550 must consider the reboot case in general for storing mode. In non
storing mode we doesn't have such issue
Thanks
Rabi

On Fri 4 May, 2018, 6:42 AM Rahul Jadhav, <[email-masked]> wrote:

> There is no text in 6550 which says dtsn should be maintained in
> persistent storage. But  then 6550 does not talk about handling reboots in
> general.
>
>
> On Thu, 3 May 2018 at 1:47 PM, Pascal Thubert (pthubert) <
> [email-masked]> wrote:
>
>> Hello Rahul :
>>
>>
>>
>> from, say, 250. Or you may increase the sequence by more than one. All that
>> to stay less in the straight piece of the lollipop.
>>

---

From: Rahul Arvind Jadhav <[email-masked]>
Date: Mon, 7 May 2018 10:57:08 +0000

Hi Michael,

That's a good point you made about DTSN not been a lollipop counter (this was clearly misinterpreted by us and others in implementations). Section 7 explains "Sequence Counters" but DAO Trigger Sequence Number (DTSN) is not considered as sequence counter in that section!!

Please find my response inline ... 

-----Original Message-----
From: Roll [mailto:[email-masked]] On Behalf Of Michael Richardson
Sent: 04 May 2018 01:15
To: Routing Over Low power and Lossy networks <[email-masked]>
Subject: Re: [Roll] draft-rahul-roll-rpl-observations-00 Section 2.1: Wear leveling

    > There might be a confusion with the DODAG version. DTSN is sourced at

I'm not saying it has to be synchronized, but rather it can always be > than parent. If one does that, then there is no bootstrap problem.

[RJ] Even in non-storing case the DTSN is not derived from the parent's DTSN as per the spec. 

I looked again, and I don't see anything about DTSN being a lollipop counter that starts at 240, and wraps at 239.  That would, I think make it clear when a node has booted, and maybe that solves the problem.

[RJ] Very interesting point. We assumed that DTSN is a lollipop counter in our implementation. But looks like it is not. Section 7 explicitly states that only DAOSequence, DODAGVersionNumber, and Path Sequence are lollipop counters ... Looks like we read the spec wrong, also Contiki (including ng) has got it wrong (i.e. uses lollipop counter for DTSN). RIOT (rightly as per spec) does not assume lollipop counter for DTSN.
I think DTSN needs to be a lollipop counter which alleviates the problem but still does not completely avoid the persistent storage (discussed in other mail in same thread). This point should be added to the draft.

---

From: Michael Richardson <[email-masked]>
Date: Mon, 07 May 2018 12:05:53 -0400

Rahul Arvind Jadhav <[email-masked]> wrote:
    > That's a good point you made about DTSN not been a lollipop counter
    > (this was clearly misinterpreted by us and others in
    > implementations). Section 7 explains "Sequence Counters" but DAO
    > Trigger Sequence Number (DTSN) is not considered as sequence counter in
    > that section!! 

    mcr> I looked again, and I don't see anything about DTSN being a lollipop
    mcr> counter that starts at 240, and wraps at 239.  That would, I think
    mcr> make it clear when a node has booted, and maybe that solves the
    mcr> problem. 

    > [RJ] Very interesting point. We assumed that DTSN is a lollipop counter
    > in our implementation. But looks like it is not. Section 7 explicitly
    > states that only DAOSequence, DODAGVersionNumber, and Path Sequence are
    > lollipop counters ... Looks like we read the spec wrong, also Contiki
    > (including ng) has got it wrong (i.e. uses lollipop counter for
    > DTSN). RIOT (rightly as per spec) does not assume lollipop counter for
    > DTSN.
    
    > I think DTSN needs to be a lollipop counter which alleviates the
    > problem but still does not completely avoid the persistent storage
    > (discussed in other mail in same thread). This point should be added to
    > the draft.

So I think that the errata against 6550 is that it should be a lollipop
counter, and I think that this solves the reboot problem, and eliminates
having to write it to flash.

---

From: Rahul Jadhav <[email-masked]>
Date: Mon, 07 May 2018 16:19:29 +0000

On Mon, 7 May 2018 at 9:36 PM, Michael Richardson <[email-masked]>
wrote:

> having to write it to flash.

[RJ] this still does not solve the reboot issue... The text in the draft
actually considers dtsn to be a lollipop counter and explains the problem
when rebooting within the sequence window of lollipop counter. Please refer
to section 7.2 of the draft for example.
Errata is nonetheless needed for dtsn to be a lollipop counter.

---

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Fri, 11 May 2018 07:14:41 +0000

Hello Michael:
 
> So I think that the errata against 6550 is that it should be a lollipop counter,
> and I think that this solves the reboot problem, and eliminates having to write
> it to flash.
> 
Agreed, it should have been listed in section 7.

Take care,
Pascal

---

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Fri, 11 May 2018 07:18:04 +0000

Hello Rahul

From: Roll <[email-masked]> On Behalf Of Rahul Jadhav
Sent: lundi 7 mai 2018 18:19
To: Routing Over Low power and Lossy networks <[email-masked]>
Subject: Re: [Roll] draft-rahul-roll-rpl-observations-00 Section 2.1: Wear leveling

On Mon, 7 May 2018 at 9:36 PM, Michael Richardson <[email-masked]<mailto:[email-masked]>> wrote:

[RJ] this still does not solve the reboot issue... The text in the draft actually considers dtsn to be a lollipop counter and explains the problem when rebooting within the sequence window of lollipop counter. Please refer to section 7.2 of the draft for example.
Errata is nonetheless needed for dtsn to be a lollipop counter.

[PT>] Well, yes, but that’s true for any of the sequence counters, not just DTSN, and was not perceived as an “issue”.
The lollipop by design detects one reboot during the straight part. More will cause a delay.
There are ways to avoid that like persistent memory or alleviate it like making the straight part shorted on reliable links.
I’m not sure there’s much to add to that or to fix, is there?

Pascal

---

From: Michael Richardson <[email-masked]>
Date: Sat, 12 May 2018 14:57:51 -0400

Pascal Thubert (pthubert) <[email-masked]> wrote:
    mcr> So I think that the errata against 6550 is that it should be a lollipop
    mcr> counter, and I think that this solves the reboot problem, and eliminates
    mcr> having to write it to flash.

    rj> [RJ] this still does not solve the reboot issue... The text in the
    rj> draft actually considers dtsn to be a lollipop counter and explains
    rj> the problem when rebooting within the sequence window of lollipop
    rj> counter. Please refer to section 7.2 of the draft for example. Errata
    rj> is nonetheless needed for dtsn to be a lollipop counter.

    pt> Well, yes, but that’s true for any of the sequence counters, not just
    pt> DTSN, and was not perceived as an “issue”. 
    pt> The lollipop by design detects one reboot during the straight
    pt> part. More will cause a delay. There are ways to avoid that like
    pt> persistent memory or alleviate it like making the straight part
    pt> shorted on reliable links.  I’m not sure there’s much to add to that
    pt> or to fix, is there?

Pascal, I feel that Rahul has some point that we've missed.

>   behalf of node A).  As per section 7.2. of [RFC6550], when node B
>   receives this packet it will consider the information to be old
>   (since 240 < 250).

>   Based on this figure, there is dead zone (240 to 0) in which if A
>   operates after reboot then the seqcnt will always be considered
>   smaller.  Thus node A needs to maintain the seqcnt in persistent
>   storage and reuse this on reboot.

My instinct is there are one of two bugs (maybe both documentation and code):
   a) node A SHOULD rapidly move from 240 to 0, that the numbers from 240 to
      0 should be considered "setup", and node A was wrong to advance only
      to 250, and not as soon as it felt "stable", move to 0.

   b) that node B, should recognize that during the 240 to 0 stage, that
      the counters should not be compared.  Anything < 0 is "setup", and is
      always "new", but that 251 > 250 require immediate action.

I found it unusual that the lollipop should require 16 values in the stem, I
think that one ('0') would have worked, but maybe there was some reason to
prefer 16.

---

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Sat, 12 May 2018 19:59:06 +0000

Hello Michael 

The default of 16 points in the straight line is there to cover multiple losses in a row; 16 may be too large for a reliable Mac like TSCH in which case a node may start at another value, e.g., 250. Starting at 255 looks optimistic though. In case of a loss numbers will move to the circular part and will be uncomparable.

It is not really that the node is stable that counts but that all other nodes got a chance to see that this node as rebooted so they can reset their reference value and keep numbers comparable.

I’m happy to pursue this discussion to see more clearly than I do now; at this point I still do not see what change we could make - and why - to the sequence counter behavior...

Take care,

Pascal

> Le 12 mai 2018 à 20:58, Michael Richardson <[email-masked]> a écrit :
> 
>

---

From: Michael Richardson <[email-masked]>
Date: Sun, 13 May 2018 12:55:02 -0400

Pascal Thubert (pthubert) <[email-masked]> wrote:
    > The default of 16 points in the straight line is there to cover
    > multiple losses in a row; 16 may be too large for a reliable Mac like
    > TSCH in which case a node may start at another value, e.g.,
    > 250. Starting at 255 looks optimistic though. In case of a loss numbers
    > will move to the circular part and will be uncomparable.

okay, so with TSCH it would make sense to start a higher number?

    > It is not really that the node is stable that counts but that all other
    > nodes got a chance to see that this node as rebooted so they can reset
    > their reference value and keep numbers comparable.

Yes, agreed.
The goal is to have a way for children to see that the node is freshly
rebooted and to reset their threshold.

    > I’m happy to pursue this discussion to see more clearly than I do now;
    > at this point I still do not see what change we could make - and why -
    > to the sequence counter behavior... 

Maybe there is no change, but you claim the DTSN does need to be written to
flash, and Rahul says that it must be.   (Correct me if my understanding is wrong).
A protocol which does not require flash writes is clearly better.

---

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Mon, 14 May 2018 07:43:36 +0000

Hello Michael:

> Sent: dimanche 13 mai 2018 18:55

[PT>] Yes, I think so. It also depends on the chances to reboot again soon.

> The goal is to have a way for children to see that the node is freshly rebooted
> and to reset their threshold.
> 
[PT>] There is DTSN and there are all the other numbers. The hints I'm giving above are for general sequence counters in RPL.
DTSN is in fact simpler since it is not propagated by the protocol over multiple hops. A node that talks one-to-one with another will hopefully not receive packets out of order - unless packets can mix up in the transmitter queues? So literally any change in the DTSN could be seen as a new DTSN, regardless of whether the new value is higher or lower. We may improve the text in RPL to indicate that; Not sure we really need to list it in section 7 after all. Maybe we need more thoughts/reread of the spec, but it feels like DTSN can be started at boot at any random value and any change in DTSN is a pull.

Cheers,

Pascal

---

From: Rahul Arvind Jadhav <[email-masked]>
Date: Mon, 14 May 2018 15:43:24 +0000

Please find my comment inline...

-----Original Message-----
From: Roll [mailto:[email-masked]] On Behalf Of Pascal Thubert (pthubert)
Sent: 14 May 2018 13:14
To: Routing Over Low power and Lossy networks <[email-masked]>
Subject: Re: [Roll] draft-rahul-roll-rpl-observations-00 Section 2.1: Wear leveling

[RJ] Any change in DTSN is a pull contradicts lollipop behavior. So isn’t dtsn a lollipop counter unlike our previous discussion? Also in another scenario, if the chosen random value turns out to be the same at which the node previously rebooted, then still it is an issue. 

Cheers,

Pascal

---

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Mon, 14 May 2018 16:24:44 +0000

Hello Rahul:

I think it is clear that there are corner cases when not using a persistent storage, and that the problem is not a dead lock but something that solves itself sooner or later. This leaves room for different device cost for different usages.

I agree we can reread the whole DTSN text and improve it. I'll need to do that before agreeing on any proposed update.

To answer your questions: Looking at it again, I'm not sure we need to declare it a lollipop counter. It was not listed as such but I fail to remember why. Maybe it was effectively because it only needs to change to trigger the pull. If it is picked at random and uses the same value as before, the first retry will be a change and from there you go.

All the best,

Pascal

> From: Roll <[email-masked]> On Behalf Of Rahul Arvind Jadhav
> Sent: lundi 14 mai 2018 17:43
> [PT>] There is DTSN and there are all the other numbers. The hints I'm giving
> above are for general sequence counters in RPL.
> DTSN is in fact simpler since it is not propagated by the protocol over multiple
> hops. A node that talks one-to-one with another will hopefully not receive
> packets out of order - unless packets can mix up in the transmitter queues? So
> literally any change in the DTSN could be seen as a new DTSN, regardless of
> whether the new value is higher or lower. We may improve the text in RPL to
> indicate that; Not sure we really need to list it in section 7 after all. Maybe we
> need more thoughts/reread of the spec, but it feels like DTSN can be started
> at boot at any random value and any change in DTSN is a pull.
> 
> [RJ] Any change in DTSN is a pull contradicts lollipop behavior. So isn’t dtsn a
> lollipop counter unlike our previous discussion? Also in another scenario, if
> the chosen random value turns out to be the same at which the node
> previously rebooted, then still it is an issue.
> 
> Cheers,
> 
> Pascal
>