Subject: Re: [conex] WGLC for draft-ietf-conex-abstract-mech-06.txt

From: Bob Briscoe <[email-masked]>
Date: Mon, 10 Dec 2012 21:52:12 +0000

Ingemar,

You're right. Good point. Certainly true for downstream cases out of 
the cellular network, where long-running flows are not uncommon.

When I wrote that, I wasn't thinking of mobility, I was thinking of 
much less frequent re-routing after failures, so I wasn't trying to 
make it seamless.

For explicit hand-overs as in the case you describe, there's explicit 
hand-over of operational state, so it wouldn't be impossible to 
hand-over credit too. Am I right, at least in an arm-wavy sense?

We should make it clear that the only state that re-creates itself is 
the flow-ID state, not the credit associated with it. I wrote that 
sentence (not Matt), and my motivation was to explain that a switch 
to a different audit wouldn't be catastrophic for the flow. I wasn't 
trying to claim that it would hand-over without a glitch. The audit 
would drop some packets, so the flow would send some Re-Echo-Loss, 
which would establish some credit and the flow could continue.

Bob

Sorry for delay replying.

At 13:27 21/11/2012, Ingemar Johansson S wrote:
>Hi
>
>First time I had time to read carefully through this document
>The document is comprehensible and I don't find any serious issues with it.
>
>One comment though on page 5, 3rd para.
>" the flow-state required for audit creates itself as it detects new 
>flows.  Therefore a flow will not fail if it is re-
>    routed away from the audit box currently holding its flow-state. "
>
>If I map ConEx to a 3GPP LTE use case, the audit functions are best 
>placed in the eNodeB. When a new (long lived) flow is created it is 
>preferably preloaded with credit marks which are stored in the 
>auditor in the eNodeB which the terminal is "connected" to. When the 
>terminal hands over to another eNodeB the credit marks will be 
>lost.  This means that the ConEx markings will in this case be at 
>least one RTT behind with a higher risk of false positives in the 
>auditor in the new eNodeB
>To avoid this the credit marks would need to be forwarded to the new 
>eNodeB via e.g the X2 interface.
>
>So my question is, is it needed to add a statement that mentions this ?
>
>/Ingemar
>
>-----Original Message-----
>From: [email-masked] [mailto:[email-masked]] On 
>Behalf Of marcelo bagnulo braun
>Sent: 23 October 2012 06:54
>To: 'ConEx IETF list'
>Subject: [conex] WGLC for draft-ietf-conex-abstract-mech-06.txt
>
>Hi,
>
>This note issues the WGLC for
>
>draft-ietf-conex-abstract-mech-06.txt
>
>Please reivew the document and provide comments. The WGLC will close 
>on the 20th of november.
>
>For you convenience, the draft can be found at:
>
>https://datatracker.ietf.org/doc/draft-ietf-conex-abstract-mech
>
>Regards, marcelo
>
>_______________________________________________
>conex mailing list
>[email-masked]
>https://www.ietf.org/mailman/listinfo/conex
>
>_______________________________________________
>conex mailing list
>[email-masked]
>https://www.ietf.org/mailman/listinfo/conex

________________________________________________________________
Bob Briscoe,                                BT Innovate & Design

---

From: Ingemar Johansson S <[email-masked]>
Date: Thu, 13 Dec 2012 09:46:51 +0000

Hi

Comments inline

/Ingemar

> From: Bob Briscoe [mailto:[email-masked]]
> Sent: den 10 december 2012 22:52
> To: Ingemar Johansson S
> Cc: [email-masked]; Ingemar Johansson S
> Subject: Re: [conex] WGLC for draft-ietf-conex-abstract-mech-06.txt
> 
> Ingemar,
> 
> You're right. Good point. Certainly true for downstream cases out of the
> cellular network, where long-running flows are not uncommon.
[IJ] Also, novel features like HTTP 2.0 and SPDY may make persistent TCP connections more common.

> 
> When I wrote that, I wasn't thinking of mobility, I was thinking of much less
> frequent re-routing after failures, so I wasn't trying to make it seamless.
> 
> For explicit hand-overs as in the case you describe, there's explicit hand-over
> of operational state, so it wouldn't be impossible to hand-over credit too. Am
> I right, at least in an arm-wavy sense?
[IJ] Yes, one option is to transfer the credit via the X2 interface, needs standardization of course (not in IETF though)

> 
> We should make it clear that the only state that re-creates itself is the flow-
> ID state, not the credit associated with it. I wrote that sentence (not Matt),
> and my motivation was to explain that a switch to a different audit wouldn't
> be catastrophic for the flow. I wasn't trying to claim that it would hand-over
> without a glitch. The audit would drop some packets, so the flow would send
> some Re-Echo-Loss, which would establish some credit and the flow could
> continue.
[IJ] Hmm, perhaps I did not get that part, what you say is that if the audit drops a packet, that will count as a credit ?.  This credit thing is is still a mystery so I probably need to get this explained better...

> >First time I had time to read carefully through this document The
> >document is comprehensible and I don't find any serious issues with it.
> >
> >preferably preloaded with credit marks which are stored in the auditor
> >in the eNodeB which the terminal is "connected" to. When the terminal
> >hands over to another eNodeB the credit marks will be lost.  This means
> >that the ConEx markings will in this case be at least one RTT behind
> >with a higher risk of false positives in the auditor in the new eNodeB
> >From: [email-masked] [mailto:[email-masked]] On Behalf
> >Of marcelo bagnulo braun
> on
> >the 20th of november.

---

From: Bob Briscoe <[email-masked]>
Date: Thu, 13 Dec 2012 13:24:57 +0000

Ingemar,

At 09:46 13/12/2012, Ingemar Johansson S wrote:
>[IJ] Hmm, perhaps I did not get that part, what you say is that if 
>the audit drops a packet, that will count as a credit ?.  This 
>credit thing is is still a mystery so I probably need to get this 
>explained better...

OK, I think I promised Marcelo I would write a draft about audit, 
with a couple of example algorithms. I suspect that would help clear 
up many of these questions.

For now, example stateful and stateless audit pseudocode and 
evaluations are here:
<http://www.bobbriscoe.net/pubs.html#refb-dis> (Section 7.6)

Here's an overview:

Very brief (for those who mostly get it already):
----------
The audit function doesn't count its own losses when balancing 
Re-Echo-Loss against Loss, because they are /audit/ losses, not 
congestion losses.

So, if a flow is re-routed to a new audit function that doesn't hold 
any state about this flow, it will start discarding  a low level of 
packets (audit discard). This will trigger the sender to Re-Echo-Loss 
in the next round trip, which will make the flow's audit balance 
positive and stop further audit discards.

So, the flow has naturally created new state at the new audit 
function without the flow explicitly knowing that the burst of loss 
was due to a re-route into a new audit function which caused the 
flow's credit state at the original audit function to be lost.

Brief but fuller explanation:
----------------------------
Re-Echo is always at least a round trip behind (longer for e.g. 
RTCP). The audit function makes the sender responsible for making the 
allowance for its feedback delay, by having to mark sufficient credit 
packets. Then the audit function can just test instantaneous 
negativity, without having to allow for RTT or feedback delays that 
it doesn't know.

I'll assume a stateful audit function (stateless is possible, but it 
has to be less precise). For the duration of a flow, a stateful audit 
function counts bytes of different ConEx markings, and it will start 
dropping packets if
* the balance of (Credit + Re-Echo-ECN  - ECN)  goes negative, or
* the balance of (Credit + Re-Echo-Loss - Loss) goes negative. {See note}

Consider a re-route where a flow is switched to a different stateful 
audit function, and assume the audit functions don't hand-over their state.

* As soon as the flow suffers any loss or ECN in the network, the 
flow will go negative at the new audit function for at least a RTT, 
because it has no credit and no Re-Echo for at least a round trip.
* So audit starts dropping packets (a very small but growing proportion).
* Importantly, the audit doesn't count these discards as loss because 
they are its own /audit/ losses not congestion losses.
* The sender gets the loss feedback from the receiver and sends some 
Re-Echo-Loss markings to balance.
* These make the flow positive at the new audit function, because it 
didn't count its own discards.

So, the flow has naturally created new state at the new audit 
function without explicitly knowing that the burst of loss was due to 
a re-route.

Cheers

Bob

{Note: Let's set aside the question of how an audit function knows 
how much loss an upstream node has done - abstract-mech talks about 
that separately.}

---

From: Ingemar Johansson S <[email-masked]>
Date: Thu, 13 Dec 2012 14:20:46 +0000

Hi

Thanks for the explanation now I get it!

/Ingemar

> Sent: den 13 december 2012 14:25
> To: Ingemar Johansson S
> Cc: [email-masked]
> Subject: RE: [conex] WGLC for draft-ietf-conex-abstract-mech-06.txt
> 
> Ingemar,
> 
> > > without a glitch. The audit would drop some packets, so the flow
> > > would send some Re-Echo-Loss, which would establish some credit and
> > > the flow could continue.
> >[IJ] Hmm, perhaps I did not get that part, what you say is that if the
> >audit drops a packet, that will count as a credit ?.  This credit thing
> >is is still a mystery so I probably need to get this explained
> >better...
> 
> OK, I think I promised Marcelo I would write a draft about audit, with a
> couple of example algorithms. I suspect that would help clear up many of
> these questions.
> 
> For now, example stateful and stateless audit pseudocode and evaluations
> are here:
> The audit function doesn't count its own losses when balancing Re-Echo-Loss
> against Loss, because they are /audit/ losses, not congestion losses.
> 
> So, if a flow is re-routed to a new audit function that doesn't hold any state
> about this flow, it will start discarding  a low level of packets (audit discard).
> This will trigger the sender to Re-Echo-Loss in the next round trip, which will
> make the flow's audit balance positive and stop further audit discards.
> 
> So, the flow has naturally created new state at the new audit function
> without the flow explicitly knowing that the burst of loss was due to a re-
> route into a new audit function which caused the flow's credit state at the
> original audit function to be lost.
> 
> 
> allowance for its feedback delay, by having to mark sufficient credit packets.
> Then the audit function can just test instantaneous negativity, without
> having to allow for RTT or feedback delays that it doesn't know.
> 
> I'll assume a stateful audit function (stateless is possible, but it has to be less
> precise). For the duration of a flow, a stateful audit function counts bytes of
> different ConEx markings, and it will start dropping packets if
> Consider a re-route where a flow is switched to a different stateful audit
> function, and assume the audit functions don't hand-over their state.
> 
> * As soon as the flow suffers any loss or ECN in the network, the flow will go
> negative at the new audit function for at least a RTT, because it has no credit
> and no Re-Echo for at least a round trip.
> * Importantly, the audit doesn't count these discards as loss because they
> are its own /audit/ losses not congestion losses.
> * The sender gets the loss feedback from the receiver and sends some Re-
> Echo-Loss markings to balance.
> * These make the flow positive at the new audit function, because it didn't
> count its own discards.
> 
> So, the flow has naturally created new state at the new audit function
> without explicitly knowing that the burst of loss was due to a re-route.
> 
> 
> Cheers
> 
> 
> Bob
> 
> {Note: Let's set aside the question of how an audit function knows how
> much loss an upstream node has done - abstract-mech talks about that
> separately.}
> 
>