Subject: Path mtu discovery

From: Sebastian Rohde <[email-masked]>
Date: 03 Dec 2003 13:54:00 +0100

Hi,

I am currently working on the sctplib to implement path mtu discovery.

Unfortunately I have a problem with that ;) 
Please let me give you an example of what i am thinking about:
Let's assume somebody sends a large Message with the DF-Bit set - After
that but before we receive the icmp frag needed we send a Message
containing a data chunk which is - along with all headers - the size of
the current known path mtu.
When this message is lost due to a lower path mtu and a frag needed is
received the path mtu for this path will be adjusted and all further
data chunks will be bundled to fit within this new mtu.
The problem is now that we already have bundled at least one data chunk
at a size that exceeds the current path mtu. I don't think that
rebundling would be an option as we would run into trouble with the
continuity of tsns. 
On the other hand at least the linux kernel does not allow to send a
larger sized packet as he remembers from the new path mtu and returns an
error as long as path mtu discovery is enabled.
An Option would be to retransmit the packets which are too big without
the df bit set and accept that this message will be fragmented.

How do others handle this problem?
 
Another question related to this would be whether to send all packets
with df set or just selected packets. If doing the latter with ipv6 the
linux kernel does not allow(as least as far as i know) to send messages
greater than the minimum link mtu 1280 which is really undesirable. 

I am currently thinking about a solution which sends every packet with
df bit set and only if I have to send a packet larger than the current
path mtu(resulting from the scenario above) I will disable the pmtud and
let the linux kernel fragment the packet into his know path mtu size or
for ipv6 in 1280 sized packets.

I am not working very long with these internals of sctp and linux so
please excuse and correct me if i got something wrong ;-).

Regards,
        Sebastian Rohde

---

From: Michael Tuexen <[email-masked]>
Date: Wed, 3 Dec 2003 22:29:36 +0100

Sebastian,

see my comments below.

Best regards
Michael

> data chunks will be bundled to fit within this new mtu.
I'm not sure what you mean by bundling here. You can change the bundling
with each packet you send. The problem is if you send a packet 
consisting
of a common header and one large DATA chunk only. And now your path MTU
shrinks. You have only two choices:
1. Send the packet again and do not set the DF bit and make use if IP 
fragmentation.
2. Use PR-SCTP to tell the peer to drop the packet and segment the
    user data into more DATA chunks and send this again. This might not
    be possible because your peer does not support PR-SCTP or the upper 
layer
    can not handle the possible reordering of the user data.

> error as long as path mtu discovery is enabled.
I do not see the problem here. You should not set this option because
the sctplib runs on multiple platforms and you should not rely on a
feature only available on one platform. The sctplib should handle
path MTU discovery by itself only by having a raw IP socket for ICMP
and having a callback for the ICMP messages.
> for ipv6 in 1280 sized packets.
i would strongly suggest not to make use of any feature in the linux 
kernel.
Just use a raw socket to receive ICMP packets. This works on all 
platforms.

---

From: "Randall Stewart (cisco)" <[email-masked]>
Date: Wed, 03 Dec 2003 17:31:55 -0600

Michael:

I posted earlier.. It did not go to the list due
to an error in the filtering script which Don as now fixed...

Michael Tuexen wrote:

> with each packet you send. The problem is if you send a packet consisting

The above is the standard way you handle this in SCTP or TCP... KAME does
it this way...

>    can not handle the possible reordering of the user data.

unless of course the packet is PR-SCTP .. and both sides support it as you
illustrate above :->

> and having a callback for the ICMP messages.

We had this working in the original ref-impl that Q and I did.. we had
both the raw IP socket and a raw ICMP socket.. you have to
look at a few more packets and only pay attention to the SCTP
ones.. but it should work fine :->

R

-- 
Randall R. Stewart
ITD
Cisco Systems Inc.
[email-masked] 815-342-5222(cell) or 815-477-2127(office)

---

From: Sebastian Rohde <[email-masked]>
Date: 08 Dec 2003 11:17:28 +0100

Thank you for your answers. I was out of office on thursday and friday
so I was unable answer these days.
Please see my comments below.

> > 1. Send the packet again and do not set the DF bit and make use if IP 
This was what i meant ;)

> >    can not handle the possible reordering of the user data.
I was thinking about that but as you said - we would not be anymore
ordered in this case.
> >> error as long as path mtu discovery is enabled.
As far as I know there is no option to just set the df bit in the linux
kernel - You can only enable PMTU Discovery or disable it. Normally this
is no problem but when using ipv6 this becomes a small problem in a
scenario as stated below.
> ones.. but it should work fine :->
This is the way I was handling it(And I will go on handling it this way
;)) before i discovered the problem and searched for good alternatives.
> > Just use a raw socket to receive ICMP packets. This works on all 
The point here was that with ipv6 the linux kernel works in some
circumstances against the own implementation ;-(. Originally we planned
not to send every packet with the df bit set but only one large packet
in a given timeframe to allow a better handling of firewalls which block
all icmp traffic - The linux kernel does in fact work against this with
ipv6 as it automatically fragments every packet which is larger than
1280 when the IPV6_PMTUDISC_DONT sockoption is set.

I think for ipv6 I will send every packet with the df bit set and if a
packet gets to big ipv6 I will switch that off...

Sebastian

---

From: Michael Tuexen <[email-masked]>
Date: Mon, 8 Dec 2003 12:11:11 +0100

Sebastian,

see my comments below.

Best regards
Michael

> I was thinking about that but as you said - we would not be anymore
This is not a general solution.
> scenario as stated below.
You can use a raw IP socket an set it by yourself. Using the IP_HDRINCL
option you have to provide the complete IP-header by your own. So 
setting
the bit is no problem. And it works on all platforms currently 
supported by
the sctplib.
> 1280 when the IPV6_PMTUDISC_DONT sockoption is set.
Please keep in mind that the sctplib is not Linux only. You should 
provide
a solution for all OSes. You can use an optimized version for a 
particular
OS but I would prefer a more general solution. This is and will always 
be
a prototype implementation and any special handling on a per OS basis 
has to
be tested separately.

---

From: "Randall Stewart (cisco)" <[email-masked]>
Date: Mon, 08 Dec 2003 15:14:58 -0600

>packet gets to big ipv6 I will switch that off...
Sebastian:

IPv6 does NOT have a DF bit. What happens is the IPv6 layer
will fragment for you.. you just need to pay attention to path
mtu changes and set your MTU accordingly...

R