Subject: FreeBSD 7.0BETA2 freezes when using SCTP and IPSec

From: Oliver Roll <[email-masked]>
Date: Wed, 14 Nov 2007 17:37:04 +0100

Dear all,

I'm experiencing problems when sending SCTP-traffic between two hosts
over a transport-mode IPSec-connection on FreeBSD 7.0BETA2.
The setup is very low-level: a listening server and a client sending one
line of text to the server; connection is secured with setkey, using
hmac-md5 for AH and triple DES for ESP.
On my 7.0Beta2 traffic which is only authenticated with AH is
trouble-free. Once using ESP to encrypt the traffic the sending system
(client) freezes. A "panic: double fault" is displayed on the client's
console and the core is dumped. Error message:

Fatal double fault:
eip = 0xc07a4912
esp = 0xcbd44ffc
ebp = 0xcbd45018

A strange issue is that only the sending side is affected. If only the
server->client traffic (INIT_ACK, SACK etc.) is encrypted, both machines
are still alive after transferring data.
My setup works well on FreeBSD 6.1 machines without any freezes.

Can anybody please help me?

By the way: the reason for using 7.0BETA2 was that I was not able to
compile a kernel on 6.2 machines with both SCTP and IPSEC activated in
the kernel (using a sunday-night cvs-checkout from
stewart.chicago.il.us/sctpCVS). Kernelbuild failed with error message:

in file included from /usr/src/sys/netipsec/ipsec.h:46,
from /usr/src/sys/netinet/sctp_os_bsd.h:78,
from /usr/src/sys/netinet/sctp_os.h:59,
from /usr/src/sys/netinet/in_proto.c:92:
/usr/src/sys/netipsec/keydb.h:54 error: redefinition of 'struct secasindex'

As far as I can evaluate this there is a conflict between ipsec and
netkey but I was not able to solve this conflig. Maybe this problem is
known already?

Thank you, best regards,

Oliver

---

From: Michael Tuexen <[email-masked]>
Date: Wed, 14 Nov 2007 20:07:27 +0100

Hi Oliver,

do you have a core and provide a stack backtrace such
that we can see at least where the crash happens?

Best regards
Michael

---

From: Oliver Roll <[email-masked]>
Date: Wed, 14 Nov 2007 21:16:27 +0100

Hello Michael,

Yes, I have a dumped kernel and this the trace:

##########
[root@freebsd70-client /usr/obj/usr/src/sys/GENERIC]# kgdb kernel.debug
/var/crash/vmcore.0
[... snipped GDB copyright notice ...]

Unread portion of the kernel message buffer:

Fatal double fault:
eip = 0xc07a4ba2
esp = 0xcbd3effc
ebp = 0xcbd3f018
cpuid = 0; apic id = 00
panic: double fault
cpuid = 0
Uptime: 3m47s
Physical memory: 243 MB
Dumping 31 MB: 16

#0  doadump () at pcpu.h:195
195             __asm __volatile("movl %%fs:0,%0" : "=r" (td));
(kgdb) list *0xc07a4ba2
0xc07a4ba2 is in m_copydata (/usr/src/sys/kern/uipc_mbuf.c:816).
811                     m = m->m_next;
812             }
813             while (len > 0) {
814                     KASSERT(m != NULL, ("m_copydata, length > size
of mbuf chain"));
815                     count = min(m->m_len - off, len);
816                     bcopy(mtod(m, caddr_t) + off, cp, count);
817                     len -= count;
818                     cp += count;
819                     off = 0;
820                     m = m->m_next;
(kgdb)

##########

Please let me know if you need more data.

Best regards,

Oliver

---

From: Michael Tuexen <[email-masked]>
Date: Wed, 14 Nov 2007 22:07:14 +0100

Hi Oliver,

what is the result of

where

Best regards
Michael

---

From: Oliver Roll <[email-masked]>
Date: Wed, 14 Nov 2007 22:31:18 +0100

Hi Michael,

(kgdb) list *0xc07a4ba2
0xc07a4ba2 is in m_copydata (/usr/src/sys/kern/uipc_mbuf.c:816).
811                     m = m->m_next;
812             }
813             while (len > 0) {
814                     KASSERT(m != NULL, ("m_copydata, length > size
of mbuf chain"));
815                     count = min(m->m_len - off, len);
816                     bcopy(mtod(m, caddr_t) + off, cp, count);
817                     len -= count;
818                     cp += count;
819                     off = 0;
820                     m = m->m_next;
(kgdb) where
#0  doadump () at pcpu.h:195
#1  0xc07595a7 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:409
#2  0xc0759869 in panic (fmt=Variable "fmt" is not available.
) at /usr/src/sys/kern/kern_shutdown.c:563
#3  0xc0a51f4b in dblfault_handler () at /usr/src/sys/i386/i386/trap.c:901
#4  0x00000000 in ?? ()
(kgdb)

Best regards,

Oliver

---

From: Michael Tuexen <[email-masked]>
Date: Wed, 14 Nov 2007 22:52:07 +0100

Hi Oliver,

not very useful... Maybe Peter or Randy have an idea...
I'm mainly using/debugging on Mac OS X...

Best regards
Michael

On Nov 14, 2007, at 10:31 PM, Oliver Roll wrote:

>>>>> The setup is very low-level: a listening server and a client  
>>>>> sending
>>>>> one

---

From: Randall Stewart <[email-masked]>
Date: Thu, 15 Nov 2007 08:21:23 -0500

Oliver/Michael:

Yes I have an idea...

This looks like a stack blow... i.e. you only get
2 pages of stack.. and I bet the combination of
SCTP + IPSEC pushes us over that edge.

I had not looked into the 6.x problem with IPSEC since I know
a lot of this was caused by the introduction in 7.x of a
new IPSEC stack .. from gnn.. the code does the 7.x ipsec
stuff (whatever gnn put in) but not the old 6.x kame stack.

I will try to look in to how much stack we are using on
the send side.. and see if I can find a way to
reduce that... not sure how much ipsec uses... I
do know that the compiler change caused us a similar
problem before that was fixed by adding some non-inline
declarations.. this may be the problem.. its all
getting glombed together in a big function that pulls
to much stack..

But let me go dig into how much we actually use
on the sending side..

R

-- 
Randall Stewart
NSSTG - Cisco Systems Inc.
803-345-0369 <or> 803-317-4952 (cell)