Hi,

We have SCTP client program (C++) running under RHEL4 (using kerne= l version: 2.6.9-55ELsmp and lksctp-tools-1.0.8-1). We are using one-to-one= style socket using connect () system call to setup an association with a S= CTP server (i.e., SUT of vendor). Then it calls sctp_sendmsg () library fun= ction to send a message from a socket while using the following advanced fe= atures of SCTP (i.e., SCTP_UNORDERED for un-ordered delivery of the message= ) and timetolive (TTL) is set to 1000 milliseconds for a given message. We have typical= defaults values for these RTO.Initial =3D 3000, RTO.Min =3D 1000, RTO.Max = =3D 60000 milliseconds. T= he PR-SCTP extension is enabled in the kernel (by default).

We are using lot (4000) SCTP associations on a single system, all = are in ESTABLISHED state and data communication goes on fine without any dr= op/loss of message. But when the peer (i.e., SCTP server) gets restarted, <= o:p>

a) All the SCTP clients receive INIT messages (with same a= ddress to the association) from SCTP server. The INIT ACK is being sent wit= h the new InitTag and Verification Tag set to the InitTag received in INIT = for each association.

b) The four-way handshake gets completed. I believe the e= xisting association, including its current state, and the corresponding TCB= does not get changed.

c) Afterward, all SCTP clients use DATA chunks to exchange= information with SCTP server.

The trouble is that, after sometime (30-45 minutes), DATA chunks s= tart to get dropped or lost and that goes on increasing as the time advance= s. The dat= a was not transmitted to the peer at least once. Note that, the time after = which this problem is being encountered is also not consistent. I mean to s= ay, sometime we noticed after 30 minutes, 45 minutes and an hour. But when = we lessen the number of SCTP associations to 2000, such problem does not ap= pear.

a) = ; Is there any chance that, that data mess= ages time-out within the RTO although both the values are same? As a result= , the data will be skipped and no longer transmitted.

b) = ; Does it indicate that the message could = never be transmitted to the peer (e.g., flow control prevented the message = from being sent before its lifetime expired), so the peer never received it= ? Please note that, we are using 1 Gig Intel NIC.

The preliminary ana= lysis says that, timetolive (in sctp_sendmsg() API) can be set on a message= independent of Partial Reliability. The timer (i.e., sinfo_ti= metolive) only runs while the message is in the kernel send buffer and has = not yet been put on the wire (for the first time). As soon as the message i= s put on the wire the first time, the timer is dropped. It is used to indic= ate that a time based lifetime is being applied to the data. It is th= en a number of milliseconds for which the data is attempted to be transmitt= ed. If that many milliseconds elapsed and the data has not been transmitted= , As a result, they move the abandoned list and are never retransmitted. Ple= ase feel free to correct me if I am wrong.

Should we try w= ith 0 ms timetolive which indicates no timeout should occur on this message= and is treated as reliable? Please suggest and also let me know if you nee= d any additional information.

Thanking you in= advance for your response.

Regards,

Chinmaya

= ---725201809-900650471-1338489317=:38384-- From cbenson@adax.com Thu May 31 11:43:15 2012 Return-Path: X-Original-To: tsvwg@ietfa.amsl.com Delivered-To: tsvwg@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 094F721F864C for ; Thu, 31 May 2012 11:43:15 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.147 X-Spam-Level: X-Spam-Status: No, score=-2.147 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, MIME_8BIT_HEADER=0.3, SARE_SUB_ENC_UTF8=0.152] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id URkdT8EzvKBY for ; Thu, 31 May 2012 11:43:14 -0700 (PDT) Received: from mail1.adax.com (mail1.adax.com [70.36.226.176]) by ietfa.amsl.com (Postfix) with ESMTP id 54EE421F8643 for ; Thu, 31 May 2012 11:43:14 -0700 (PDT) Received: from adax (adax [12.0.0.88]) by mail1.adax.com (Postfix) with ESMTP id C80AC1E10FF; Thu, 31 May 2012 11:43:13 -0700 (PDT) Received: by adax (Postfix, from userid 243) id B78608EE6A; Thu, 31 May 2012 11:43:13 -0700 (PDT) Received: from localhost (localhost [127.0.0.1]) by adax (Postfix) with ESMTP id B02498EE69; Thu, 31 May 2012 11:43:13 -0700 (PDT) Date: Thu, 31 May 2012 11:43:13 -0700 (PDT) From: Chris Benson X-X-Sender: cbenson@adax.adax To: Chinmaya Dwibedy In-Reply-To: <1338489317.38384.YahooMailNeo@web130103.mail.mud.yahoo.com> Message-ID: References: <4FBE562E.6000909@gmail.com> <1338463399.19395.YahooMailNeo@web130103.mail.mud.yahoo.com> <1338467385.57530.YahooMailNeo@web130105.mail.mud.yahoo.com> <1338489317.38384.YahooMailNeo@web130103.mail.mud.yahoo.com> MIME-Version: 1.0 Content-Type: MULTIPART/MIXED; BOUNDARY="808960-2099520786-1338489793=:23407" Cc: "tsvwg@ietf.org" Subject: Re: [tsvwg] =?utf-8?q?When_SCTP_server_is_restarted=2C_DATA_chunks_st?= =?utf-8?q?art_to_get_dropped_or_lost_after_sometime_at_SCTP_client?= =?utf-8?b?4oCZcyBlbmQu?= X-BeenThere: tsvwg@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Transport Area Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 May 2012 18:43:15 -0000 This message is in MIME format. The first part should be readable text, while the remaining parts are likely unreadable without MIME-aware tools. --808960-2099520786-1338489793=:23407 Content-Type: TEXT/PLAIN; charset=UTF-8 Content-Transfer-Encoding: QUOTED-PRINTABLE Chinmaya, Please forgive me if this is too basic and obvious a question. Are you sure that your "application" (SCTP-user) at the client=20 side is consuming the incoming data AT LEAST AS FAST as it arrives? If not, then you will see something like the behaviour you=20 describe. With thanks, from Chris On Thu, 31 May 2012, Chinmaya Dwibedy wrote: >> Date: Thu, 31 May 2012 11:35:17 -0700 (PDT) >> From: Chinmaya Dwibedy >> To: "tsvwg@ietf.org" >> Subject: [tsvwg] When SCTP server is restarted, >> DATA chunks start to get dropped or lost after sometime at SCTP cli= ent >> end. >> =20 >> Hi,=20 We have SCTP client program (C++) running under RHEL4 (using kernel version= : 2.6.9-55ELsmp and lksctp-tools-1.0.8-1). We are using one-to-one style so= cket using connect () system call to setup an association with a SCTP serve= r (i.e., SUT of vendor). Then it calls sctp_sendmsg () library function to = send a message from a socket while using the following advanced features of= SCTP (i.e., SCTP_UNORDERED for un-ordered delivery of the message) and tim= etolive (TTL) is set to 1000 milliseconds for a given message. We have typi= cal defaults values for these RTO.Initial =3D 3000, RTO.Min =3D 1000, RTO.M= ax =3D 60000 milliseconds.The PR-SCTP extension is enabled in the kernel (b= y default). We are using lot (4000) SCTP associations on a single system, all are in ES= TABLISHED state and data communication goes on fine without any drop/loss o= f message. But when the peer (i.e., SCTP server) gets restarted,=20 a)=C2=A0=C2=A0=C2=A0 All the SCTP clients receive INIT messages (with same = address to the association) from SCTP server. The INIT ACK is being sent wi= th the new InitTag and Verification Tag set to the InitTag received in INIT= for each association.=20 b)=C2=A0=C2=A0=C2=A0 The four-way handshake gets completed. I believe the e= xisting association, including its current state, and the corresponding TCB= does not get changed.=20 c)=C2=A0=C2=A0=C2=A0 Afterward, all SCTP clients use DATA chunks to exchang= e information with SCTP server. The trouble is that, after sometime (30-45 minutes), DATA chunks start to g= et dropped or lost and that goes on increasing as the time advances.The dat= a was not transmitted to the peer at least once. Note that, the time after = which this problem is being encountered is also not consistent. I mean to s= ay, sometime we noticed after 30 minutes, 45 minutes and an hour. But when = we lessen the number of SCTP associations to 2000, such problem does not ap= pear. =C2=A0=C2=A0=C2=A0 a)=C2=A0=C2=A0=C2=A0 Is there any chance that, that data messages time-out = within the RTO although both the values are same? As a result, the data wil= l be skipped and no longer transmitted. b)=C2=A0=C2=A0=C2=A0 Does it indicate that the message could never be trans= mitted to the peer (e.g., flow control prevented the message from being sen= t before its lifetime expired), so the peer never received it? Please note = that, we are using 1 Gig Intel NIC.=20 The preliminary =C2=A0analysis says that, timetolive (in sctp_sendmsg() API= ) can be set on a message independent of Partial Reliability.The timer (i.e= =2E, sinfo_timetolive) only runs while the message is in the kernel send bu= ffer and has not yet been put on the wire (for the first time). As soon as = the message is put on the wire the first time, the timer is dropped. It is = used to indicate that a time based lifetime is being applied to the data.= =C2=A0 It is then a number of milliseconds for which the data is attempted = to be transmitted. If that many milliseconds elapsed and the data has not b= een transmitted, As a result, they move the abandoned list and are never re= transmitted. Please feel free to correct me if I am wrong. Should we try with 0 ms timetolive which indicates no timeout should occur = on this message and is treated as reliable? Please suggest and also let me = know if you need any additional information. Thanking you in advance for your response.=20 Regards, Chinmaya --808960-2099520786-1338489793=:23407-- From ckdwibedy@yahoo.com Thu May 31 12:20:16 2012 Return-Path: X-Original-To: tsvwg@ietfa.amsl.com Delivered-To: tsvwg@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 4FE6221F8638 for ; Thu, 31 May 2012 12:20:16 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -1.217 X-Spam-Level: X-Spam-Status: No, score=-1.217 tagged_above=-999 required=5 tests=[AWL=0.929, BAYES_00=-2.599, HTML_MESSAGE=0.001, MIME_8BIT_HEADER=0.3, SARE_SUB_ENC_UTF8=0.152] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id j-m3k+vtaqRF for ; Thu, 31 May 2012 12:20:15 -0700 (PDT) Received: from nm3-vm0.bullet.mail.sp2.yahoo.com (nm3-vm0.bullet.mail.sp2.yahoo.com [98.139.90.230]) by ietfa.amsl.com (Postfix) with SMTP id 0314621F8634 for ; Thu, 31 May 2012 12:20:15 -0700 (PDT) Received: from [72.30.22.78] by nm3.bullet.mail.sp2.yahoo.com with NNFMP; 31 May 2012 19:20:09 -0000 Received: from [98.139.91.46] by tm12.bullet.mail.sp2.yahoo.com with NNFMP; 31 May 2012 19:20:09 -0000 Received: from [127.0.0.1] by omp1046.mail.sp2.yahoo.com with NNFMP; 31 May 2012 19:20:09 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 92242.68868.bm@omp1046.mail.sp2.yahoo.com Received: (qmail 99160 invoked by uid 60001); 31 May 2012 19:20:08 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1338492008; bh=XNrXFE6jYPO9HsYv+hoti8VbDNwOIXEkK45NR+nOF9Y=; h=X-YMail-OSG:Received:X-Mailer:References:Message-ID:Date:From:Reply-To:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type; b=O/9JxOvXYzQrfRU+YOUk0S9P/0DYVZgRZvtSQxtCLB5fHuxOtrAiUg929XLXBS12pEKFLIkvMEpzoHgRJUx6n/Ddvwd8Vvw9nLGaYwVqrSSPbJ40eKYb3zvwpU2zIM5jLg5dL8Y7o2vN/A7Ho918RFyCwlAtsq2BmJElQxqGz58= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=X-YMail-OSG:Received:X-Mailer:References:Message-ID:Date:From:Reply-To:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type; b=NYcr7Zkll3xp0bstq7+ZtASCoWWN2NKfPqky0W6uPMWfSE5UD+qNk+n+ofJh4KNxfwiQHNSXw61i3OYaPfbCtO1Fhu/A6Ryar3GUaK4s8TRS21SyyYehgd+B4Zewb06DpqTm0EwTbsPPHDYzuWobShnQ4Puyv/7pFfKY9HXbkv8=; X-YMail-OSG: sHLRHv4VM1kzxhlIGUB48KSNqCgfb1oilRGAKgikjS6Pchz XzU0GP60Qfh89tKIw.RzYZyo4xdIm9cutaMtiA4vUu.35lbioJij23MXBB1. ReggnC1G8dc8PwIpRigTej2UP2RRpQcNxk6jKXQIRXxvXbbL5v4afeQ481t8 0xhib8Lv4_AOI9Vw6aow9cxaJZbVKk9cLCI2l6Eof5peTmIquATauOfETgoF 3ro1DfBx9T8sGzOA60IfZ4uvqCVrpyzr9_qqk4PpjdlvbUhQPWstGfws_A6n P8gz2JaLhXV7Cfmz.V8.SFBYlm5i_soAXciJu__SrQkIBfKHws76_mUN6htL qnVXiN39WBhJIlZQ.G238gwuGNfsVZwbfzewdlxs.tfeLMfdPXYRE.2YQAVd RENNFBRs941WxzaEZUIK2cW0hg.bjQ1FxgxstiP7R2c.ZWR_U4J7fx9yVnEX ga8JDpTUSnjUUaMd0m6wPg5_g Received: from [117.197.235.65] by web130103.mail.mud.yahoo.com via HTTP; Thu, 31 May 2012 12:20:08 PDT X-Mailer: YahooMailWebService/0.8.118.349524 References: <4FBE562E.6000909@gmail.com> <1338463399.19395.YahooMailNeo@web130103.mail.mud.yahoo.com> <1338467385.57530.YahooMailNeo@web130105.mail.mud.yahoo.com> <1338489317.38384.YahooMailNeo@web130103.mail.mud.yahoo.com> Message-ID: <1338492008.97916.YahooMailNeo@web130103.mail.mud.yahoo.com> Date: Thu, 31 May 2012 12:20:08 -0700 (PDT) From: Chinmaya Dwibedy To: Chris Benson In-Reply-To: MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="-725201809-312286038-1338492008=:97916" Cc: "tsvwg@ietf.org" Subject: Re: [tsvwg] =?utf-8?q?When_SCTP_server_is_restarted=2C_DATA_chunks_st?= =?utf-8?q?art_to_get_dropped_or_lost_after_sometime_at_SCTP_client?= =?utf-8?b?4oCZcyBlbmQu?= X-BeenThere: tsvwg@ietf.org X-Mailman-Version: 2.1.12 Precedence: list Reply-To: Chinmaya Dwibedy List-Id: Transport Area Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 May 2012 19:20:16 -0000 ---725201809-312286038-1338492008=:97916 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Hi Chris,=0AThanks for your response.=0AOur application consumes the messag= e fast enough. Note that, we are using single Intel Xeon Quad-Core L5518 pr= ocessor, 8 logical CPUs (speed of the processor is 2.13GHz) and 12GB RAM. M= oreover there are no other apps running on besides our application. This pr= oblem appears after SCTP server gets restarted. There is absolutely no pack= ets drop if we won=E2=80=99t restart the server.=0A=C2=A0=0ADo you recommen= d us to try with 0 ms timetolive which indicates no timeout should occur on= this message and is treated as reliable?=0A=C2=A0=0ARegards,=0AChinmaya=0A= =0A=0A=0A________________________________=0AFrom: Chris Benson =0ATo: Chinmaya Dwibedy =0ACc: "tsvwg@ietf.org"= =0ASent: Friday, June 1, 2012 12:13 AM=0ASubject: Re: [ts= vwg] When SCTP server is restarted, DATA chunks start to get dropped or los= t after sometime at SCTP client=E2=80=99s end.=0A=0AChinmaya,=0A=0APlease f= orgive me if this is too basic and obvious a question.=0A=0AAre you sure th= at your "application" (SCTP-user) at the client =0Aside is consuming the in= coming data AT LEAST AS FAST as it arrives?=0A=0AIf not, then you will see = something like the behaviour you =0Adescribe. With thanks, from Chris=0A=0A= On Thu, 31 May 2012, Chinmaya Dwibedy wrote:=0A=0A>>=C2=A0 Date: Thu, 31 Ma= y 2012 11:35:17 -0700 (PDT)=0A>>=C2=A0 From: Chinmaya Dwibedy =0A>>=C2=A0 To: "tsvwg@ietf.org" =0A>>=C2=A0 Subjec= t: [tsvwg] When SCTP server is restarted,=0A>>=C2=A0 =C2=A0 =C2=A0 DATA chu= nks start to get dropped or lost after sometime at SCTP client=0A>>=C2=A0 = =C2=A0 =C2=A0 end.=0A>>=C2=A0 =0A>>=C2=A0 Hi, =0AWe have SCTP client progra= m (C++) running under RHEL4 (using kernel version: 2.6.9-55ELsmp and lksctp= -tools-1.0.8-1). We are using one-to-one style socket using connect () syst= em call to setup an association with a SCTP server (i.e., SUT of vendor). T= hen it calls sctp_sendmsg () library function to send a message from a sock= et while using the following advanced features of SCTP (i.e., SCTP_UNORDERE= D for un-ordered delivery of the message) and timetolive (TTL) is set to 10= 00 milliseconds for a given message. We have typical defaults values for th= ese RTO.Initial =3D 3000, RTO.Min =3D 1000, RTO.Max =3D 60000 milliseconds.= The PR-SCTP extension is enabled in the kernel (by default).=0AWe are using= lot (4000) SCTP associations on a single system, all are in ESTABLISHED st= ate and data communication goes on fine without any drop/loss of message. B= ut when the peer (i.e., SCTP server) gets restarted, =0Aa)=C2=A0=C2=A0=C2= =A0 All the SCTP clients receive INIT messages (with same address to the as= sociation) from SCTP server. The INIT ACK is being sent with the new InitTa= g and Verification Tag set to the InitTag received in INIT for each associa= tion. =0Ab)=C2=A0=C2=A0=C2=A0 The four-way handshake gets completed. I beli= eve the existing association, including its current state, and the correspo= nding TCB does not get changed. =0Ac)=C2=A0=C2=A0=C2=A0 Afterward, all SCTP= clients use DATA chunks to exchange information with SCTP server.=0AThe tr= ouble is that, after sometime (30-45 minutes), DATA chunks start to get dro= pped or lost and that goes on increasing as the time advances.The data was = not transmitted to the peer at least once. Note that, the time after which = this problem is being encountered is also not consistent. I mean to say, so= metime we noticed after 30 minutes, 45 minutes and an hour. But when we les= sen the number of SCTP associations to 2000, such problem does not appear. = =C2=A0=C2=A0=C2=A0=0A=0A=0Aa)=C2=A0=C2=A0=C2=A0 Is there any chance that, t= hat data messages time-out within the RTO although both the values are same= ? As a result, the data will be skipped and no longer transmitted.=0Ab)=C2= =A0=C2=A0=C2=A0 Does it indicate that the message could never be transmitte= d to the peer (e.g., flow control prevented the message from being sent bef= ore its lifetime expired), so the peer never received it? Please note that,= we are using 1 Gig Intel NIC. =0AThe preliminary =C2=A0analysis says that,= timetolive (in sctp_sendmsg() API) can be set on a message independent of = Partial Reliability.The timer (i.e., sinfo_timetolive) only runs while the = message is in the kernel send buffer and has not yet been put on the wire (= for the first time). As soon as the message is put on the wire the first ti= me, the timer is dropped. It is used to indicate that a time based lifetime= is being applied to the data.=C2=A0 It is then a number of milliseconds fo= r which the data is attempted to be transmitted. If that many milliseconds = elapsed and the data has not been transmitted, As a result, they move the a= bandoned list and are never retransmitted. Please feel free to correct me i= f I am wrong.=0AShould we try with 0 ms timetolive which indicates no timeo= ut should occur on this message and is treated as reliable? Please suggest = and also let me know if you need any additional information.=0AThanking you= in advance for your response. =0ARegards,=0AChinmaya ---725201809-312286038-1338492008=:97916 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable

Hi Chris,

Thanks for your response.

Our application consumes the message fast enough. Not= e that, we are using single Intel Xeon Quad-Core L5518 processor, 8 logical= CPUs (speed of the processor is 2.13GHz) and 12GB RAM. Moreover there are = no other apps running on besides our application. This problem appears afte= r SCTP server gets restarted. There is absolutely no packets drop if we won= =E2=80=99t restart the server.

Do you recommend us to try with 0 ms timetolive which= indicates no timeout should occur on this message and is treated as reliab= le?

Regards,

= Chinmaya

From:= Chris Benson <cbenson@adax.com>
To: Chinmaya Dwibedy <ckdwibedy@yahoo.com> =
Cc: "tsvwg@ietf.org" &l= t;tsvwg@ietf.org>
Sent:<= /B> Friday, June 1, 2012 12:13 AM
S= ubject: Re: [tsvwg] When SCTP server is restarted, DATA chunks s= tart to get dropped or lost after sometime at SCTP client=E2=80=99s end.

Chinmaya,

Please forgive me if this is too basic a= nd obvious a question.

Are you sure that your "application" (SCTP-user) at the c= lient
side is consuming the incoming data AT LEAST AS FAST as it arrive= s?

If not, then you will see something like the behaviour you
de= scribe. With thanks, from Chris

On Thu, 31 May 2012, Chinmaya Dwibed= y wrote:

>> Date: Thu, 31 May 2012 11:35:17 -0700 (PDT)<= BR>>> From: Chinmaya Dwibedy <ckdwibedy@yahoo.com>=
>> To: "tsvwg@ietf.org" <tsvwg@ietf.org>
>> = Subject: [tsvwg] When SCTP server is restarted,
>> &n= bsp; DATA chunks start to get dropped or lost after sometime at SCTP client=
>> end.
>>
>> = Hi,
We have SCTP client program (C++) running under RHEL4 (using kernel ve= rsion: 2.6.9-55ELsmp and lksctp-tools-1.0.8-1). We are using one-to-one sty= le socket using connect () system call to setup an association with a SCTP = server (i.e., SUT of vendor). Then it calls sctp_sendmsg () library functio= n to send a message from a socket while using the following advanced featur= es of SCTP (i.e., SCTP_UNORDERED for un-ordered delivery of the message) an= d timetolive (TTL) is set to 1000 milliseconds for a given message. We have= typical defaults values for these RTO.Initial =3D 3000, RTO.Min =3D 1000, = RTO.Max =3D 60000 milliseconds.The PR-SCTP extension is enabled in the kern= el (by default).
We are using lot (4000) SCTP associations on a single s= ystem, all are in ESTABLISHED state and data communication goes on fine wit= hout any drop/loss of message. But when the peer (i.e., SCTP server) gets r= estarted,
a)    All the SCTP clients receive INIT messages (with same address to the association) from SCTP server. The INIT= ACK is being sent with the new InitTag and Verification Tag set to the Ini= tTag received in INIT for each association.
b)    The fo= ur-way handshake gets completed. I believe the existing association, includ= ing its current state, and the corresponding TCB does not get changed.
= c)    Afterward, all SCTP clients use DATA chunks to exchang= e information with SCTP server.
The trouble is that, after sometime (30-= 45 minutes), DATA chunks start to get dropped or lost and that goes on incr= easing as the time advances.The data was not transmitted to the peer at lea= st once. Note that, the time after which this problem is being encountered = is also not consistent. I mean to say, sometime we noticed after 30 minutes= , 45 minutes and an hour. But when we lessen the number of SCTP association= s to 2000, such problem does not appear.

a)    Is there any chance tha= t, that data messages time-out within the RTO although both the values are = same? As a result, the data will be skipped and no longer transmitted.
b= )    Does it indicate that the message could never be transm= itted to the peer (e.g., flow control prevented the message from being sent= before its lifetime expired), so the peer never received it? Please note t= hat, we are using 1 Gig Intel NIC.
The preliminary analysis says = that, timetolive (in sctp_sendmsg() API) can be set on a message independen= t of Partial Reliability.The timer (i.e., sinfo_timetolive) only runs while= the message is in the kernel send buffer and has not yet been put on the w= ire (for the first time). As soon as the message is put on the wire the fir= st time, the timer is dropped. It is used to indicate that a time based lif= etime is being applied to the data. It is then a number of milliseconds for which the data is attempted to be transmitted. If that ma= ny milliseconds elapsed and the data has not been transmitted, As a result,= they move the abandoned list and are never retransmitted. Please feel free= to correct me if I am wrong.
Should we try with 0 ms timetolive which i= ndicates no timeout should occur on this message and is treated as reliable= ? Please suggest and also let me know if you need any additional informatio= n.
Thanking you in advance for your response.
Regards,
Chinmaya

---725201809-312286038-1338492008=:97916-- From randall@lakerest.net Thu May 31 16:35:56 2012 Return-Path: X-Original-To: tsvwg@ietfa.amsl.com Delivered-To: tsvwg@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id EF96311E8080 for ; Thu, 31 May 2012 16:35:56 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.449 X-Spam-Level: X-Spam-Status: No, score=-2.449 tagged_above=-999 required=5 tests=[AWL=-0.150, BAYES_00=-2.599, MIME_8BIT_HEADER=0.3] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 8TGw6yeypOSf for ; Thu, 31 May 2012 16:35:56 -0700 (PDT) Received: from lakerest.net (lakerest.net [70.155.160.98]) by ietfa.amsl.com (Postfix) with ESMTP id 3823711E807F for ; Thu, 31 May 2012 16:35:56 -0700 (PDT) Received: from [10.1.1.141] (bsd3.lakerest.net [70.155.160.101]) (authenticated bits=0) by lakerest.net (8.14.4/8.14.3) with ESMTP id q4VNYeRN044665 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NOT); Thu, 31 May 2012 19:34:40 -0400 (EDT) (envelope-from randall@lakerest.net) Mime-Version: 1.0 (Apple Message framework v1278) Content-Type: text/plain; charset=us-ascii From: Randy Stewart In-Reply-To: <1338467385.57530.YahooMailNeo@web130105.mail.mud.yahoo.com> Date: Thu, 31 May 2012 19:34:40 -0400 Content-Transfer-Encoding: quoted-printable Message-Id: References: <4FBE562E.6000909@gmail.com> <1338463399.19395.YahooMailNeo@web130103.mail.mud.yahoo.com> <1338467385.57530.YahooMailNeo@web130105.mail.mud.yahoo.com> To: Chinmaya Dwibedy X-Mailer: Apple Mail (2.1278) Cc: "tsvwg@ietf.org" Subject: Re: [tsvwg] =?windows-1252?q?A_query_on_SCTP_client=92s_=28running_un?= =?windows-1252?q?der_RHEL4=29_performance_in_terms_of_aggregate_throughpu?= =?windows-1252?q?t_rate?= X-BeenThere: tsvwg@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Transport Area Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 May 2012 23:35:57 -0000 Chinmaya: If you are using a connected socket aka one-2-one model, then you can = also use send() and that should work. The sctp_sendmsg() is also used by the one-2-many model socket, and = there you need a socket address. Some implementations allow you to place a NULL in the = socket address of a connected socket for sctp_sendmsg(), not sure if linux does. But = this is all implementation specific stuff. R On May 31, 2012, at 8:29 AM, Chinmaya Dwibedy wrote: >=20 >=20 > Hi. > =20 > We have SCTP client program (C++) running under RHEL4 (using kernel = version: 2.6.9-55ELsmp and lksctp-tools-1.0.8-1). We are using = one-to-one socket (also called TCP-style sockets) using connect () = system call to setup an association with a server. Then it calls = sctp_sendmsg () library function to send a message from a socket while = using the following advanced features of SCTP i.e., SCTP_UNORDERED for = un-ordered delivery of the message. The sctp_sendmsg () API takes a = pointer to "sockaddr" as a parameter indicating the address of the = destination. Since our client application knows well that, it will be = sending the multiple messages to the same configured destination = address, it would become more efficient if we will connect the socket = explicitly. In summary, we can say that a SCTP client can call connect = only if that process uses the SCTP socket to communicate with exactly = one peer. I mean, calling connect () on the SCTP socket once simply = records the specified IP address and port number as being the desired = communications partner and send/write call. Anything written to a = connected SCTP socket is automatically sent to the protocol address = (e.g., IP address and port) specified by connect. > After connect() system call, there will be absolutely no need of = again passing socket address structure containing the destination IP = address and port for each message in sctp_sendmsg(). It would be = unnecessary memory copy between user space and kernel space, which is a = performance hit. Rather we should pass the destination address of the = message and length of the destination address to zero in sctp_sendmsg () = API. I believe, with this change, it will improve performance (i.e., = aggregate throughput rate) when using a lot (32K) associations on a = single system with 8 logical CPUs (speed of the processor is 2.13GHz) = and 12GB RAM. Can someone please validate my understanding and correct = me if I am wrong? Thanks in advance for your response. > Regards, > Chinmaya >=20 >=20 ----- Randall Stewart randall@lakerest.net From randall@lakerest.net Thu May 31 16:37:00 2012 Return-Path: X-Original-To: tsvwg@ietfa.amsl.com Delivered-To: tsvwg@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 27E8111E8080 for ; Thu, 31 May 2012 16:37:00 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.374 X-Spam-Level: X-Spam-Status: No, score=-2.374 tagged_above=-999 required=5 tests=[AWL=-0.075, BAYES_00=-2.599, MIME_8BIT_HEADER=0.3] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id DU7+MuLO+Z-y for ; Thu, 31 May 2012 16:36:59 -0700 (PDT) Received: from lakerest.net (lakerest.net [70.155.160.98]) by ietfa.amsl.com (Postfix) with ESMTP id 3E63311E8073 for ; Thu, 31 May 2012 16:36:59 -0700 (PDT) Received: from [10.1.1.141] (bsd3.lakerest.net [70.155.160.101]) (authenticated bits=0) by lakerest.net (8.14.4/8.14.3) with ESMTP id q4VNZfIg044675 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NOT); Thu, 31 May 2012 19:35:41 -0400 (EDT) (envelope-from randall@lakerest.net) Mime-Version: 1.0 (Apple Message framework v1278) Content-Type: text/plain; charset=windows-1252 From: Randy Stewart In-Reply-To: <1338492008.97916.YahooMailNeo@web130103.mail.mud.yahoo.com> Date: Thu, 31 May 2012 19:35:41 -0400 Content-Transfer-Encoding: quoted-printable Message-Id: References: <4FBE562E.6000909@gmail.com> <1338463399.19395.YahooMailNeo@web130103.mail.mud.yahoo.com> <1338467385.57530.YahooMailNeo@web130105.mail.mud.yahoo.com> <1338489317.38384.YahooMailNeo@web130103.mail.mud.yahoo.com> <1338492008.97916.YahooMailNeo@web130103.mail.mud.yahoo.com> To: Chinmaya Dwibedy X-Mailer: Apple Mail (2.1278) Cc: "tsvwg@ietf.org" Subject: Re: [tsvwg] =?windows-1252?q?When_SCTP_server_is_restarted=2C_DATA_ch?= =?windows-1252?q?unks_start_to_get_dropped_or_lost_after_sometime_at_SCTP?= =?windows-1252?q?_client=92s_end=2E?= X-BeenThere: tsvwg@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Transport Area Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 May 2012 23:37:00 -0000 Chinmaya: I would use that has a starting point. Don't put a time-to-expire on it = and see if you no longer see this behavior. If you do, then it sounds like a bug in = the linux kernel implementation R On May 31, 2012, at 3:20 PM, Chinmaya Dwibedy wrote: > Hi Chris, > Thanks for your response. > Our application consumes the message fast enough. Note that, we are = using single Intel Xeon Quad-Core L5518 processor, 8 logical CPUs (speed = of the processor is 2.13GHz) and 12GB RAM. Moreover there are no other = apps running on besides our application. This problem appears after SCTP = server gets restarted. There is absolutely no packets drop if we won=92t = restart the server. > =20 > Do you recommend us to try with 0 ms timetolive which indicates no = timeout should occur on this message and is treated as reliable? > =20 > Regards, > Chinmaya >=20 > From: Chris Benson > To: Chinmaya Dwibedy =20 > Cc: "tsvwg@ietf.org" =20 > Sent: Friday, June 1, 2012 12:13 AM > Subject: Re: [tsvwg] When SCTP server is restarted, DATA chunks start = to get dropped or lost after sometime at SCTP client=92s end. >=20 > Chinmaya, >=20 > Please forgive me if this is too basic and obvious a question. >=20 > Are you sure that your "application" (SCTP-user) at the client=20 > side is consuming the incoming data AT LEAST AS FAST as it arrives? >=20 > If not, then you will see something like the behaviour you=20 > describe. With thanks, from Chris >=20 > On Thu, 31 May 2012, Chinmaya Dwibedy wrote: >=20 > >> Date: Thu, 31 May 2012 11:35:17 -0700 (PDT) > >> From: Chinmaya Dwibedy > >> To: "tsvwg@ietf.org" > >> Subject: [tsvwg] When SCTP server is restarted, > >> DATA chunks start to get dropped or lost after sometime at = SCTP client > >> end. > >> =20 > >> Hi,=20 > We have SCTP client program (C++) running under RHEL4 (using kernel = version: 2.6.9-55ELsmp and lksctp-tools-1.0.8-1). We are using = one-to-one style socket using connect () system call to setup an = association with a SCTP server (i.e., SUT of vendor). Then it calls = sctp_sendmsg () library function to send a message from a socket while = using the following advanced features of SCTP (i.e., SCTP_UNORDERED for = un-ordered delivery of the message) and timetolive (TTL) is set to 1000 = milliseconds for a given message. We have typical defaults values for = these RTO.Initial =3D 3000, RTO.Min =3D 1000, RTO.Max =3D 60000 = milliseconds.The PR-SCTP extension is enabled in the kernel (by = default). > We are using lot (4000) SCTP associations on a single system, all are = in ESTABLISHED state and data communication goes on fine without any = drop/loss of message. But when the peer (i.e., SCTP server) gets = restarted,=20 > a) All the SCTP clients receive INIT messages (with same address to = the association) from SCTP server. The INIT ACK is being sent with the = new InitTag and Verification Tag set to the InitTag received in INIT for = each association.=20 > b) The four-way handshake gets completed. I believe the existing = association, including its current state, and the corresponding TCB does = not get changed.=20 > c) Afterward, all SCTP clients use DATA chunks to exchange = information with SCTP server. > The trouble is that, after sometime (30-45 minutes), DATA chunks start = to get dropped or lost and that goes on increasing as the time = advances.The data was not transmitted to the peer at least once. Note = that, the time after which this problem is being encountered is also not = consistent. I mean to say, sometime we noticed after 30 minutes, 45 = minutes and an hour. But when we lessen the number of SCTP associations = to 2000, such problem does not appear. =20 >=20 >=20 > a) Is there any chance that, that data messages time-out within the = RTO although both the values are same? As a result, the data will be = skipped and no longer transmitted. > b) Does it indicate that the message could never be transmitted to = the peer (e.g., flow control prevented the message from being sent = before its lifetime expired), so the peer never received it? Please note = that, we are using 1 Gig Intel NIC.=20 > The preliminary analysis says that, timetolive (in sctp_sendmsg() = API) can be set on a message independent of Partial Reliability.The = timer (i.e., sinfo_timetolive) only runs while the message is in the = kernel send buffer and has not yet been put on the wire (for the first = time). As soon as the message is put on the wire the first time, the = timer is dropped. It is used to indicate that a time based lifetime is = being applied to the data. It is then a number of milliseconds for = which the data is attempted to be transmitted. If that many milliseconds = elapsed and the data has not been transmitted, As a result, they move = the abandoned list and are never retransmitted. Please feel free to = correct me if I am wrong. > Should we try with 0 ms timetolive which indicates no timeout should = occur on this message and is treated as reliable? Please suggest and = also let me know if you need any additional information. > Thanking you in advance for your response.=20 > Regards, > Chinmaya >=20 ----- Randall Stewart randall@lakerest.net From internet-drafts@ietf.org Thu May 31 23:34:18 2012 Return-Path: X-Original-To: tsvwg@ietfa.amsl.com Delivered-To: tsvwg@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 251D921F85E1; Thu, 31 May 2012 23:34:18 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -102.486 X-Spam-Level: X-Spam-Status: No, score=-102.486 tagged_above=-999 required=5 tests=[AWL=0.114, BAYES_00=-2.599, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 0lBNd0ZciPFC; Thu, 31 May 2012 23:34:17 -0700 (PDT) Received: from ietfa.amsl.com (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 933C221F85E3; Thu, 31 May 2012 23:34:17 -0700 (PDT) MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable From: internet-drafts@ietf.org To: i-d-announce@ietf.org X-Test-IDTracker: no X-IETF-IDTracker: 4.02 Message-ID: <20120601063417.29442.54679.idtracker@ietfa.amsl.com> Date: Thu, 31 May 2012 23:34:17 -0700 Cc: tsvwg@ietf.org Subject: [tsvwg] I-D Action: draft-ietf-tsvwg-port-use-00.txt X-BeenThere: tsvwg@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Transport Area Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Jun 2012 06:34:18 -0000 A New Internet-Draft is available from the on-line Internet-Drafts director= ies. This draft is a work item of the Transport Area Working Group Working = Group of the IETF. Title : Recommendations for Transport Port Uses Author(s) : Joe Touch Filename : draft-ietf-tsvwg-port-use-00.txt Pages : 11 Date : 2012-05-30 This document provides recommendations to application and service designers on how to use the transport protocol port number space to help in its preservation. **NOTE THAT THIS CURRENT VERSION IS LARGELY AN OUTLINE OF ISSUES**. A URL for this Internet-Draft is: http://www.ietf.org/internet-drafts/draft-ietf-tsvwg-port-use-00.txt Internet-Drafts are also available by anonymous FTP at: ftp://ftp.ietf.org/internet-drafts/ This Internet-Draft can be retrieved at: ftp://ftp.ietf.org/internet-drafts/draft-ietf-tsvwg-port-use-00.txt The IETF datatracker page for this Internet-Draft is: https://datatracker.ietf.org/doc/draft-ietf-tsvwg-port-use/