From nobody Sun Jan 3 21:03:26 2016 Return-Path: X-Original-To: dtn-users@ietfa.amsl.com Delivered-To: dtn-users@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id CE0E81B2AF6 for ; Sun, 3 Jan 2016 21:03:24 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: 0.251 X-Spam-Level: X-Spam-Status: No, score=0.251 tagged_above=-999 required=5 tests=[BAYES_50=0.8, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-0.7, SPF_PASS=-0.001] autolearn=ham Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id gdJ7RJIVkvcx for ; Sun, 3 Jan 2016 21:03:23 -0800 (PST) Received: from mail-wm0-f54.google.com (mail-wm0-f54.google.com [74.125.82.54]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 36C451B2AF7 for ; Sun, 3 Jan 2016 21:03:21 -0800 (PST) Received: by mail-wm0-f54.google.com with SMTP id f206so159803926wmf.0 for ; Sun, 03 Jan 2016 21:03:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:reply-to:in-reply-to:references:date:message-id :subject:from:to:cc:content-type; bh=iItEqOQMKk7fFziPgBf84BNfut1bctJvmVobItuWRlw=; b=P02MfWVktV8W9l6mVjWFBgoF21fZXYlF0fpbovkOIyiaP48gtW37hqSMf83+oasg0W 2lbmoMGRrKoxEMAi9BinrBJkXIdCS28nwiJ2EpEgoqEwwBqhq8kdFbMbAmT+6zBFH+Zc 4MPDYsfU5DHTUv6RxGcxKYv024VdAymbnX8W7eC4/Iiuo3Fznp4jQ7cVx1ppsWnzQhA9 uAY5JRdSfalm1TFumwlq8j4dWyaLncyBuI3yOdU0PruBSlWiyS8yJlsVwjR75PgqA2yQ t1hWkFvJOdw4AfALnU3+AoP0P0DT13DXy/UCIs1qzKWQ59GnYwBKn6qWv1b3Y+bpM4Yf FkVg== MIME-Version: 1.0 X-Received: by 10.195.18.100 with SMTP id gl4mr35691377wjd.177.1451883799209; Sun, 03 Jan 2016 21:03:19 -0800 (PST) Received: by 10.194.92.164 with HTTP; Sun, 3 Jan 2016 21:03:19 -0800 (PST) In-Reply-To: References: Date: Mon, 4 Jan 2016 09:03:19 +0400 Message-ID: From: Nik Ansell To: kevin prima Content-Type: multipart/alternative; boundary=001a11c296081b6b0505287b0a50 Archived-At: Cc: dtn-users@ietfa.amsl.com Subject: Re: [dtn-users] Problem DTN2 daemon X-BeenThere: dtn-users@irtf.org X-Mailman-Version: 2.1.15 Precedence: list Reply-To: nikansell00@gmail.com List-Id: "The Delay-Tolerant Networking Research Group \(DTNRG\) - Users." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Jan 2016 05:03:25 -0000 --001a11c296081b6b0505287b0a50 Content-Type: text/plain; charset=UTF-8 Hi Kevin, I am pretty new to DTN2, so not really the best person to give advice. However, I did come across a similar spin lock error during my recent testing. In my case, the spin lock error was a symptom of an unrelated problem i.e a red herring, I ended up fixing my problem by updating my configs. This view is also shared by the N4C project, who hid the spin lock error by changing the log level to INFO/NOTICE. http://www.n4c.eu/Download/8.4/n4c-wp8-006-M8.5-summer_test_2010_Slovenia_V06.pdf Good luck fixing your problem. Kind Regards, Nik On Thu, Dec 31, 2015 at 12:17 PM, kevin prima wrote: > Dear all, > > I want to run 5 DTN node in 2 hop topology scenario with DTLSR routing, > source and destination node connected with dtn node 1,2,and 3 but couldn't > reach each other. > When I try to send ping packet use dtnping, sometimes it show error like > this in node 1, node 2, and node 3: > > warning: Bundle::del_ref is waiting for spin lock held by (null), which > has reached spin limit > STACK TRACE: 0x76f51c84 0x3eda8 0x3ed00 0x59444 > > The impact is node 1, 2 or 3 become unavailable. What should I do to fix > this problem ? > > Thanks, > Kevin > > _______________________________________________ > dtn-users mailing list > dtn-users@irtf.org > https://www.irtf.org/mailman/listinfo/dtn-users > > --001a11c296081b6b0505287b0a50 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hi Kevin,

I am pretty new to DTN2, so n= ot really the best person to give advice.
However, I did come acr= oss a similar spin lock error during my recent testing.
In my cas= e, the spin lock error was a symptom of an unrelated problem i.e a red herr= ing, I ended up fixing my problem by updating my configs.
This vi= ew is also shared by the N4C project, who hid the spin lock error by changi= ng the log level to INFO/NOTICE.


Good luck fixing your problem= .

Kind Regards,
Nik

On Thu, Dec 31, 2015 at 12:17 PM, kevin prim= a <sir.kevinprima@gmail.com> wrote:
Dear all,

I want to run 5 DTN node in 2 hop topology s= cenario with DTLSR routing, source and destination node connected with dtn = node 1,2,and 3 but couldn't reach each other.
When I try to send ping packet use dtnping, sometimes it s= how error like this in node 1, node 2, and node 3:

warning: Bundle::del_ref is waiting for spin lock held by (null), which h= as reached spin limit
STACK TRACE: 0x76f51c84 0x3eda8 0x3ed00 0x59444
=

The impact is node 1, 2 or 3 become unavailable. What sho= uld I do to fix this problem ?

Thanks,=
Kevin

_______________________________________________
dtn-users mailing list
dtn-users@irtf.org
https://www.irtf.org/mailman/listinfo/dtn-users

--001a11c296081b6b0505287b0a50-- From nobody Sun Jan 24 00:05:21 2016 Return-Path: X-Original-To: dtn-users@ietfa.amsl.com Delivered-To: dtn-users@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 1835A1A9107 for ; Sun, 24 Jan 2016 00:05:20 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: 0.75 X-Spam-Level: X-Spam-Status: No, score=0.75 tagged_above=-999 required=5 tests=[BAYES_40=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, J_CHICKENPOX_39=0.6, SPF_PASS=-0.001] autolearn=no Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 9p6AcVxEu52t for ; Sun, 24 Jan 2016 00:05:18 -0800 (PST) Received: from mail-io0-x22e.google.com (mail-io0-x22e.google.com [IPv6:2607:f8b0:4001:c06::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 561911A9106 for ; Sun, 24 Jan 2016 00:05:18 -0800 (PST) Received: by mail-io0-x22e.google.com with SMTP id 77so125245487ioc.2 for ; Sun, 24 Jan 2016 00:05:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:reply-to:date:message-id:subject:from:to:content-type; bh=5Ar2VYjAbR0e5xym337qPxDupYngTOy0u6Z60RDfQKE=; b=pim7zE7D4GzGbW55+nUETSGun3mUIo9siW/NVAZeuNUv/SWnPQNhFEemdWftbdBw53 mFQ1lI4bvWBREU9JrPI8H7ecTQrx5x5vKT1GzJjU1fyAa38ph1wcOTL9ReE0wzHN2Sc7 41pLj8V3YUPJVCbI16JmbESFjiQWqIdLP1pHJlBKZGju2saBpAVpoMZuj7HwfQo8HKH9 OOD3zANfdgdfh9buShG3qiQLMQlmQYMaPCoqAsq251U+nHH3tHb17rW7cosvpQIJMIp5 iuL8q+8SSC2FJEEy8waeK55ERpI5p+K9Si4RnPwdKwyUZp40MJ86dAY6m9kWgTByA+Mi Smhw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:reply-to:date:message-id:subject :from:to:content-type; bh=5Ar2VYjAbR0e5xym337qPxDupYngTOy0u6Z60RDfQKE=; b=PZXmmvdrY3aQpNp9q03ArrgHeILkoh1BtwNFvfBOJXH9254raChIQON1XMkIIsxhIj 8mNXNjAFK8+51d0EEEshXpZw/EWjs0M37nNeABGZKJcFJp0PkYUK/xWEtdbgC8QZLqXE Iv+4Tuz0z74Yt3ALgFQcZmcod00f4T3idqz6PBJij8xN2EYP0rW34k0UYNWeZ5s5MN34 j+WrQiFzKuH2EEfwXuO4j0/wXah4nRoFZZghPgwsapk/QMUQLhApNxb0olQI3SkMV8+Z LZoK4WN+3uJDdOIIeAg3G/sgfogIIc6/mYHUaSThWvladKk7E7TNLFBYSva8pX+9kJAF aMBg== X-Gm-Message-State: AG10YORDb3klRbNit5JDTIlXHr4tuVkjDRIVR+OUEpJ5B6WwQeSHztX9NjzOopxIDdgz4bk2OmUxqRe0sRN3tQ== MIME-Version: 1.0 X-Received: by 10.107.28.80 with SMTP id c77mr13067871ioc.98.1453622717527; Sun, 24 Jan 2016 00:05:17 -0800 (PST) Received: by 10.79.24.2 with HTTP; Sun, 24 Jan 2016 00:05:17 -0800 (PST) Date: Sun, 24 Jan 2016 12:05:17 +0400 Message-ID: From: Nik Ansell To: dtn-users@irtf.org Content-Type: multipart/alternative; boundary=001a11409b18b74493052a0fe97f Archived-At: Subject: [dtn-users] DTN2 - Frequent Spinlock errors X-BeenThere: dtn-users@irtf.org X-Mailman-Version: 2.1.15 Precedence: list Reply-To: nikansell00@gmail.com List-Id: "The Delay-Tolerant Networking Research Group \(DTNRG\) - Users." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 24 Jan 2016 08:05:20 -0000 --001a11409b18b74493052a0fe97f Content-Type: text/plain; charset=UTF-8 Hello All, I am running some experiments to identify bundle delivery characteristics of DTN2 (+LTPlib), ION and IBR-DTN on the Raspberry Pi B2. Currently I am trying to identify reliable bundle delivery scenarios for all 3 DTN implementations, so I can then apply certain network simulation scenarios to observe how each DTN implementation and convergence layer behaves. The scope of my experimentation is below for info: ION: UDPCL, TCPCL, LTPCL DTN2: UDPCL, TCLCP, LTPCL - using LTPlib IBR-DTN: UDPCL, TCPCL I have determined reliable, repeatable scenarios for both ION and IBR-DTN, however DTN2 is producing produce a number of errors when using UDP, TCP or LTP. The errors also seem to appear randomly after any number of for loop iterations, e.g. (5,13,50,48,95,413, 567, 988, etc etc). Errors that cause dtnd to use 200% CPU and stop delivering bundles: warning: deliver_front is waiting for spin lock held by (null), which has reached spin limit warning: Bundle::del_ref is waiting for spin lock held by (null), which has reached spin limit warning: Bundle::is_queued_on is waiting for spin lock held by (null), which has reached spin limit warning: LinkBlockSet::find_blocks is waiting for spin lock held by (null), which has reached spin limit warning: BundleList::erase is waiting for spin lock held by (null), which has reached spin limit warning: Bundle::add_ref is waiting for spin lock held by (null), which has reached spin limit warning: ForwardingLog::get_count is waiting for spin lock held by (null), which has reached spin limit Errors: that cause dtnd to quit: LTP:ASSERTION FAILED (is_locked_by_me()) at thread/SpinLock.cc:91 Errors that seem to loop forever but not cause dtnd to quit or use 200% CPU (This error seems to happen consistently after sending 99 LTP bundles): /dtn/cl/ltp/sender error] Unable to create Sender LTP Socket in 5 seconds - retrying To send files I am calling dtnsource in a for loop as below, which allows me to adjust the time between each bundle transmission, or each block of 10 bundle transmissions. For UDP and LTP I am using bundles of 63K, for TCP I am using 1M bundles. I have tried several combinations for $WAIT (0-5 seconds) and $BULKWAIT (0-20 seconds), but cannot seem to find a working combination. for ((i=1;i<=END;i++)); do echo "($COUNT) dtnsource -s dtn://tx.dtn/a -d dtn://rx.dtn/g -b $SIZE" dtnsource -s dtn://tx.dtn/a -d dtn://rx.dtn/g -b $SIZE sleep $WAIT let "COUNT++" # Sleep for x secs after each 10 if (( $COUNT % 10 == 0 )) then echo "Sleeping for $BULKWAIT seconds" sleep $BULKWAIT fi done The daemon is stopped and restarted after each test, I have also tried deleting and recreating the berkley DB each time, but this does not seem to have much effect. I have also managed to recreate some of the errors on a virtual Ubuntu test-bed, specifically Bundle::del_ref. After reading through the oasys SpinLock::lock code, it looks like this is possibly an infinite loop that tries up to 1,000,000 times to get a lock, then resets the counter to zero and puts out a warning message. Has anyone had a similar experience or can suggest any troubleshooting tips? Any suggestions to workaround or fix the problem will be very gratefully received! Kind Regards, Nik --001a11409b18b74493052a0fe97f Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hello All,

I am running some= experiments to identify bundle delivery characteristics of DTN2 (+LTPlib),= ION and IBR-DTN on the Raspberry Pi B2. Currently I am trying to identify = reliable bundle delivery scenarios for all 3 DTN implementations, so I can = then apply certain network simulation scenarios to observe how each DTN imp= lementation and convergence layer behaves.

The sco= pe of my experimentation is below for info:

ION: U= DPCL, TCPCL, LTPCL
DTN2: UDPCL, TCLCP, LTPCL - using LTPlib
=
IBR-DTN: UDPCL, TCPCL

I have determined relia= ble, repeatable scenarios for both ION and IBR-DTN, however DTN2 is produci= ng produce a number of errors when using UDP, TCP or LTP. The errors also s= eem to appear randomly after any number of for loop iterations, e.g. (5,13,= 50,48,95,413, 567, 988, etc etc).

Errors that caus= e dtnd to use 200% CPU and stop delivering bundles:
warning: deli= ver_front is waiting for spin lock held by (null), which has reached spin l= imit
warning: Bundle::del_ref is waiting for spin lock held by (n= ull), which has reached spin limit
warning: Bundle::is_queued_on is wait= ing for spin lock held by (null), which has reached spin limit
warning: = LinkBlockSet::find_blocks is waiting for spin lock held by (null), which ha= s reached spin limit
warning: BundleList::erase is waiting for sp= in lock held by (null), which has reached spin limit
warning:=C2= =A0Bundle::add_ref is waiting for spin lock held by (null), which has reach= ed spin limit
warning: ForwardingLog::get_count is waiting for spin lock= held by (null), which has reached spin limit

Errors: that cause dtn= d to quit:
LTP:ASSERTION FAILED (is_locked_by_me()) at thread/Spi= nLock.cc:91

Errors that seem to loop forever but not cause dtnd to q= uit or use 200% CPU (This error seems to happen consistently after sending = 99 LTP bundles):
/dtn/cl/ltp/sender error] Unable to create Sende= r LTP Socket in 5 seconds - retrying

To send files I am calling dtns= ource in a for loop as below, which allows me to adjust the time between ea= ch bundle transmission, or each block of 10 bundle transmissions. For UDP a= nd LTP I am using bundles of 63K, for TCP I am using 1M bundles. I have tri= ed several combinations for $WAIT (0-5 seconds) and $BULKWAIT (0-20 seconds= ), but cannot seem to find a working combination.

=
for ((i=3D1;i<=3DEND;i++)); do
=C2=A0 =C2=A0 echo "(= $COUNT) dtnsource -s dtn://tx.dtn/a -d dtn://rx.dtn/g -b $SIZE"
<= div>=C2=A0 =C2=A0 dtnsource -s dtn://tx.dtn/a -d dtn://rx.dtn/g -b $SIZE
=C2=A0 =C2=A0 sleep $WAIT
=C2=A0 =C2=A0 let "COUNT++= "
=C2=A0 =C2=A0=C2=A0
=C2=A0 =C2=A0 # Sleep for x secs after each 10
=C2=A0 =C2=A0 if (( $COUNT % 10 =3D=3D 0 ))
=C2=A0 =C2=A0 t= hen
=C2=A0 =C2=A0 echo "Sleeping for $BULKWAIT seconds"
=C2=A0 =C2=A0= sleep $BULKWAIT
=C2=A0 =C2=A0 fi

done

The daemon is stopped and restarted after each test, I have also trie= d deleting and recreating the berkley DB each time, but this does not seem = to have much effect. I have also managed to recreate some of the errors on = a virtual Ubuntu test-bed, specifically Bundle::del_ref.

After reading through the oasys SpinLock::lock code, it looks like t= his is possibly an infinite loop that tries up to 1,000,000 times to get a = lock, then resets the counter to zero and puts out a warning message.
=

Has anyone had a similar experience or can suggest any = troubleshooting tips?
Any suggestions to workaround or fix the pr= oblem will be very gratefully received!

Kind Regards= ,
Nik
--001a11409b18b74493052a0fe97f-- From nobody Sun Jan 24 23:19:24 2016 Return-Path: X-Original-To: dtn-users@ietfa.amsl.com Delivered-To: dtn-users@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id A6D1C1ACEC1 for ; Sun, 24 Jan 2016 23:19:23 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: 1.551 X-Spam-Level: * X-Spam-Status: No, score=1.551 tagged_above=-999 required=5 tests=[BAYES_50=0.8, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, J_CHICKENPOX_39=0.6, SPF_PASS=-0.001] autolearn=no Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id RJ0UByHFqAA3 for ; Sun, 24 Jan 2016 23:19:21 -0800 (PST) Received: from mail-ig0-x235.google.com (mail-ig0-x235.google.com [IPv6:2607:f8b0:4001:c05::235]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 7BB601ACEBD for ; Sun, 24 Jan 2016 23:19:21 -0800 (PST) Received: by mail-ig0-x235.google.com with SMTP id h5so25172629igh.0 for ; Sun, 24 Jan 2016 23:19:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:reply-to:in-reply-to:references:date:message-id :subject:from:to:content-type; bh=Z5LGOlv+TsJRmZyWjyZZXiPYY6V+5k2C8cJmGI53EEk=; b=LLpCXNRg6iNQYoxzcMIgCzADeKnjPR2KDdb5iaYeDyff1/7MS2zKvFQicqgkHiXa77 NExdQkeYgvu7KeUU+w6xyLc9xkkblyXcWh6zhMuTKzUGzkFB3PuuoH6ZlSYCUDNnoNIp lS2Jy6ndDONbOIBtd+ToLq8r+vI+7OKaPqYZuPowuRgg1fx2kJsxjhIVNQBDU1YHzbpJ wy4P9ogjgFJQuaZoLIuY2KGE40hb1MJ/61IcFXhpVspjGtpmxBV32WNy86lCffIYHihc r+hAS/H/P/+xWczvPEk0L5fEOGBQ9YUssb7F2GLNWfkDIL9EJDUoGSR58xzi5ZlucxZl 32AA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:reply-to:in-reply-to:references :date:message-id:subject:from:to:content-type; bh=Z5LGOlv+TsJRmZyWjyZZXiPYY6V+5k2C8cJmGI53EEk=; b=J/S4FyNmPXCgrOrwWMtYNrZwftRe3ZntFLQOwW5YKLlSxSHAo2U3+GIovPgcuBynZw FwW5/q3TKOf7ZaY3BcZ/ZNxooX3t3+KVdVf8zD2CjuT/aPiCXX/NKIlI2OXBTFls2fna 3c7H6zvbevGzJiGEYhDD+gQmjPrmRHuw1jWJRgtv7jdKBqT4MuyQDHDvEX7+JnF6BoiN VV7E7JUdJCgRFJCpd3aZBcE+EzCM3AXYx6rsWrFsm6OhTxFTlnC7dEEGHtp5dwohEQPI MLJ/1kLaRyGjomWGBa0nN/n5ipZBqDodgbAeDY6SZ8miCHycOzqDBPvtpsaHAAX5EAu7 uREA== X-Gm-Message-State: AG10YOSFt/BemKfe9M4sBYA0a/qInSgiw7sw2Bg7ZrC1F03pwiB4fng2WoJ6Ejvf6UAqHXNRC+xLqjNO3YUAow== MIME-Version: 1.0 X-Received: by 10.50.147.34 with SMTP id th2mr11304203igb.27.1453706360605; Sun, 24 Jan 2016 23:19:20 -0800 (PST) Received: by 10.79.24.2 with HTTP; Sun, 24 Jan 2016 23:19:20 -0800 (PST) In-Reply-To: References: Date: Mon, 25 Jan 2016 11:19:20 +0400 Message-ID: From: Nik Ansell To: dtn-users@irtf.org Content-Type: multipart/alternative; boundary=089e014952383b59c5052a2363f1 Archived-At: Subject: Re: [dtn-users] DTN2 - Frequent Spinlock errors X-BeenThere: dtn-users@irtf.org X-Mailman-Version: 2.1.15 Precedence: list Reply-To: nikansell00@gmail.com List-Id: "The Delay-Tolerant Networking Research Group \(DTNRG\) - Users." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 25 Jan 2016 07:19:23 -0000 --089e014952383b59c5052a2363f1 Content-Type: text/plain; charset=UTF-8 I've done some more digging and have a few observations (below), I will continue down this path but think I may be out of my depth already! :) 1. SpinLock.h: SpinLock::lock(): a) The error message appearing in the log (below) has a null lock_holder_name_ - Does this mean the lock is invalid (i.e perhaps the calling thread terminated due to an exception) and can be safely released? fprintf(stderr, "warning: %s is waiting for spin lock held by %s, which has reached spin limit\n", lock_user, lock_holder_name_) b) The call to (Thread.h) Thread::spin_yield() (which calls either thread_id_yield() or sched_yield() ). Is possibly failing in some cases - not sure how to debug this one....Has anyone any idea if I can somehow force the release of the lock if (as in above) lock_holder_name_ == NULL? c) The function atomic_cmpxchg32(&lock_count_,0,1) !=0 called is contained within several Atomic-.h files e.g. Atomic-arm.h or Atomic-x86.h - I am not sure how to confirm the correct header file was used during the build, but will try to build oasys and dtn2 again using the --host=arm-linux argument to configure to see if this makes a difference. 2. I have noticed there are some locking debugging functions available (Thread::lock_debugger->add_lock / remove_lock) if the --enable_debug_locking argument is passed during the build of oasys and dtn2. I am considering enabling this to get some more info on the problem. Kind Regards, Nik On Sun, Jan 24, 2016 at 12:05 PM, Nik Ansell wrote: > Hello All, > > I am running some experiments to identify bundle delivery characteristics > of DTN2 (+LTPlib), ION and IBR-DTN on the Raspberry Pi B2. Currently I am > trying to identify reliable bundle delivery scenarios for all 3 DTN > implementations, so I can then apply certain network simulation scenarios > to observe how each DTN implementation and convergence layer behaves. > > The scope of my experimentation is below for info: > > ION: UDPCL, TCPCL, LTPCL > DTN2: UDPCL, TCLCP, LTPCL - using LTPlib > IBR-DTN: UDPCL, TCPCL > > I have determined reliable, repeatable scenarios for both ION and IBR-DTN, > however DTN2 is producing produce a number of errors when using UDP, TCP or > LTP. The errors also seem to appear randomly after any number of for loop > iterations, e.g. (5,13,50,48,95,413, 567, 988, etc etc). > > Errors that cause dtnd to use 200% CPU and stop delivering bundles: > warning: deliver_front is waiting for spin lock held by (null), which has > reached spin limit > warning: Bundle::del_ref is waiting for spin lock held by (null), which > has reached spin limit > warning: Bundle::is_queued_on is waiting for spin lock held by (null), > which has reached spin limit > warning: LinkBlockSet::find_blocks is waiting for spin lock held by > (null), which has reached spin limit > warning: BundleList::erase is waiting for spin lock held by (null), which > has reached spin limit > warning: Bundle::add_ref is waiting for spin lock held by (null), which > has reached spin limit > warning: ForwardingLog::get_count is waiting for spin lock held by (null), > which has reached spin limit > > Errors: that cause dtnd to quit: > LTP:ASSERTION FAILED (is_locked_by_me()) at thread/SpinLock.cc:91 > > Errors that seem to loop forever but not cause dtnd to quit or use 200% > CPU (This error seems to happen consistently after sending 99 LTP bundles): > /dtn/cl/ltp/sender error] Unable to create Sender LTP Socket in 5 seconds > - retrying > > To send files I am calling dtnsource in a for loop as below, which allows > me to adjust the time between each bundle transmission, or each block of 10 > bundle transmissions. For UDP and LTP I am using bundles of 63K, for TCP I > am using 1M bundles. I have tried several combinations for $WAIT (0-5 > seconds) and $BULKWAIT (0-20 seconds), but cannot seem to find a working > combination. > > for ((i=1;i<=END;i++)); do > echo "($COUNT) dtnsource -s dtn://tx.dtn/a -d dtn://rx.dtn/g -b $SIZE" > dtnsource -s dtn://tx.dtn/a -d dtn://rx.dtn/g -b $SIZE > sleep $WAIT > let "COUNT++" > > # Sleep for x secs after each 10 > if (( $COUNT % 10 == 0 )) > then > echo "Sleeping for $BULKWAIT seconds" > sleep $BULKWAIT > fi > > done > > The daemon is stopped and restarted after each test, I have also tried > deleting and recreating the berkley DB each time, but this does not seem to > have much effect. I have also managed to recreate some of the errors on a > virtual Ubuntu test-bed, specifically Bundle::del_ref. > > After reading through the oasys SpinLock::lock code, it looks like this is > possibly an infinite loop that tries up to 1,000,000 times to get a lock, > then resets the counter to zero and puts out a warning message. > > Has anyone had a similar experience or can suggest any troubleshooting > tips? > Any suggestions to workaround or fix the problem will be very gratefully > received! > > Kind Regards, > Nik > --089e014952383b59c5052a2363f1 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
I've done some more digging and have a few observation= s (below), I will continue down this path but think I may be out of my dept= h already! :)

1. SpinLock.h: SpinLock::lock():
a) The error message appearing in the log (below) has a null lock_holder_n= ame_ - Does this mean the lock is invalid (i.e perhaps the calling thread t= erminated due to an exception) and can be safely released?
fprintf(stderr, "warning: %s is waiting for spin lock held by %s, w= hich has reached spin limit\n", lock_user, lock_holder_name_)

b) The call to (Thread.h) Thread::spin_yield() =C2= =A0(which calls either thread_id_yield() or sched_yield() ). Is possibly fa= iling in some cases - not sure how to debug this one....Has anyone any idea= if I can somehow force the release of the lock if (as in above) lock_holde= r_name_ =3D=3D NULL?

c) The function atomic_cm= pxchg32(&lock_count_,0,1) !=3D0 called is contained within several Atom= ic-<CPUARCH>.h files e.g. Atomic-arm.h or Atomic-x86.h - I am not sur= e how to confirm the correct header file was used during the build, but wil= l try to build oasys and dtn2 again using the --host=3Darm-linux argument t= o configure to see if this makes a difference.

2.= =C2=A0 I have noticed there are some locking debugging functions available = (Thread::lock_debugger->add_lock / remove_lock) if the=C2=A0--enable_deb= ug_locking argument is passed during the build of oasys and dtn2. I am cons= idering enabling this to get some more info on the problem.

Kind Regards,
Nik

On Sun, Jan 24, 2016 at 12:05 PM, Nik Ansell= <nikansell00@gmail.com> wrote:
Hello All,

I am ru= nning some experiments to identify bundle delivery characteristics of DTN2 = (+LTPlib), ION and IBR-DTN on the Raspberry Pi B2. Currently I am trying to= identify reliable bundle delivery scenarios for all 3 DTN implementations,= so I can then apply certain network simulation scenarios to observe how ea= ch DTN implementation and convergence layer behaves.

The scope of my experimentation is below for info:

<= div>ION: UDPCL, TCPCL, LTPCL
DTN2: UDPCL, TCLCP, LTPCL - using LT= Plib
IBR-DTN: UDPCL, TCPCL

I have determ= ined reliable, repeatable scenarios for both ION and IBR-DTN, however DTN2 = is producing produce a number of errors when using UDP, TCP or LTP. The err= ors also seem to appear randomly after any number of for loop iterations, e= .g. (5,13,50,48,95,413, 567, 988, etc etc).

Errors= that cause dtnd to use 200% CPU and stop delivering bundles:
war= ning: deliver_front is waiting for spin lock held by (null), which has reac= hed spin limit
warning: Bundle::del_ref is waiting for spin lock = held by (null), which has reached spin limit
warning: Bundle::is_queued_= on is waiting for spin lock held by (null), which has reached spin limitwarning: LinkBlockSet::find_blocks is waiting for spin lock held by (null)= , which has reached spin limit
warning: BundleList::erase is wait= ing for spin lock held by (null), which has reached spin limit
wa= rning:=C2=A0Bundle::add_ref is waiting for spin lock held by (null), which = has reached spin limit
warning: ForwardingLog::get_count is waiting for = spin lock held by (null), which has reached spin limit

Errors: that = cause dtnd to quit:
LTP:ASSERTION FAILED (is_locked_by_me()) at t= hread/SpinLock.cc:91

Errors that seem to loop forever but not cause = dtnd to quit or use 200% CPU (This error seems to happen consistently after= sending 99 LTP bundles):
/dtn/cl/ltp/sender error] Unable to cre= ate Sender LTP Socket in 5 seconds - retrying

To send files I am cal= ling dtnsource in a for loop as below, which allows me to adjust the time b= etween each bundle transmission, or each block of 10 bundle transmissions. = For UDP and LTP I am using bundles of 63K, for TCP I am using 1M bundles. I= have tried several combinations for $WAIT (0-5 seconds) and $BULKWAIT (0-2= 0 seconds), but cannot seem to find a working combination.

for ((i=3D1;i<=3DEND;i++)); do
=C2=A0 =C2=A0 ech= o "($COUNT) dtnsource -s dtn://tx.dtn/a -d dtn://rx.dtn/g -b $SIZE&quo= t;
=C2=A0 =C2=A0 dtnsource -s dtn://tx.dtn/a -d dtn://rx.dtn/g -b= $SIZE
=C2=A0 =C2=A0 sleep $WAIT
=C2=A0 =C2=A0 let &quo= t;COUNT++"
=C2=A0 =C2=A0=C2=A0
=C2=A0 =C2=A0 # Sleep for x secs after each 10
=C2=A0 =C2=A0 if (( $COUNT % 10 =3D=3D 0 ))
=C2=A0 =C2=A0 = then
=C2=A0 =C2=A0 e= cho "Sleeping for $BULKWAIT seconds"
=C2=A0 =C2=A0 sleep $BULKWAIT
=C2=A0 = =C2=A0 fi

done

The = daemon is stopped and restarted after each test, I have also tried deleting= and recreating the berkley DB each time, but this does not seem to have mu= ch effect. I have also managed to recreate some of the errors on a virtual = Ubuntu test-bed, specifically Bundle::del_ref.

Aft= er reading through the oasys SpinLock::lock code, it looks like this is pos= sibly an infinite loop that tries up to 1,000,000 times to get a lock, then= resets the counter to zero and puts out a warning message.

<= /div>
Has anyone had a similar experience or can suggest any troublesho= oting tips?
Any suggestions to workaround or fix the problem will= be very gratefully received!

Kind Regards,
Nik
=

--089e014952383b59c5052a2363f1-- From nobody Wed Jan 27 07:34:44 2016 Return-Path: X-Original-To: dtn-users@ietfa.amsl.com Delivered-To: dtn-users@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id D97751B386B for ; Wed, 27 Jan 2016 07:34:43 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: 2.151 X-Spam-Level: ** X-Spam-Status: No, score=2.151 tagged_above=-999 required=5 tests=[BAYES_50=0.8, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, J_CHICKENPOX_39=0.6, J_CHICKENPOX_82=0.6, SPF_PASS=-0.001] autolearn=no Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 8QKILGiHWbst for ; Wed, 27 Jan 2016 07:34:41 -0800 (PST) Received: from mail-io0-x230.google.com (mail-io0-x230.google.com [IPv6:2607:f8b0:4001:c06::230]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 2C6361B388F for ; Wed, 27 Jan 2016 07:34:41 -0800 (PST) Received: by mail-io0-x230.google.com with SMTP id f81so22655571iof.0 for ; Wed, 27 Jan 2016 07:34:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:reply-to:in-reply-to:references:date:message-id :subject:from:to:content-type; bh=hlO2PYht/whCVczJ/zrTAYthq0efk0Dapd38dImID4o=; b=wpVSlrNNvVPlb6gPGl2GRE1xFK041j/C92fXFFbdpmAEkxkHY/VaKZ/gwtiIAhbp7e SGxxCtisDrJPj4/S4YjKe7f/SjllJVDtysMqvtHYKQD/HWjaAqUrtSPpo8D3jgE2E6yx 543qAdA3eGoG4oqklrpugceUvUKm57b9U0ZbL/UmkmXuML0ZgrR1RhEqKfD5HFREROTB LLWHUX7k66gqyRMxmtf2U+i+0B7j+uYCVqDhulPFzqkWMbpj2nRKPra7q1VBHRyaa2y0 WNDB+hnpljiNtyqX8w0X9lOJpNLwxu56WtkWLYN4L0u5yr48yKRtC7fnNwMaZpRtyZLC l6kA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:reply-to:in-reply-to:references :date:message-id:subject:from:to:content-type; bh=hlO2PYht/whCVczJ/zrTAYthq0efk0Dapd38dImID4o=; b=GdIT9BDmkhwoXmoBLOHXQtCRmyvG+QnC9aDlqnZUk7LK+H17R6SVvEqVDSIakB2+B6 nyAcrnDUQm1YXhAjV6DHf6tIeVs9ZN70EVC7kzUzR8kG+Mb+951qE1BlKiJLk/QXG4dk aT2VdXQ6VTAKFBtcR2rksh2/vCY3jtFIk4znE7MvN5GkNaPj8bngr9zT9F/BnIs7Fwmq WJjNHglTmuD68frKSyvjNYmQkxCI3j/MPX4Q5SDvgVS5bVirVkSo/0okjjBbTSfA08W3 W7jjYJFAWkCfDatTPvYWakrwbziCD3ioPJfRwmK0N43NuacwwF/ZwMcaoYTmwzYRLnok cgLA== X-Gm-Message-State: AG10YOThW/n+V/zlUe1J+NXuLnqMBN0I5PVk+UUGnOVHvsIR7oEqGedTp7gLqmJqOZt0dlLB+UnChkW/rs9Wdw== MIME-Version: 1.0 X-Received: by 10.107.169.101 with SMTP id s98mr29216718ioe.126.1453908880336; Wed, 27 Jan 2016 07:34:40 -0800 (PST) Received: by 10.79.24.2 with HTTP; Wed, 27 Jan 2016 07:34:40 -0800 (PST) In-Reply-To: References: Date: Wed, 27 Jan 2016 19:34:40 +0400 Message-ID: From: Nik Ansell To: dtn-users@irtf.org Content-Type: multipart/alternative; boundary=001a1142e192592f91052a528adc Archived-At: Subject: Re: [dtn-users] DTN2 - Frequent Spinlock errors X-BeenThere: dtn-users@irtf.org X-Mailman-Version: 2.1.15 Precedence: list Reply-To: nikansell00@gmail.com List-Id: "The Delay-Tolerant Networking Research Group \(DTNRG\) - Users." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Jan 2016 15:34:44 -0000 --001a1142e192592f91052a528adc Content-Type: text/plain; charset=UTF-8 Hello All, I think I have found a fix for this, it is rather crude but seems to work consistently well after putting my test-bed through its paces using TCPCL, UDPCL and LTPCL. In my testing I added some debug code to SpinLock.cc which printed the Lock class properties to the dtnd.log file. I noticed that adding the debug code actually caused the spinlock errors to appear less frequently. I also noticed that when the spinlock error loop occurs, the lock_holder_name_ property of the Lock class (the thread that created the lock) was NULL. In some cases I also received an error (ASSERTION FAILED (is_locked_by_me()) at thread/SpinLock.cc:91) - this is the check at the start of the unlock function to ensure that a thread is not trying to unlock memory that was locked by another thread. The most common spin lock errors I experienced were: sending node: OpenFdCache::unpin waiting for spinlock held by null receiving node: BundleList::del_bundle waiting for lock held by null I developed two changes (below) to the oasys-1.6.0 build, which resulted in the ASSERTION FAILED error message and spin lock error message going away and much better overall performance on my test-bed. Then of course I reran make and make install to rebuild and install the required files. I have never written a line of c++ code in my life, nor did I know anything about linux spinlocks before my digging began. Therefore I am open to any suggestions to improve the code, or to learn of any issues I may have caused by implementing these changes. I appreciate this will potentially impact the performance of some systems, but as the pause only occurs while waiting to acquire a lock I suspect this will not be too noticeable, as it yields the thread (i.e passes the context back to the CPU) before pausing. 1. Add the following code to the start of the function, to allow locking or unlocking when lock_holder_name_ is NULL. Lock.h -> is_locked_by_me(): bool is_locked_by_me() { if (lock_holder_name_ == NULL) { return is_locked() && true; } 2. Add the following 11,000 microsecond pause, if the thread is waiting to acquire a lock: SpinLock.cc -> SpinLock::lock(): while (atomic_cmpxchg32(&lock_count_, 0, 1) != 0) { Thread::spin_yield(); // 0.011 sec sleep usleep(11000); Kind Regards, Nik On Mon, Jan 25, 2016 at 11:19 AM, Nik Ansell wrote: > I've done some more digging and have a few observations (below), I will > continue down this path but think I may be out of my depth already! :) > > 1. SpinLock.h: SpinLock::lock(): > a) The error message appearing in the log (below) has a null > lock_holder_name_ - Does this mean the lock is invalid (i.e perhaps the > calling thread terminated due to an exception) and can be safely released? > fprintf(stderr, "warning: %s is waiting for spin lock held by %s, which > has reached spin limit\n", lock_user, lock_holder_name_) > > b) The call to (Thread.h) Thread::spin_yield() (which calls either > thread_id_yield() or sched_yield() ). Is possibly failing in some cases - > not sure how to debug this one....Has anyone any idea if I can somehow > force the release of the lock if (as in above) lock_holder_name_ == NULL? > > c) The function atomic_cmpxchg32(&lock_count_,0,1) !=0 called is contained > within several Atomic-.h files e.g. Atomic-arm.h or Atomic-x86.h - > I am not sure how to confirm the correct header file was used during the > build, but will try to build oasys and dtn2 again using the > --host=arm-linux argument to configure to see if this makes a difference. > > 2. I have noticed there are some locking debugging functions available > (Thread::lock_debugger->add_lock / remove_lock) if > the --enable_debug_locking argument is passed during the build of oasys and > dtn2. I am considering enabling this to get some more info on the problem. > > Kind Regards, > Nik > > On Sun, Jan 24, 2016 at 12:05 PM, Nik Ansell > wrote: > >> Hello All, >> >> I am running some experiments to identify bundle delivery characteristics >> of DTN2 (+LTPlib), ION and IBR-DTN on the Raspberry Pi B2. Currently I am >> trying to identify reliable bundle delivery scenarios for all 3 DTN >> implementations, so I can then apply certain network simulation scenarios >> to observe how each DTN implementation and convergence layer behaves. >> >> The scope of my experimentation is below for info: >> >> ION: UDPCL, TCPCL, LTPCL >> DTN2: UDPCL, TCLCP, LTPCL - using LTPlib >> IBR-DTN: UDPCL, TCPCL >> >> I have determined reliable, repeatable scenarios for both ION and >> IBR-DTN, however DTN2 is producing produce a number of errors when using >> UDP, TCP or LTP. The errors also seem to appear randomly after any number >> of for loop iterations, e.g. (5,13,50,48,95,413, 567, 988, etc etc). >> >> Errors that cause dtnd to use 200% CPU and stop delivering bundles: >> warning: deliver_front is waiting for spin lock held by (null), which has >> reached spin limit >> warning: Bundle::del_ref is waiting for spin lock held by (null), which >> has reached spin limit >> warning: Bundle::is_queued_on is waiting for spin lock held by (null), >> which has reached spin limit >> warning: LinkBlockSet::find_blocks is waiting for spin lock held by >> (null), which has reached spin limit >> warning: BundleList::erase is waiting for spin lock held by (null), which >> has reached spin limit >> warning: Bundle::add_ref is waiting for spin lock held by (null), which >> has reached spin limit >> warning: ForwardingLog::get_count is waiting for spin lock held by >> (null), which has reached spin limit >> >> Errors: that cause dtnd to quit: >> LTP:ASSERTION FAILED (is_locked_by_me()) at thread/SpinLock.cc:91 >> >> Errors that seem to loop forever but not cause dtnd to quit or use 200% >> CPU (This error seems to happen consistently after sending 99 LTP bundles): >> /dtn/cl/ltp/sender error] Unable to create Sender LTP Socket in 5 seconds >> - retrying >> >> To send files I am calling dtnsource in a for loop as below, which allows >> me to adjust the time between each bundle transmission, or each block of 10 >> bundle transmissions. For UDP and LTP I am using bundles of 63K, for TCP I >> am using 1M bundles. I have tried several combinations for $WAIT (0-5 >> seconds) and $BULKWAIT (0-20 seconds), but cannot seem to find a working >> combination. >> >> for ((i=1;i<=END;i++)); do >> echo "($COUNT) dtnsource -s dtn://tx.dtn/a -d dtn://rx.dtn/g -b $SIZE" >> dtnsource -s dtn://tx.dtn/a -d dtn://rx.dtn/g -b $SIZE >> sleep $WAIT >> let "COUNT++" >> >> # Sleep for x secs after each 10 >> if (( $COUNT % 10 == 0 )) >> then >> echo "Sleeping for $BULKWAIT seconds" >> sleep $BULKWAIT >> fi >> >> done >> >> The daemon is stopped and restarted after each test, I have also tried >> deleting and recreating the berkley DB each time, but this does not seem to >> have much effect. I have also managed to recreate some of the errors on a >> virtual Ubuntu test-bed, specifically Bundle::del_ref. >> >> After reading through the oasys SpinLock::lock code, it looks like this >> is possibly an infinite loop that tries up to 1,000,000 times to get a >> lock, then resets the counter to zero and puts out a warning message. >> >> Has anyone had a similar experience or can suggest any troubleshooting >> tips? >> Any suggestions to workaround or fix the problem will be very gratefully >> received! >> >> Kind Regards, >> Nik >> > > --001a1142e192592f91052a528adc Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hello All,

I think I have found a fix f= or this, it is rather crude but seems to work consistently well after putti= ng my test-bed through its paces using TCPCL, UDPCL and LTPCL.
In my testing I added some debug code to SpinLock.cc which pri= nted the Lock class properties to the dtnd.log file. I noticed that adding = the debug code actually caused the spinlock errors to appear less frequentl= y. I also noticed that when the spinlock error loop occurs, the lock_holder= _name_ property of the Lock class (the thread that created the lock) was NU= LL. In some cases I also received an error (ASSERTION FAILED (is_locked_by_= me()) at thread/SpinLock.cc:91) - this is the check at the start of the unl= ock function to ensure that a thread is not trying to unlock memory that wa= s locked by another thread.

The most common spin l= ock errors I experienced were:

sending node: OpenF= dCache::unpin waiting for spinlock held by null
receiving node: B= undleList::del_bundle waiting for lock held by null

I developed two changes (below) to the oasys-1.6.0 build, which resulted = in the ASSERTION FAILED error message and spin lock error message going awa= y and much better overall performance on my test-bed. Then of course I rera= n make and make install to rebuild and install the required files.

I have never written a line of c++ code in my life, nor di= d I know anything about linux spinlocks before my digging began.
= Therefore I am open to any suggestions to improve the code, or to learn of = any issues I may have caused by implementing these changes. I appreciate th= is will potentially impact the performance of some systems, but as the paus= e only occurs while waiting to acquire a lock I suspect this will not be to= o noticeable, as it yields the thread (i.e passes the context back to the C= PU) before pausing.

1. Add the following code to t= he start of the function, to allow locking or unlocking when lock_holder_na= me_ is NULL.

Lock.h -> is_locked_by_me():

=C2=A0 =C2=A0 bool is_locked_by_me()
=C2=A0 =C2=A0 {

=C2=A0 =C2=A0 =C2=A0 = =C2=A0 if (lock_holder_name_ =3D=3D NULL) {
=C2=A0 =C2=A0 =C2=A0 = =C2=A0 return is_locked(= ) && true;
=C2=A0 =C2=A0 =C2=A0 =C2=A0 }
=
2. Add the following 11,000 microsecond pause, if the thread= is waiting to acquire a lock:

SpinLock.cc -> S= pinLock::lock():

=C2=A0 =C2=A0 while (atomic_= cmpxchg32(&lock_count_, 0, 1) !=3D 0)
=C2=A0 =C2=A0 {
=C2=A0 =C2=A0 =C2=A0 =C2=A0 Thread::spin_yield();
=C2=A0 =C2= =A0 =C2=A0 =C2=A0 // 0.01= 1 sec sleep
=C2=A0 =C2=A0 =C2=A0 =C2=A0 usleep(11000);

Kind Regards,
Nik

On Mon, Jan 25, 2016 at 11:19 AM, Nik Ansell= <nikansell00@gmail.com> wrote:
I've done some more digging and have a few o= bservations (below), I will continue down this path but think I may be out = of my depth already! :)

1. SpinLock.h: SpinLock::lock():=
a) The error message appearing in the log (below) has a null loc= k_holder_name_ - Does this mean the lock is invalid (i.e perhaps the callin= g thread terminated due to an exception) and can be safely released?
fprintf(stderr, "warning: %s is waiting for spin lock hel= d by %s, which has reached spin limit\n", lock_user, lock_holder_name_= )

b) The call to (Thread.h) Thread::spin_yie= ld() =C2=A0(which calls either thread_id_yield() or sched_yield() ). Is pos= sibly failing in some cases - not sure how to debug this one....Has anyone = any idea if I can somehow force the release of the lock if (as in above) lo= ck_holder_name_ =3D=3D NULL?

c) The function a= tomic_cmpxchg32(&lock_count_,0,1) !=3D0 called is contained within seve= ral Atomic-<CPUARCH>.h files e.g. Atomic-arm.h or Atomic-x86.h - I am= not sure how to confirm the correct header file was used during the build,= but will try to build oasys and dtn2 again using the --host=3Darm-linux ar= gument to configure to see if this makes a difference.

=
2.=C2=A0 I have noticed there are some locking debugging functions ava= ilable (Thread::lock_debugger->add_lock / remove_lock) if the=C2=A0--ena= ble_debug_locking argument is passed during the build of oasys and dtn2. I = am considering enabling this to get some more info on the problem.

=
Kind Regards,
Nik

On Sun, Jan 24, 2016 at 12:05 PM, Nik Ansell= <nikansell00@gmail.com> wrote:
Hello All,

I am ru= nning some experiments to identify bundle delivery characteristics of DTN2 = (+LTPlib), ION and IBR-DTN on the Raspberry Pi B2. Currently I am trying to= identify reliable bundle delivery scenarios for all 3 DTN implementations,= so I can then apply certain network simulation scenarios to observe how ea= ch DTN implementation and convergence layer behaves.

The scope of my experimentation is below for info:

<= div>ION: UDPCL, TCPCL, LTPCL
DTN2: UDPCL, TCLCP, LTPCL - using LT= Plib
IBR-DTN: UDPCL, TCPCL

I have determ= ined reliable, repeatable scenarios for both ION and IBR-DTN, however DTN2 = is producing produce a number of errors when using UDP, TCP or LTP. The err= ors also seem to appear randomly after any number of for loop iterations, e= .g. (5,13,50,48,95,413, 567, 988, etc etc).

Errors= that cause dtnd to use 200% CPU and stop delivering bundles:
war= ning: deliver_front is waiting for spin lock held by (null), which has reac= hed spin limit
warning: Bundle::del_ref is waiting for spin lock = held by (null), which has reached spin limit
warning: Bundle::is_queued_= on is waiting for spin lock held by (null), which has reached spin limitwarning: LinkBlockSet::find_blocks is waiting for spin lock held by (null)= , which has reached spin limit
warning: BundleList::erase is wait= ing for spin lock held by (null), which has reached spin limit
wa= rning:=C2=A0Bundle::add_ref is waiting for spin lock held by (null), which = has reached spin limit
warning: ForwardingLog::get_count is waiting for = spin lock held by (null), which has reached spin limit

Errors: that = cause dtnd to quit:
LTP:ASSERTION FAILED (is_locked_by_me()) at t= hread/SpinLock.cc:91

Errors that seem to loop forever but not cause = dtnd to quit or use 200% CPU (This error seems to happen consistently after= sending 99 LTP bundles):
/dtn/cl/ltp/sender error] Unable to cre= ate Sender LTP Socket in 5 seconds - retrying

To send files I am cal= ling dtnsource in a for loop as below, which allows me to adjust the time b= etween each bundle transmission, or each block of 10 bundle transmissions. = For UDP and LTP I am using bundles of 63K, for TCP I am using 1M bundles. I= have tried several combinations for $WAIT (0-5 seconds) and $BULKWAIT (0-2= 0 seconds), but cannot seem to find a working combination.

for ((i=3D1;i<=3DEND;i++)); do
=C2=A0 =C2=A0 ech= o "($COUNT) dtnsource -s dtn://tx.dtn/a -d dtn://rx.dtn/g -b $SIZE&quo= t;
=C2=A0 =C2=A0 dtnsource -s dtn://tx.dtn/a -d dtn://rx.dtn/g -b= $SIZE
=C2=A0 =C2=A0 sleep $WAIT
=C2=A0 =C2=A0 let &quo= t;COUNT++"
=C2=A0 =C2=A0=C2=A0
=C2=A0 =C2=A0 # Sleep for x secs after each 10
=C2=A0 =C2=A0 if (( $COUNT % 10 =3D=3D 0 ))
=C2=A0 =C2=A0 = then
=C2=A0 =C2=A0 e= cho "Sleeping for $BULKWAIT seconds"
=C2=A0 =C2=A0 sleep $BULKWAIT
=C2=A0 = =C2=A0 fi

done

The = daemon is stopped and restarted after each test, I have also tried deleting= and recreating the berkley DB each time, but this does not seem to have mu= ch effect. I have also managed to recreate some of the errors on a virtual = Ubuntu test-bed, specifically Bundle::del_ref.

Aft= er reading through the oasys SpinLock::lock code, it looks like this is pos= sibly an infinite loop that tries up to 1,000,000 times to get a lock, then= resets the counter to zero and puts out a warning message.

<= /div>
Has anyone had a similar experience or can suggest any troublesho= oting tips?
Any suggestions to workaround or fix the problem will= be very gratefully received!

Kind Regards,
Nik
=


--001a1142e192592f91052a528adc-- From nobody Wed Jan 27 07:41:30 2016 Return-Path: X-Original-To: dtn-users@ietfa.amsl.com Delivered-To: dtn-users@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id E08F21B3882 for ; Wed, 27 Jan 2016 07:41:27 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -0.55 X-Spam-Level: X-Spam-Status: No, score=-0.55 tagged_above=-999 required=5 tests=[BAYES_40=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-0.7, SPF_PASS=-0.001] autolearn=ham Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id zbccb-eXV8bW for ; Wed, 27 Jan 2016 07:41:26 -0800 (PST) Received: from mail-io0-f171.google.com (mail-io0-f171.google.com [209.85.223.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 114491B388F for ; Wed, 27 Jan 2016 07:41:26 -0800 (PST) Received: by mail-io0-f171.google.com with SMTP id d63so6003731ioj.2 for ; Wed, 27 Jan 2016 07:41:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:reply-to:in-reply-to:references:date:message-id :subject:from:to:cc:content-type; bh=q3S/LZBq1us/3GH+W5eVs6Mo1V8SbBLu6wCOx1DItA0=; b=gRnGq3u96IIHhhKQX2/W2k8H4P810TIdLL1dnch3wKkBTwhPEEcCUArG6gZfk2VPjg v//fBP9A691xa/dsNteFIsKZssS02k6SjQOL2a+sqGuNt4AERMM/jAHGAoHi5ipzE6h7 ZIB+9tJCKRbXLYe5xrWshx3bf9dHxaSsZLUT9pu6AaDoTiKwWDATDZQXpjZc3ZHjIjS+ p8WLS0e2iQVKrhPKixqOgl29G/02LMPrS4MBi18/CAW6S8T+deDjD/Vwvn/WRmp5smfQ 73YpWHg6Wmtv4OPdtRbdAAteQE1T+VAq0eIaniLyAr+T/pQCblLGUKcpmHW0ektNXj9d Xn/w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:reply-to:in-reply-to:references :date:message-id:subject:from:to:cc:content-type; bh=q3S/LZBq1us/3GH+W5eVs6Mo1V8SbBLu6wCOx1DItA0=; b=X5WIQz/0YRwpbDI6jQYYdScw2vI3jkJNSDBAqN9URpV4BJC+TsqDuklRBdjiVT9ZxX wtURmVyvE3PDfzWgU2hAmHK+5K+K3xWCWnnYxIKOTaMJFCzKZzC5KaPacyXP4vJ31beB afmC2ffeSqb5dZRj1LTyJ0608DXlsU/bTXrnSdXf8c0FFazj1QEFwPdCrxe/BOpwVnvA UXpczwJmngW0spdr8Eim5GyFaknRoEih5o9Wgv+lw3nDUsXzSTXWOi7L0YvNV/s8N/eg ZI9TA8MftLl/xlirdj7hIkyHt/VyCnJGQRcJcEPg23aDm2wqaxnvmZ8S4EnP74KioaJN ZcfQ== X-Gm-Message-State: AG10YOSwCicp+YJj6dBPfMDCZBzPJeMybN4yiICKtPqd8hazlYr+0u0xqLySp2ycmUfeRkPVMO1VkHvvv7HHGQ== MIME-Version: 1.0 X-Received: by 10.107.170.212 with SMTP id g81mr33312150ioj.44.1453909285115; Wed, 27 Jan 2016 07:41:25 -0800 (PST) Received: by 10.79.24.2 with HTTP; Wed, 27 Jan 2016 07:41:25 -0800 (PST) In-Reply-To: References: Date: Wed, 27 Jan 2016 19:41:25 +0400 Message-ID: From: Nik Ansell To: kevin prima Content-Type: multipart/alternative; boundary=001a11425b70799ea2052a52a2e2 Archived-At: Cc: dtn-users@ietfa.amsl.com Subject: Re: [dtn-users] Problem DTN2 daemon X-BeenThere: dtn-users@irtf.org X-Mailman-Version: 2.1.15 Precedence: list Reply-To: nikansell00@gmail.com List-Id: "The Delay-Tolerant Networking Research Group \(DTNRG\) - Users." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Jan 2016 15:41:28 -0000 --001a11425b70799ea2052a52a2e2 Content-Type: text/plain; charset=UTF-8 Hi Kevin, I have been doing some digging and managed to find a fix which works on my tested. Assuming your configuration files are correct, you may want to try my fix and see how you get on, you can always back-out the change if you back up the original files. My fix has been posted to the dtn-users forum, in response to my post on frequent spinlock errors, but I have copied the fixes into the email below FYI. After making the changes, simply re-run make && make install from the oasys directory. 1. Add the following code to the start of the function, to allow locking or unlocking when lock_holder_name_ is NULL. Lock.h -> is_locked_by_me(): bool is_locked_by_me() { if (lock_holder_name_ == NULL) { return is_locked() && true; } 2. Add the following 11,000 microsecond pause, if the thread is waiting to acquire a lock: SpinLock.cc -> SpinLock::lock(): while (atomic_cmpxchg32(&lock_count_, 0, 1) != 0) { Thread::spin_yield(); // 0.011 sec sleep usleep(11000); Kind Regards, Nik On Thu, Dec 31, 2015 at 12:17 PM, kevin prima wrote: > Dear all, > > I want to run 5 DTN node in 2 hop topology scenario with DTLSR routing, > source and destination node connected with dtn node 1,2,and 3 but couldn't > reach each other. > When I try to send ping packet use dtnping, sometimes it show error like > this in node 1, node 2, and node 3: > > warning: Bundle::del_ref is waiting for spin lock held by (null), which > has reached spin limit > STACK TRACE: 0x76f51c84 0x3eda8 0x3ed00 0x59444 > > The impact is node 1, 2 or 3 become unavailable. What should I do to fix > this problem ? > > Thanks, > Kevin > > _______________________________________________ > dtn-users mailing list > dtn-users@irtf.org > https://www.irtf.org/mailman/listinfo/dtn-users > > --001a11425b70799ea2052a52a2e2 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hi Kevin,

I have been doing some diggin= g and managed to find a fix which works on my tested.
Assuming yo= ur configuration files are correct, you may want to try my fix and see how = you get on, you can always back-out the change if you back up the original = files.

My fix has been posted to the dtn-users for= um, in response to my post on frequent spinlock errors, but I have copied t= he fixes into the email below FYI.
After making the changes, simp= ly re-run make && make install from the oasys directory.
=
1. Add the following code to the start of the function,= to allow locking or unlocking when lock_holder_name_ is NULL.
Lock.h -> is_locked_by_me():

=C2= =A0 =C2=A0 bool is_locked_by_me()
=C2=A0 =C2=A0 {

<= /div>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 if (lock_holder_name_ =3D=3D NULL) {<= /div>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0return is_locked() && t= rue;
=C2=A0 =C2=A0 =C2=A0 =C2=A0 }

2. Ad= d the following 11,000 microsecond pause, if the thread is waiting to acqui= re a lock:

SpinLock.cc -> SpinLock::lock():

=C2=A0 =C2=A0 while (atomic_cmpxchg32(&lock_count= _, 0, 1) !=3D 0)
=C2=A0 =C2=A0 {
=C2=A0 =C2=A0 =C2=A0 = =C2=A0 Thread::spin_yield();
=C2=A0 =C2=A0 =C2=A0 =C2=A0 // 0.011= sec sleep
=C2=A0 =C2=A0 =C2=A0 =C2=A0 usleep(11000);
=

Kind Regards,
Nik
=

On Thu, Dec 31, 2015 at 12:17 PM, kevin prim= a <sir.kevinprima@gmail.com> wrote:
Dear all,

I want to run 5 DTN node in 2 hop topology s= cenario with DTLSR routing, source and destination node connected with dtn = node 1,2,and 3 but couldn't reach each other.
When I try to send ping packet use dtnping, sometimes it s= how error like this in node 1, node 2, and node 3:

warning: Bundle::del_ref is waiting for spin lock held by (null), which h= as reached spin limit
STACK TRACE: 0x76f51c84 0x3eda8 0x3ed00 0x59444
=

The impact is node 1, 2 or 3 become unavailable. What sho= uld I do to fix this problem ?

Thanks,=
Kevin

_______________________________________________
dtn-users mailing list
dtn-users@irtf.org
https://www.irtf.org/mailman/listinfo/dtn-users

--001a11425b70799ea2052a52a2e2--