Subject: Re: [sidr] sidrSlides for "RPKI Over BitTorrent" presentation

From: Wes Hardaker <[email-masked]>
Date: Wed, 23 May 2012 18:22:33 -0700

Rob Austein <[email-masked]> writes:

> For those who didn't get to see the end of the "RPKI Over BitTorrent"
> presentation in today's SIDR meeting, the full slide deck is available
> at
>
>   http://www.ietf.org/proceedings/83/slides/slides-83-sidr-9.pdf
>
> and should be relatively self-explanatory.

I've been thinking about this a lot lately, as it'll certainly be a
challenge to ensure everyone is up to date.  It doesn't matter whether
you "should have pulled" or whether "you pull right when you get a
request".  Either way, there is an underlying data distribution problem.

Most importantly, every bit of the repository needs to pretty much get
everywhere.  Rdist works well for trying not to duplicate data.
Bittorrent works well for sharing the load, but either requires a lot of
bittorrent start files (whatever they're called), which then becomes
hard to syncronize; or a small number of bittorrent files (one per
aggregation point) that indicate you need to refetch and entire .zip
file even if you already have most of it anyway.

But the real underlying problem is what I said above: every bit needs to
get efficiently to every place, ideally at time of initial publication.

This sure smells to me like a solution in need of multicast (or, more
accurately, "deployed multicast") with a fall back to something like
bittorrent for bootstrapping or when you've missed session data and
can't recover.  Multicast as a data stream would be much more efficient
is collecting updates, although there are still issues that would need
to be worked through like "how do you know you heard everything".  Oh,
and it only works if you multicast deployed to your site of course.
-- 
Wes Hardaker
SPARTA, Inc.

---

From: heasley <[email-masked]>
Date: Thu, 24 May 2012 13:18:31 +0000

> get efficiently to every place, ideally at time of initial publication.

I have probably have missed it, but has an IRR distribution protocol-like
mechanism been discussed?  See Merit's IRRd documentation, but simply its
DNS serial number-like with record "add" and "delete" functions.  I would
expect this to be relative low impact and, at least with Merit's IRRd, to
be low maintenance.

And, regarding the comment about NNTP in the slides, I believe that is
false.  If one had a server used solely for sidr, rather than sidr +
alt.whatever.stream.of.garbage, I believe it would be found to be very
low maintenance.  I haven't run a server for about 15 years, but I never
had trouble with the software and certainly the software has likely
improved rather than degraded.  The problems were always hardware (disk)
failures and the volume.

> and it only works if you multicast deployed to your site of course.

please, we do not want multicast as the only solution.  does your noc know
how to debug multicast forwarding problems?

---

From: Wes Hardaker <[email-masked]>
Date: Thu, 24 May 2012 06:36:51 -0700

> how to debug multicast forwarding problems?

I didn't say only :-) . In fact, I pointed at bittorrent as a backup.  I
simply said it would be more efficient when sending out *new* changes.
-- 
Wes Hardaker
SPARTA, Inc.

---

From: heasley <[email-masked]>
Date: Thu, 24 May 2012 13:47:19 +0000

> simply said it would be more efficient when sending out *new* changes.

understood; i'm registering/re-enforcing that if that is developed it must
not be the only mathod.

---

From: Wes Hardaker <[email-masked]>
Date: Thu, 24 May 2012 12:44:02 -0700

> understood; i'm registering/re-enforcing that if that is developed it must

Agreed.  If we can't do it over carrier pigeon, we have no real backup.
-- 
Wes Hardaker
SPARTA, Inc.

---

From: "John G. Scudder" <[email-masked]>
Date: Thu, 24 May 2012 16:13:16 -0400

Wes,

> file even if you already have most of it anyway.

I'm far from an expert on Bittorrent but I believe this is not right. A .torrent file (what you called a "bittorrent start file") can contain contain a list of files to be transferred, there's no mandate to zip them into a single file first. Presumably the BT protocol is smart about not transferring files you already have locally -- that's kind of the whole point. Rob's "operational overview" slide makes it clear he's imagining operating in the collection-of-files mode. 

Probably there are some issues with this mode of operation too -- if you list every single RPKI object individually then the size of the .torrent file scales linearly in the size of the unauthenticated/ tree. This seems likely to suck since I am guessing BT's normal operating regime is to transfer small numbers of large files rather than potentially vast numbers of relatively tiny files.

The zip files Rob's slides talk about just contain the .torrent file and manifest (which is also O(N) in the size of the tree, sigh).

Speaking purely as a Bittorrent amateur + not the author of the slides!

--John

---

From: Christopher Morrow <[email-masked]>
Date: Thu, 24 May 2012 17:11:55 -0400

On Thu, May 24, 2012 at 4:13 PM, John G. Scudder <[email-masked]> wrote:
> I'm far from an expert on Bittorrent but I believe this is not right. A
> .torrent file (what you called a "bittorrent start file") can contain
> contain a list of files to be transferred, there's no mandate to zip them
> into a single file first. Presumably the BT protocol is smart about not
> transferring files you already have locally -- that's kind of the whole
> point. Rob's "operational overview" slide makes it clear he's imagining
> operating in the collection-of-files mode.

per publication-point I think? So you have to collect N-thousand of
these torrent files, and keep N-thousand torrent jobs running and keep
state on N-thousand remote torrent files changing over time... some of
this could be alleviated in many ways, worst case though is that.

> Probably there are some issues with this mode of operation too -- if you
> list every single RPKI object individually then the size of the .torrent
> file scales linearly in the size of the unauthenticated/ tree. This seems
> likely to suck since I am guessing BT's normal operating regime is to
> transfer small numbers of large files rather than potentially vast numbers
> of relatively tiny files.

well, folk like fedora/ubuntu/etc offer full distributions over
bittorrent ... I've not done a find . | wc -l on a system like this in
a long time, but 4-8 gb over bittorrent for a few thousands of files
seems to work fairly well for them.

> The zip files Rob's slides talk about just contain the .torrent file and
> manifest (which is also O(N) in the size of the tree, sigh).

maybe looking at a bittorrent-like solution for intra-as distribution
is interesting though? though that also seems (to me) to fall into
'local implementation' land, from the vendor/arms-dealer that sold me
the solution.

> Speaking purely as a Bittorrent amateur + not the author of the slides!

me too, as an amateur of bittorrentz I mean.

-chris

> --John
> _______________________________________________
> sidr mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/sidr

---

From: Rob Austein <[email-masked]>
Date: Thu, 24 May 2012 18:14:09 -0400

> >> file even if you already have most of it anyway.

The .zip file is just a wrapper around a .torrent file and an
associated text manifest (which is mostly present to provide a
stronger checksum than BitTorrent provides).

.zip files over HTTPS are the control channel.  RPKI data files over
BitTorrent are the data channel.  Confusing the two will lead to brain
trauma.

> per publication-point I think?

No.  It's one torrent per gatherer.  More precisely, one torrent at
time per gatherer, as each gatherer generates a new torrent when it
thinks it has a new batch of data that's worth sharing.

> So you have to collect N-thousand of these torrent files, and keep
> N-thousand torrent jobs running and keep state on N-thousand remote
> torrent files changing over time... some of this could be alleviated
> in many ways, worst case though is that.

One torrent per gatherer at a time.  The mechanism outlined in the
slides uses HTTPS polling to distribute the .zip files containing the
.torrent files.  The HTTPS service used for this can be replicated via
the usual techniques.

So the clients poll the HTTPS servers to get the .torrent files, then
use the .torrent files to retrieve the ten zillion little object files
from the BitTorrent cloud.  Note that BitTorrent is clever enough not
to retrieve files it already has, so in effect this produces a similar
effect to rsync (client only pulls what it needs), without imposing
the same kind of server load.