Subject: [sidr] Slides for "RPKI Over BitTorrent" presentation

From: Rob Austein <[email-masked]>
Date: Mon, 26 Mar 2012 19:33:05 +0200

For those who didn't get to see the end of the "RPKI Over BitTorrent"
presentation in today's SIDR meeting, the full slide deck is available
at

  http://www.ietf.org/proceedings/83/slides/slides-83-sidr-9.pdf

and should be relatively self-explanatory.

---

From: Danny McPherson <[email-masked]>
Date: Wed, 28 Mar 2012 20:33:24 -0400

i don't think the rsync scale issues surprise anyone that was paying attention.  If we're already considering new architectures, substrates, et al., here perhaps we shouldn't be so quick on the trigger for Standards Track work and move this and related "investigation" to the IRTF, or at least ensure they're only Experimental until broader experience is gained.

Looking at the charts you presented I can only imagine what will happen with 40K RPs and >1M objects (which might be a reasonable assumption if this were fully deployed today - and that's only focusing on routed number resources). 

-danny

On Mar 26, 2012, at 1:33 PM, Rob Austein wrote:

> _______________________________________________
> sidr mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/sidr

---

From: Christopher Morrow <[email-masked]>
Date: Thu, 29 Mar 2012 04:05:34 -0400

> i don't think the rsync scale issues surprise anyone that was paying attention.  If we're already considering new architectures, substrates, et al., here perhaps we shouldn't be so quick on the tri…

terry seemed interested (and had a draft from 08 that looked like an
alternate option for transport) in gathering some requirements and
options for transport...

I think in the end having more than one transport is good, having one
to start things off though is also good.

-chris

---

From: Rob Austein <[email-masked]>
Date: Thu, 29 Mar 2012 12:04:34 +0200

At Wed, 28 Mar 2012 20:33:24 -0400, Danny McPherson wrote:
> 
> i don't think the rsync scale issues surprise anyone that was paying
> attention.  If we're already considering new architectures,
> substrates, et al., here perhaps we shouldn't be so quick on the
> trigger for Standards Track work and move this and related
> "investigation" to the IRTF, or at least ensure they're only
> Experimental until broader experience is gained.

If you're talking about moving all of the existing SIDR protocols to
Experimental, that's a cheap shot and you know it.  Any protocol will
behave badly if misused.  In my opinion the RIRs are currently using
rsync badly, given the way they've chosen to set up their
repositories.  This is not an inherent property of the protocol, just
a bad configuration decision which could be fixed at any time.  It may
be possible to find or invent better protocols than rsync, but rsync
was a reasonable choice for initial deployment.

If you're just talking about RPKI over BitTorrent, the BitTorrent
experiment was just that, an experiment.  The slides say so, and
state, in so many words, that this was not a proposal to change the
SIDR protocol suite.

For the record, this presentation was originally targeted at the IEPG,
as a direct follow-up to multiple suggestions received at a previous
IEPG meeting that it might be interesting to see how BitTorrent works
in this environment.  I presented this at the SIDR WG meeting because
the chairs thought it might be of interest to the WG.

Several other people have made comments which pretty clearly indicate
that they either did not read or did not understand the slides.  If
the former, please do so; if the latter, my apologies, please ask.

> Looking at the charts you presented I can only imagine what will
> happen with 40K RPs and >1M objects (which might be a reasonable
> assumption if this were fully deployed today - and that's only
> focusing on routed number resources).

Thank you for making my point for me.  We can do much better than what
we're seeing right now, but the first step is understanding where the
problems are, which involves studying behavior and reporting results.
This is less likely to happen if every report is viewed as an occasion
for an axe-grinding contest, so perhaps we should focus on the
technical issues.

---

From: "Murphy, Sandra" <[email-masked]>
Date: Thu, 29 Mar 2012 12:54:10 +0000

>i don't think the rsync scale issues surprise anyone that was paying attention.  

I am not sure what part of the presentation you are talking about.  In the "day in the life" presentation, I saw only differences due to implementation, not differences due to scale.

And in the "rpki over bittorrent" presentation, the point was made that the implementation choice creates performance differences.

>perhaps we shouldn't be so quick on the trigger for Standards Track work and move this and related "investigation" to the IRTF, or at least ensure they're only Experimental until broader experience is gained.

There's a process in the IETF for protocols to go from Proposed to full standard as lessons are learned from operational experience.   The progress through maturity levels is a fundamental part of the IETF process.  You seem to suggest that new protocols should start out in Experimental "until broader experience is gained" - perhaps you should suggest an IETF process change.

Having one protocol for everyone to use was necessary for interoperability and rsync was chosen as the MTI.  Again, typical IETF process.

>Looking at the charts you presented I can only imagine what will happen with 40K RPs and >1M objects (which might be a reasonable assumption if this were fully deployed today - and that's only focus…

I don't imagine that 40K relying parties or even full certification will be any time soon.  In the meantime we have a MTI protocol for interop.  And the architecture is extensible so that other transports could be introduced as deemed necessary from experience.

Don't worry so much.  The sky is not falling here.

--Sandy

________________________________________
From: [email-masked] [[email-masked]] on behalf of Danny McPherson [[email-masked]]
Sent: Wednesday, March 28, 2012 8:33 PM
To: sidr wg list
Subject: Re: [sidr] Slides for "RPKI Over BitTorrent" presentation

If we're already considering new architectures, substrates, et al., here perhaps we shouldn't be so quick on the trigger for Standards Track work and move this and related "investigation" to the IRTF, or at least ensure they're only Experimental until broader experience is gained.

-danny

---

From: "Danny McPherson" <[email-masked]>
Date: Thu, 29 Mar 2012 09:07:32 -0400 (EDT)

> Experimental, that's a cheap shot and you know it.

I didn't say that.

> state, in so many words, that this was not a proposal to change the

I actually agree with the premise here, that there are scale and freshness
and distribution issues in the current model that need to be worked out.

> for an axe-grinding contest, so perhaps we should focus on the

This is not "axe grinding".  These designs are likely to considerably
influence how networks and routing architectures and support systems are
designed and operated - in the real world, not in a lab or 'pilot'.  I'd
like to get this right as much as you would (moreso, because my $dayjob
relies on it), and that should encourage us to fully understand the
long-term desirable objectives before stepping on the accelerator for full
standardization.  Whatever we design today should at least be able to
accommodate full deployment in the current system (even when focusing on
*routed* resource certification), which given the data you've presented at
the past coupled meetings it's already strained and there appears to be a
disconnect between those designing the RPKI and the initial folks captive
to operationalizing it (e.g., the RIRs).

Out of curiosity, what were the initial design / scale goals of the RPKI
team for aggregate object count, frequency of publication and access by
RPs?

-danny

---

From: Arturo Servin <[email-masked]>
Date: Fri, 30 Mar 2012 12:24:25 +0200

Rob,

	I think it is a bit premature to conclude that. Under the same data set I could conclude that your network, validator and research methodology is broken. But that is not the point, the point is that we may have a problem, let's try to solve it and to validate how bad it is.

	In another email Chris Morrow suggested to measure using different validator implementation and different locations, which I think it is a very good idea to see how bad we are and how we can improve.

	Meanwhile, as an repository operator I would love to hear your suggestion to improve. 

Cheers,
/as

On 29 Mar 2012, at 12:04, Rob Austein wrote:

>  In my opinion the RIRs are currently using
> repositories.

---

From: Carlos Martinez-Cagnazzo <[email-masked]>
Date: Fri, 30 Mar 2012 10:45:58 -0300

Hi,

>> rsync badly, given the way they've chosen to set up their
<all hats off, speaking only for myself>

If you provide us with some guidelines on how you believe the
repositories should be organized, who knows, we may be willing to try
them out!

Seriously, I think we all want the technology to succeed and to be
widely adopted. At least on that we are on the same boat. I can't help
but feel that we could be doing a much better job at cooperating.

<hats on>

The one time you emailed me about problems with our certs and RPKI, We
had them solved within a couple of days. I think we should try that
approach more often.

cheers!

Carlos

---

From: Rob Austein <[email-masked]>
Date: Fri, 30 Mar 2012 20:10:17 +0200

While I will admit to some astonishment that the following explanation
could possibly be news to long-time participants in this WG (given how
much time I've spent whining about this issue over the last five years
or so both in public and in private), let me quote from the slides:

    * How efficient [fetching RPKI repositories using rsync] is
      depends heavily on how the publication repositories are
      organized.

    * In an efficiently organized repository, filesystem hierarchy
      follows X.509 certificate hierarchy, so that one can pick up
      significant subtrees with a single rsync connection.

    * To date, the RIRs have chosen to deploy flat hierarchies where
      there is no relationship at all between filesystem hierarchy
      within the repository and certificate hierarchy.

To make that more concrete, here's an example.  Let's assume we have
the following trivial hierarchy: Bob and Betty are issued by Alice,
Carol and Carl are issued by Bob, Dave, and Dana are issued by Carol,
Dara is issued by Carl, and and all of these are hosted in a single
repository.

In an inefficient, "flat" repository, the publication points for
objects issued by these entities would look something like this:

    rsync://example.org/rpki/Alice/
    rsync://example.org/rpki/Betty/
    rsync://example.org/rpki/Bob/
    rsync://example.org/rpki/Carl/
    rsync://example.org/rpki/Carol/
    rsync://example.org/rpki/Dana/
    rsync://example.org/rpki/Dara/
    rsync://example.org/rpki/Dave/

In a hierarchical repository, the same publication points would look
more like this:

    rsync://example.org/rpki/Alice/
    rsync://example.org/rpki/Alice/Betty/
    rsync://example.org/rpki/Alice/Bob/
    rsync://example.org/rpki/Alice/Bob/Carl/
    rsync://example.org/rpki/Alice/Bob/Carl/Dara/
    rsync://example.org/rpki/Alice/Bob/Carol/
    rsync://example.org/rpki/Alice/Bob/Carol/Dana/
    rsync://example.org/rpki/Alice/Bob/Carol/Dave/

Assuming top-down tree walk (the normal case), retrieving objects
issued by this set of entities takes eight rsync connections with the
flat repository, as opposed to one rsync connection with the
hierarchical repository.

In practice one might want a slightly more complex structure to limit
the size of individual directories, but it doesn't matter so long as
the filesystem hierarchy is organized in such a way that picking up
an issuer's publication point picks up a non-trivial number of its
subjects' publication points automatically.   It doesn't have to be
perfect, just has to do enough better than the flat model to amortize
the cost of setting up and tearing down the rsync connection over a
significantly larger number of files.

This is not about PKI, it's purely an rsync efficiency issue.

Presumably there are scaling limitations to the hierarchical approach,
but anecdotal evidence among the people I've asked ("I tried ... and
it worked") suggests that, if the underlying networks and filesystems
are in good shape, a single rsync connection ought to be able to
handle up to at least 10,000 small files, perhaps a lot more than
that.  Note that this is just talking about rsync itself: mileage
might vary significantly if the underlying networks or filesystems are
seriously broken.  Also note that these anecdotal estimates have not
been tested in any rigorous fashion as far as I know, so that's
another entry on my list of things we ought to be measuring.

Hope this helps to clarify the change I've been suggesting.