Subject: Re: [sidr] Scaling properties of caching in a globally deployed RPKI / BGPSEC system

From: Eric Osterweil <[email-masked]>
Date: Thu, 6 Dec 2012 12:55:12 -0500

Sorry for the delay... $dayjob...

On Nov 27, 2012, at 6:17 PM, Sriram, Kotikalapudi wrote:

> This email is somewhat long. 
> So I also have a tech-note (pdf) version of this (including the table, figure, etc.).
> If you prefer to read it in the tech-note format, you may skip the rest of this email and click:
> http://www.nist.gov/itl/antd/upload/rpki-rsync-delay-technote.pdf

My comments below are taken from you document (as you comment above advised):

General comment:
I think a lot of the supporting information in this document seems to be derived from qualitative estimation.  I worry about this kind of ``guestimation'' from an engineering perspective, and in deference to those people that might have to operate this kind of infrastructure based on guesswork.

Specific comments:
- In Section A you discuss the merit of different sets of performance numbers: It is true that after Randy spent months presenting and supporting those performance numbers at multiple venues, he has now come back on the list with anecdotal numbers that are strangely very different.  However, I have to point out that his anecdotal snapshot numbers don't seem to be quite as credible as his _own_ longitudinal study's numbers.  If, for example, he were to do a similar longitudinal study and issue new numbers over the same protracted timescale, it would make a lot of sense to use those too.  However, (form the standpoint of both an engineer and a scientist) snapshot measurements do not constitute the same amount of information as long-term studies.  Tim's numbers are also point numbers, I'd like to find a way to factor them in as additional data points, but it is, again, hard to compare apples to oranges (one measurement vs. aggregated performance over a longitudinal study).  Remember, Randy's original numbers are the results of measuring over a long period, they include operational eventualities (like outages and performance hiccups, etc.), instead of a self-proclaimed confirmation bias.

- In Section A you present ``best case'' numbers.  First, recall that the numbers you've based your best case performance on are snapshots (at best), and appear to be somewhat anecdotal.  Our tech note illustrated that Randy's previous numbers are averages across 10 repositories, AND their average values were actually all reasonably consistent with each other in the performance (within the same order of magnitude, modulo 2 smaller repos whose numbers were basically averaged out).  Also, I have never seen ``best case'' numbers used as the justification of performance, or operational soundness.  For example, in complexity theory, one often speaks about worst-case performance (big-O notation).  In general, one wants to know how bad things can get.  Thus, you might be able to say something like, ``the best this thing could ever possibly run is x, y, z...'' But we have a long way to go before we can say we've quantified the optimal theoretical performance.  Again, from complexity theory, if you want to (for example) describe average complexity (big-omega), that is a lot harder to do than big-O, but would be very useful, and I would applaud the initiative.  Is that what your document is trying to do?  I guess, at best, it seems like we might be pursuing opposite ends of the spectrum?  I was trying to tell people that we have to be ready for operational issues to provide a worst-case estimate.  It seems you were trying to illustrate that the system's peak performance is still quite slow.  Though, I admit that I am having trouble following your methodology (perhaps your goal is loftier than mine). 

- Later in Section A, at the top of page 2: I (obviously) see the arithmetic effect of lowering the multiple, but we ought to ask _why_ the other numbers are so high?  What do _you_ see as the difference (since both were taking by the same person)?  What I see is that the original numbers include real operations, real latencies, etc. and they are taken over a long period.  We (as engineers) have to respect that.  By the way, these are _not_ my numbers.  You likely ought to cite the work you are referencing properly, since calling these ``Eric's numbers'' isn't quite right.  A citation lets readers find out the origin of cited information properly. ;)  This is, I think, especially true in this case where the actual presenter of the the original numbers is the _same_person_ that is now trying to back away from them.  This is, scientifically speaking, an important point.

- Second para on page 2:
	``normal (steady-state) operation'' citation needed.
	``rather rarely'' citation needed.
	``would be more like seconds or minutes'' citation needed.
These speculations sound like biased views, but my concerns might be totally debunked with proper citations.  As someone on the list pointed out, this is engineering.  We need evidence and measurements, and _not_ intuition, guesswork, and experimental bias.

- Section B: third para on page 2:
	``84% of all ASes are stub ... 16% are non-stub'' citation needed.

What is this statement based on?
	``The stub ASes are likely to only run simplex bgpsec and they would likely contract out publication'' 
Also, each hosted instance that is _not_ flat still needs infrastructure objects (GB, Mft, CA, etc) + DNS domain name + Also a noteworthy point about your hosted model ( ;) ) is that hosted models remove resource holders' abilities to manage their own information directly.  That is, if I have you host my info, and you have my private key, you likely cannot give that key to me for me to make my own operational changes (if you have it in an HSM, that private key cannot be shared). So, I (as a resource holder) cannot author changes to my (say) ROA, or create new router EE certs unless _YOU_ do it for me.  So, right now, that would require conversations like, ``hey ARIN, can you generate a new ROA for me since I have a new feasible origin?'' (no big deal), but in the future that would be, ``hey ARIN, can you create a new EE cert for my router, and send me both the pub/priv key pair so I can configure a new router (or $n$ new routers)?''  I'd hazard that _this_ is a much bigger deal, especially with the need for a private key exchange.  One more complexity to note. ;)

- Section C: You comment that there could be a single router key for an entire AS is simply _not_ operationally feasible.  Any operator can feel free to open this discussion again, but I think we've covered that ground here quite a few times.  In fact, I think Randy was the most recent person to hit that nail on the head (re: his 2 million router keys comment).

- Section C: Your estimates on routers per AS... Wow... So, I see some other people have taken you to task on those estimates.  I'll simply add that I think your numbers are wrong, and even Randy pointed to 1 million routers as a good starting point.

- Section D: You mention that you (``We?'') had trouble following the notion that updates will only be propagated to the whole system at about 2*T (where T is the update period).  I am struggling to find a citation now, but will keep looking.  In the meantime, here's a thought exercise that illustrates the point: Suppose you take $n$ second to crawl the rpki.  If you start polling at $t$, but at $t + \epsilon$, the first cache updates its entries.  Then you will have to wait until $t + n$ before you get to _start_ crawling the rpki again.  However, if you then pick up the new entry, you may have to expect that someone's caching software will have to finish its crawl before your local cache has all of the rpki. Why wait?  Well, if you have a hierarchical cache system (like the one Randy describes in his preso), or any large multinational AS would likely need, then the data authorities (i.e. not the ones caching) cannot be sure _everyone_ in the Internet has the fresh values until caches like this have propagated out.  Thus, $t + n + n$.  This makes your response time $2 \times n$.  I will find a citation in the literature and send it out.

In the meantime, we have been updating our tech note with a lot of great feedback and will be releasing a new revision soon.

Eric

---

From: Eric Osterweil <[email-masked]>
Date: Thu, 6 Dec 2012 12:57:08 -0500

On Nov 30, 2012, at 6:37 PM, Randy Bush wrote:

>> Actually, Randy, engineering with the upper bound is pretty typical.
> 
> from your document
> 
> "The calculations in this document, while just candidate representations
> of RPKI, project a very long lower-bound on the potential time needed
> for RP caches to gather all objects in the RPKI"

lol... Good point: computing the upper bound on time, using the lower bound on size... Different subjects, both conservative estimates, but my bad for being unclear. :)

>> The counter tends to be viewed as quite amateurish.
> 
> personally, i would be professionally embarrassed to have my name on
> such wild assed cabbage throwing as that document.

That's very constructive, thank you!

>> codifying a system's design after building an implementation, and then
>> retrofitting requirements to that design so that we can _then_ back
>> into a threat model that befits a system's capabilities is typically
>> frowned upon by engineers.
> 
> let's be honest here.  the threat model was done in the late '80s.  the
> base of the design in the early '90s.  we're just slogging through the
> saussage machine.

Um... Kind of a lot has changed since then, and I _believe_ the design was rejected back then too...  I think it might be less professionally embarrassing to ask, what needs to change to make this viable? ;)

ymmv,

Eric

---

From: Eric Osterweil <[email-masked]>
Date: Thu, 6 Dec 2012 12:57:12 -0500

Hey Tim,

My turn to apologize for the delay.  I have taken your very excellent feedback and tried to address it in a new revision of the tech note.  That should be coming out soon.  Please do let me know if it seems off again!

Eric

On Nov 23, 2012, at 11:26 AM, Tim Bruijnzeels wrote:

> Hi Eric,
> 
> Sorry for the late response..
> 
> With regards to some of your comments regarding requirements and ideas for alternative deltas, some responses inline below, but other than that see the thread where both Bryan Weber and I talk about our ideas. This is somewhat separate from the questions about "how a fully deployed RPKI tree would look like", and "what churn we might see" and "how many connecting RPs, and how frequent". All that can feed into requirements, but is in essence another discussion, so I suggest we keep them separate.
> 
> 
> On Nov 16, 2012, at 10:44 PM, Eric Osterweil <[email-masked]> wrote:
> 
>> 
>> On Nov 16, 2012, at 10:45 AM, Tim Bruijnzeels wrote:
>> 
>>> Hi,
>>> 
>>> Some more comments on the numbers and formula..
>>> 
>>> On Nov 15, 2012, at 5:36 AM, Arturo Servin <[email-masked]> wrote:
>>> 
>> 
>> <snip>
>> 
>>> Apart from any signed objects it may publish, every CA typically has:
>>> = 1 certificate published by its parent
>> 
>> But, as I asked Arturo, would we expect to have a CA from each parent (i.e. each RIR that an org may allocations from)?  While this may often still just be 1, it seems important to note, no?  THat was one reason we had for breaking it apart.  I'm more than willing to believe that I'm wrong here, but I'd like to understand how. 
>> 
>>> = 1 manifest
>>> = 1 CRL
>>> = 1 GB record (as Arturo said not widely deployed, but let's throw it in for full deployment)
>> 
>> How does the RPKI certify ASN allocations?  This is needed to certify router EE certs (they are tied to an ASN, no?).  Such an AS cert would add another 1 to the above and make it 5, right?  Otherwise, how does a forward signing peer associate itself with the ASN, instead of some prefix?  The ASN isn't necessarily allocated by the same RIR as any of the allocations, so IP allocation and ASN allocation are orthogonal, no?
> 
> Okay, let me rephrase.. it gets confusing when talking about organisations as CAs, and CA certificates: i.e. certificates that have the CA bit set and can sign and publish other certificates, signed objects etc.
> 
> We're talking 4 objects for each CA Certificate:
> = 1 CA Certificate with IPv4, IPv6 and/or ASN
> = 1 mft
> = 1 CRL
> = 1 GB
> 
> An organisation may have more than one CA certificate:
> = if it has different parents and get certificates from more than one
> = if it has 'minority space'. For example if one of our members has resources for which another RIR is the majority holder, then this is not on our normal certificate. We get a certificate with all such resources signed by the other RIR, and then we use this to sign an additional certificate for this member. Certificates can have only one parent, so we can not merge this.
> 
> Regarding the second point: I haven't done detailed analysis on our data yet. This does affect at least a portion of our members, so the average number of CA certs per member organisation is more than 1. I expect that it will not be a *lot* higher, but like I said I haven't analysed the data yet.
> 
> Having said that, it seems that the relative component of these, in a sense, boilerplate objects of the total count, are of a different order than the amount of expected ROAs and router certificates.
> 
>>> 
>>> So that's 4 objects.
>>> 
>>> During a key roll the CA will have the following additional objects:
>>> = 1 cert published by the parent
>>> = 1 manifest
>>> = 1 CRL
>>> 
>>> Making 7 objects. But typically not all CAs roll at the same time.
>> 
>> Unless it is an algorithm rollover, and that is expected to last for years (iirc).  Then this set would be doubled (plus double the numbers below), right?
>> 
> 
> I did not consider algorithm roll over, but I think you're right.
> 
>>> 
>>> The number of signed ROAs and Router certificates does,
>> 
>> And EE certs.  While 1:1 with ROAs, they require additional (very different) processing, esp if you start down the road of HSMs.  So, we claimed this additional operational requirement means that even if you double up on the downloads, those are still two separate objects.  You have to manage EE rollover keeping crypto material the same or changing it, depending on details of the ROA, etc.  That won't come for free, and (again) needing HSMs makes this a big deal.  So, we really felt it was important to call this complexity out by counting each. 
> 
> I don't understand the use of EE certs and HSMs here. And I don't see how this can significantly raise the object count.
> 
> In the rpki EE certs can only be used to sign rpki signed objects and they are embedded in those objects, not published separately.
> 
> All keys in our online system are protected by HSMs. The keys in EE certificates even more so: they are generated in an HSM, used only once, and then forgotten. In the online system the HSM protects against key theft if an attacker somehow gets access to the database/file system.. The will not be able to export the keys without access to a quorum of key cards that protect the internal key of the HSM.
> 
> If you want to use an *offline* key protected by an HSM, then you would have an additional CA cert (and mft, crl and gb). This is what we do with our TA at least:
> 
> TA (offline) -> signs CA cert for online use => signs member CAs
> 
> The idea being that if the worst happens to our online system, we can rebuild and re-issue using the more secure offline key. The hosted members don't have their own offline key, they are protected by the same one.. Non-hosted CAs may want to use an intermediate offline certificate. This adds some overhead: 4 objects per offline key (CA cert, mft, crl, gb). 
> 
>>> in my opinion, not depend on the number of CAs, but:
>>> = ROA -> The number of announcements seen in BGP * some aggregation factor (1 / # average prefixes on one ROA)
>> 
>> I, pretty much agree with this (as I think the tech-note said).  I do, however have to note that with MOAS, you need multiple ROAs.  Small point, but worth stating. :)
> 
> Yes, a ROA can have one ASN only, but multiple prefixes.
> 
> We aggregate prefixes for the same ASN as much as we can on a single ROA. So far we are managing a factor of around 3 prefixes per ROA. Lacnic is similar. The other RIRs seem to aggregate a bit less at this time.
> 
> See here:
> http://certification-stats.ripe.net/
> 
> 
>> 
>>> = Router certs -> The number of ASNs * the number of keys for each ASN
>> 
>> \times The number of eBGP speakers you mean, right?
> 
> Yes. My model assumes that this number of speakers can related to the number of ASNs.
> 
> Randy suggested a number of physical bhp sec speaking routers and that they may have two keys. That may well be a better model.
> 
> 
>> 
>>> 
>>> So I think a better model would be to say:
>>> 
>>> number           object
>>> #CA                 CA cert
>>> #CA                 MFT
>>> #CA                 CRL
>>> #CA                 GB
>> 
>> Ack, we just estimated this as the # of SIAs, and then varied it from 5 to 42,000.
> 
> 5 here would mean the current RIR TAs without any other signed content.
> 
> The total object count for this depends on the number of CA certs. See my previous best estimate below.
> 
> 
>> 
>>> 
>>> #prefixes * X  ROAs
>> 
>> Yeah, but we didn't guestimate the $X$ value.  It sounds like we should, but is there any data we can use to do so?
>> 
>>> 
>>> #ASNs * Y      Router certs
>>> 
>>> Ototal  = 4 * #CA + #prefixes * X + #ASNs * Y
>> 
>> Re: the above, I think this would be 
>> 
>> O^Total = 4 * #CA + #ASNs + #prefixes + X * #prefixes + Y * #ASNs
>> 
>> We had called out the need for AS EE cert (which was not in the equation you outlined), and we felt it was important to not omit EE certs (if for no other reason than the operational complexity they bring).
>> 
>>> 
>>> As for the numbers.. this is a bit of a guessing game.. we just really don't know at this time. We can take our best guess, but should keep in mind that our best guess is probably off, and needs re-evaluation in years to come.
>> 
>> 100% agree.  That is why we called this a back-of-the-envelope calculation and are totally seeking feedback, and are absolutely interested in pushing revisions as things evolve.
>> 
>>> 
>>> #CAs
>>> 
>>> If this were the total number of current members for all RIRs this number would be around 40k. However, there are also PI users that are not direct members of the RIRs, and some members will delegate some of their resources further. For reference I believe that in the RIPE region we have around 25k PI prefixes. I expect that a lot of the organisations that hold these resources will be happy to let a sponsoring RIR member (LIR in our region) manage their ROAs. But not all.. So I think that in a full deployment world this number may be significantly bigger. If anyone has any ideas on this, please chime in… Going on nothing more than gut feeling I would say the total could be in the order of: 
>>> 
>>> = 40k RIR members plus 40k self managing PI holders / children of members?
>>> 
>>>   80k. 
>> 
>> Really?  I had been thinking that this number was tied to the origins, but I can see your logic.  It would great to try and find a way to estimate this, so I'd like to echo your request for anyone with info to chime in.
>> 
>>> 
>>> #prefixes and 'X'
>>> 
>>> The number of announced prefixes is still rising. Currently we are nearing 500k.
>>> Worst case X is 1, meaning every ASN - prefix combination has its own ROA.
>>> 
>>> In reality this number will be lower because we can and do aggregate. But not all implementers will do this. There is something to be said for *not* fate-sharing ROAs for different prefixes from the same ASN. Also, most our members are fairly small, and they do not do huge numbers of announcements individually. Our current aggregation rate -- and we really try... is:
>>>  792 ROAs / (2197 IPv4 + 468 IPv6 prefixes) = 0.3 ROA/prefix
>>> 
>>> (see: http://certification-stats.ripe.net/) 
>>> 
>>> For scalability assessment I am not sure though that a factor of (1/0.3=) 3 between this level of aggregation, which seems best case, and no aggregation, worst case, is that significant in the big picture. I will use the mean of these two numbers below..
>> 
>> Fair enough.  I can certainly see the logic here, but if we wound up with a good way to do the estimation that would be even better, no? :)
>> 
>> <snip>
>> 
>>> In principal I like the approach to turn this around and define what an acceptable average delivery rate would be given the total number of objects and the maximum sync time. But on the other hand this can lead to rejecting any infrastructure we could come up with.. just set the goals high enough and nothing will be enough. So I think we should be cautious here. If there are absolute objective minimal requirements it would be good to know them. But other than that it may be best to be pragmatic about it and turn this back.. try to think of other ways and see if they actually perform significantly better..
>> 
>> I can respect this concern, but we really do have to deal with any systemic/complexity/operational/etc. facets of the system that we have designed.  We need to know how this design is going to behave if we are going to enshrine it.  For example, the above calculations, and ours, and any derivative formulation would make revoking a key within an hour seem to be impossible.  Is it a day, a week, a month, etc?  That may still be unclear (depending on how we model this), but how can we go any farther forward without taking a careful look at this design?  We must know if it meets our requirements, and I think measurements like these help tell us how feasible this will all be.
>> 
>>> Bearing with the document, if we take the current rsync repositories as a starting point to see where we are heading without changes:
>>> = It should be noted that fetch times depend on lay-out, hierarchical layout, allowing recursive, saves a lot of overhead (and latency)
>>> = We *do* recursive fetches on all current RIR repositories (yes, we hacked in which base directory to use)
>>> = Testing on my laptop I typically see fetch times of around 20ms per object, not 628ms
>> 
>> To be perfectly fair, I just used the #s I found form the BGPSEC design team's measurements.  I was very hesitant to use any particular set of numbers here.  As I'm sure you know, there are opinions about rsync, what it looks like under load, with asynchrony, repos' operational uptimes, restarting because of changes in the repository (I think that was something from your preso), etc.  I think you likely know (much better than me) that if a repo is under heavy load, churn, or just having an outage, that can cause cache's sync time to suffer.  Hence, I really liked getting real operational data from non-lab measurements.  I actually really feel that data is quite useful.  Consider this, you all (who are running these things) are the experts.  If we wind up w/ ~42,000 repos, they will _not_ all be run as well as you run them.
>> 
> 
> I understand your concern about many repos, some of them small and possibly not well managed.
> 
> But the number of repos is not 1:1 the number of organisations acting as CAs.
> 
> We currently see 5 different repositories for the hosted solutions provided by the RIRs. Non-hosted is being worked on, but so far not done in the production environment. When this arrives, some non-hosted people will want to do their own publishing, some others will use a bigger repository, for example with their RIR, or it may be that 3rd parties will start providing this service. In any case 42k repositories seems a bit much, though more than 5 is very likely.
> 
>>> 
>>> A full fetch based on today's numbers would then take 1M * 20 ms = 20k seconds = 330 minutes = 5.5 hours = 0.25 days.
>> 
>> Sorry, but I really think this has some problems.  First, the numbers I see in the cited preso are way larger than this, and just for the object sets we see today.  So, I have to say that this calculation doesn't seem to jive with Randy's numbers.
>> 
> 
> As you can read in my other emails I agree in general. I don't think rsync can scale to the levels we need.
> 
> 
> But the numbers on today's *small* repositories can be improved a lot by making them hierarchical, or if your validator happens to know that it can use a higher directory. We do that last thing, rcynic does not. We made our repository hierarchical though.
> 
> I think Randy's numbers may be outdated and represent the totals for our old *flat* rsync repo. This adds a huge amount of latency and setup overhead.
> 
> The numbers I got was by just running our validator* and watch the log for lines like:
> 16:51:12,883 INFO  Prefetching 'rsync://rpki.ripe.net/repository/'
> 16:52:06,980 INFO  Done prefetching for 'rsync://rpki.ripe.net/repository/'
> 16:52:26,189 INFO  Finished validating RIPE NCC RPKI Root, fetched 4447 valid Objects
> 
> So the crypto took 20 seconds. The fetching of 4447 objects took 54 seconds => 12 ms/object. For Lacnic: 280 objects in 5 seconds => 17 ms/object from my laptop on a wireless in Amsterdam to South America.
> 
> 
> *: http://www.ripe.net/lir-services/resource-management/certification/tools-and-resources
> 
> 
> 
>>> 
>>> So this is quite a bit faster. Eric, Danny how did you get to the numbers in table 3? Like I said: I just got some times from validation runs done on my laptop. We do collect data from our validators 'in the wild' though.. unfortunately the format of this data (we store a *lot*) is such that it will take some time for me to dig out more representative numbers. More time than I have now, but I will try….
>> 
>> The citation at the end of the document comes from Randy.  It shows (MRT?) graphs with these numbers on them.  This doesn't include issues like repos under load from 42,000 caches DNS, outages, etc.  I think the numbers taken from his preso are very charitable, and they are actual measurements.  Moreover, his own experiments showed that replicating this all inside a ``fairly large scale'' experiment took 660 minutes by itself... and that was with just 14,000 objects.  This actually totally contradicts the numbers you calculated above.  Sorry, I think it just doesn't add up.
>> 
>>> Another important factor is the amount of RPs that we can expect.. I know that Rob and Randy et al are looking into ways to let RPs share data and be less reliant on central repository servers. On the other hand if all ASNs run at least 1 rpki validation cache that talks to the repositories directly then we're looking at 40k clients. If they want updates, say just 3 times per day, that's 120k requests per day, so something like 1 - 1.5 per second.
>> 
>> Again, I used Randy's numbers and even his experiments on 14k objects on a small topology show it takes twice as long as your global estimates.
>> 
> 
> We are talking about different things here:
> = You're looking at how long it takes for 1 RP to get in sync.
> = I was referring to the number of RPs that a repository can expect to connect per second.
> 
>>> This slide shows the happy case where all RPs are up to date, and they just check with the server to see if there are any updates. So importantly this does not include long running data transfers.. This is not server grade hardware (just a mac mini) but it's useful as an order of magnitude indication imo. We see that the total number of RPs just checking for updates that the server can handle / second depends linearly on the repository size (it needs to do an O(n) scan). The total number of concurrent RPs also depends on the repository size. It appears that some list / index is kept in memory for every connection. Long story short, we can only process small numbers of RPs per second and it's quite trivial to end up with too many concurrent RPs pushing the servers to the memory limited cliff for huge repositories.
>> 
>> Yeah, I was wondering about that.  It felt like it was beyond my perspective to estimate, so I tried to focus this sizing analysis on a more general systemic view.  I totally appreciate the above comment, but maybe we could try to model that in another tech-note?  I'm happy to try and help, if you would like.
>> 
> 
> This is one of the main reasons why I think that rsync won't scale to the needs we can expect in full deployment. Even if all RIR repositories are hierarchical, and we don't see a lot of non-hosted CAs publishing elsewhere. We can not expect to keep getting that number of say 20 ms/object.. Not without very *huge* investments in setting up some home grown rsync CDN, spreading them like very busy root servers over the world. Not something that the non-hosted folk will likely want to do either btw..
> 
> Then regarding non-hosted CAs not publishing in their RIR repository.. I hadn't really thought of this before, but he advantage of the recursive rsync is lost here. Say 5 organisations do non-hosted and they all publish with a 3rd party providing a repository service.. they are siblings. This repo is flat.
> 
> So apart from the other things I list in the document I sent yesterday, I think we have a requirement that the delta protocol is not dependent on the PKI hierarchy.
> 
> 
> 
>>> 
>>> It's because of this that I keep going on about:
>>> = We should have a separate delta protocol and notification mechanism and not rely on rsync for this
>>> = For scalability:
>>>     = the hard work (CPU) should be done by the clients, not the server
>>>     = it should be possible to offload connections & memory away from the server (proxies)
>>> = It makes sense to look at http and scalability of existing CDNs for delivery
>> 
>> ibid.
>> 
>>> 
>>> My gut feeling (yes it's involved a lot today) tells me that this SHOULD scale a lot better.. For example serving a small update notification file over http using a CDN, 10ks of request / second, easy.. Data transfer to RPs.. probably not a whole lot better actually if you need it all -- though using existing CDNs with a global infrastructure closer to RPs may help here. But if we have a notification file that points to small deltas and fetching these small files is cheap this may actually be a big improvement.. So although it may take a while to do the first sync, *staying* in sync may actually perform a lot better. Well.. all this is thought experiment at this stage. Without a pilot and actual measurements it's hard to be sure.
>> 
>> I really think these are they types of conversations that we should be having.  Thank you very much for putting your thoughts here!
>> 
>> Eric
>

---

From: Russ White <[email-masked]>
Date: Thu, 06 Dec 2012 13:31:27 -0500

>> base of the design in the early '90s.  we're just slogging through the

"The debate is over?" There was a debate? I seem to have missed it. Were
there competing requirements documents? Who authored them? Who was
polled to find real world requirements? How were the final requirements
determined?

We're now slogging through sausage, but it's sausage we could have done
without had we started from the proper end of the problem and actually
built requirements first.

> Um... Kind of a lot has changed since then, and I _believe_ the design was rejected back then too...  I think it might be less professionally embarrassing to ask, what needs to change to make this…

Now, now --let's not interfere with the sausage making process by any
means. That might be embarrassing to the folks who chose the required
sausage.

Russ

-- 
<><
[email-masked]
[email-masked]

---

From: Christopher Morrow <[email-masked]>
Date: Thu, 6 Dec 2012 13:47:02 -0500

> such wild assed cabbage throwing as that document.

I would point out to all folks on this list, and particularly randy
this time, let's try to keep the discussion focused on technical
topics and not devolve into banter such as this example.

all this does is pull us off the more important topic of 'will it blend?'.

-chris

---

From: Randy Bush <[email-masked]>
Date: Fri, 07 Dec 2012 03:52:12 +0900

> topics and not devolve into banter such as this example.

guilty as charged.  my apologies.

randy

---

From: Arturo Servin <[email-masked]>
Date: Thu, 06 Dec 2012 17:33:46 -0200

Eric,

	Just for the sake of clarity.

	Not sure if I understand, but hosted system does not work 100% in the
way that you described.

	Yes, the hosted keep your private key, but you can modify your ROAs
anytime you want. At least the hosted systems that I know provide a web
interface to do that. The ROA would be published immediately or a few
minute/hours later, depending of the rpki-operator (my personal view and
with my network operator hat on is that it should be immediately or in
minutes).

	You could argue that effectively the hosted system "does it for you"
and "you require conversation with them", but the way described is a bit
misleading because it gives the impression that it is a slow or
human-driven process when in reality its not.

	As for EE cert for routers those do not exist yet in the rpki-operator
systems (AFAIK), so anything here would be a guess.

Cheers
/as

On 06/12/2012 15:55, Eric Osterweil wrote:
> Also, each hosted instance that is _not_ flat still needs infrastructure objects (GB, Mft, CA, etc) + DNS domain name + Also a noteworthy point about your hosted model ( ;) ) is that hosted models remove resource holders' abilities to manage their own information directly.  That is, if I have you host my info, and you have my private key, you likely cannot give that key to me for me to make my own operational changes (if you have it in an HSM, that private key cannot be shared). So, I (as a resource holder) cannot author changes to my (say) ROA, or create new router EE certs unless _YOU_ do it for me.  So, right now, that would require conversations like, ``hey ARIN, can you generate a new ROA for me since I have a new feasible origin?'' (no big deal), but in the future that would be, ``hey ARIN, can you create a new EE cert for my router, and send me both the pub/priv key pair so I can configure a new router (or $n$ new routers)?''  I'd hazard that _this_ is a much bigger 
 de
>  al, especially with the need for a private key exchange.  One more complexity to note. ;)

---

From: Russ White <[email-masked]>
Date: Thu, 06 Dec 2012 14:42:50 -0500

> with my network operator hat on is that it should be immediately or in

Modulo the propagation time...

But isn't it bad to allow automated changes to a certificate that's
designed to operate "at human speeds," (based on other conversations on
this list), and is so critical to the operation of the routing system?
Aren't we just adding to the total attack surface available against the
routing system by allowing users to go into a web page and change what's
advertised into the ROA system?

I know --it takes that bit of the problem out of the actual BGP space,
and moves it into another space altogether, but... I don't see banking
as being "more secure" because you can do all your banking on line
--it's more convenient, but I think there's a clear tradeoff in security
and convenience here, right?

If this is the type of system that's envisioned, shouldn't the risks
involved be documented someplace?

Russ

-- 
<><
[email-masked]
[email-masked]

---

From: Arturo Servin <[email-masked]>
Date: Thu, 06 Dec 2012 17:54:16 -0200

Russ,

	The "automated" changes would be similar to the ones that you would do
on your own CA.

	As adding a new attack surface by using web interfaces, yes we are as
you said, the convenience has a price. Nevertheless if well protected it
could be an acceptable trade-off. Forcing everyone to have their own CA
(possible as it should be from a security stand point) would make rpki
very complex to deploy for the small/medium players (similar to DNSSEC
today).
	
	Perhaps we should document all these.

Regards,
as

---

From: Christopher Morrow <[email-masked]>
Date: Thu, 6 Dec 2012 15:13:44 -0500

> advertised into the ROA system?

could be, but so are ssl cert things from thawte, or dns services that
permit users to interact with the dns provider...

the hosted model is there for, ideally, people who do not want to run
their own infrastructure... in a sense it's godaddy-dns for rpki data,
instead of 'running your own dns server'.

---

From: Eric Osterweil <[email-masked]>
Date: Thu, 6 Dec 2012 15:56:49 -0500

> with my network operator hat on is that it should be immediately or in

That's all fine and dandy, and while I agree with what Russ said in his response, you should note that I did say, ``no big deal.''  That said, after thinking about what Russ said and your response, I actually do think this is a big deal... now... ;)

If/when I (as a prospective user of the system) decide to move my information from your hostage model to my own repository, how will I be able to migrate the private portion of my certificate from _your_ hosted system to my own?  How are we doing this private key exchange?  Do you realize that you won't be able to use an HSM at all if the private portion will ever need to be exported?  Maybe I missed that part of the explanation?

> human-driven process when in reality its not.

Sorry, this was just a colloquialism... Not meant to be misleading.  I understand that this would likely be a web interface, or whatever.  That said, in either case, aren't we now saying that for someone to update their secure routing information, they need to go through you?  Is that a Single Point of Failure (SPF)?  What if your web interface is being DDoS'ed when I need to update my ROA (perhaps because _I'm_ being DDoS'ed and am engaging a DDoS mitigation service for the first time)?  Anyway, these are (as I said) complexities to note....

> systems (AFAIK), so anything here would be a guess.

I seriously disagree here... We are building a system that is intended to do this!  The rpki must have router certs in it (according to the bgpsec design), and we therefore must to be proactive in evaluating that aspect.  In the above hosted model, you absolutely will be on the hook for this, and if you invest in a system whose design will be architecturally (or even operationally) unable to accommodate this eventuality, you are doing yourself a _huge_ disservice by ignoring the writing on the wall.  The bottom line is, there needs to be some thinking around this, or it will bite all of us (I have packets on the interwebz too).

My 0.02, I worry that your web interface is inadequate for the eventual secure provisioning of router certs.

Eric

---

From: Richard Barnes <[email-masked]>
Date: Thu, 6 Dec 2012 16:12:12 -0500

> My 0.02, I worry that your web interface is inadequate for the eventual secure provisioning of router certs.

I'm curious what sort of UI you would recommend for provisioning router certs.  There's not a fundamental difference between trusting TLS to deliver HTTPS and trusting, say, SSH to protect your router CLI.  

Any UI presents a point of vulnerability.  I don't really see a reason to pick on web apps in this context.

--Richard

---

From: Eric Osterweil <[email-masked]>
Date: Thu, 6 Dec 2012 16:20:00 -0500

On Dec 6, 2012, at 4:12 PM, Richard Barnes wrote:

> I'm curious what sort of UI you would recommend for provisioning router certs.  

As I thought I had outlined, the fundamental problem is in the hosted model.  I don't think data owners/operators should be captive to external systemic dependencies in this case.

> There's not a fundamental difference between trusting TLS to deliver HTTPS and trusting, say, SSH to protect your router CLI.  

Right, they're both dangerous choices when you do them over the vast/untrusted/public interwebz compared to doing provisioning in house...  Not sure where you're going with this...

> Any UI presents a point of vulnerability.  I don't really see a reason to pick on web apps in this context.

No... there's a lot of systemic dependencies that get roped in when you start doing these things out over the Internet vs. in your NOC.

Eric

---

From: Richard Barnes <[email-masked]>
Date: Thu, 6 Dec 2012 16:59:47 -0500

> No... there's a lot of systemic dependencies that get roped in when you start doing these things out over the Internet vs. in your NOC.

As Chris mentioned down-thread: Could you explain how these considerations differ from considerations around hosted services for critical services, e.g., DNSSEC?

--Richard

---

From: Eric Osterweil <[email-masked]>
Date: Thu, 6 Dec 2012 17:11:18 -0500

> As Chris mentioned down-thread: Could you explain how these considerations differ from considerations around hosted services for critical services, e.g., DNSSEC?

This system (and therefore, its considerations) is/are VASTLY different than those of DNSSEC... In a lot of ways:
1 - This repository system requires RPs to have a FULLY synchronized copy of all objects at all times.  DNSSEC does not.
	a - This means all systems have to be high performing/reachable/etc. or there begin to be problems
	b - This means that global crawling must be done before computation can be reliable
	c - ...
2 - This system's hosting model is proposing to put all of the object generation information in a very few places (the hosted model), DNSSEC does not (the hosting colocation coefficient in DNSSEC is vastly 			different that 5, for rpki)
	a - I am not even encouraged to have all my DNS zones hosted by RIRs... I can get a registrar or hoster to do it, but I can choose from many many options...
	b - ...
3 - The downside of borking routing is all your [connectivity] bases are belong to us... Lots of problems exist when destinations are unreachable that do not exist (or are semantically different) when names are unresolvable.  In short, these are very different systems, with _very_ different architectures, with very _very_ different goals, etc.  I could go on, but is this sufficient?

Eric

---

From: Richard Barnes <[email-masked]>
Date: Thu, 6 Dec 2012 17:42:24 -0500

On Dec 6, 2012, at 5:11 PM, Eric Osterweil <[email-masked]> wrote:

> 	c - …

DNSSEC requires you to have up-to-date information on all the things you care about.  It's just that in BGPSEC, everyone cares about everything :)

W.r.t. (a) -- Doesn't this argue in favor of hosting?  Isn't the "all" less burdensome if there are fewer things to keep alive?  I know I would rather have my web site on Dreamhost than on my on host in a colo.

W.r.t. (b) -- Again, this seems to argue in favor of the hosted model, since you have to crawl fewer repos.  

And it's not really true that you have to crawl the whole tree before you can do anything.  If you have a partial tree, then you can validate part of the ROAs.  Especially if you crawl intelligently, e.g., trying to avoid missing links in a cert chain. I believe that RPSTIR does something like this.

> 	b - …

Not sure where you're going with this.  I thought you were arguing that the hosting model in general was a bad technology, which would not depend on the coefficient or which entities were doing the hosting.

> 3 - The downside of borking routing is all your [connectivity] bases are belong to us... Lots of problems exist when destinations are unreachable that do not exist (or are semantically different) w…

Different services have different values for different networks.  There are certainly lots of stakeholders for whom loss of DNS reachability would be as bad as loss of IP reachability, as would seem to be attested by the predominant focus in the cyber crime community on DNS hijacking rather than BGP hijacking.

To be clear, I have no particular dog in this fight.  I'm not even really clear on why this discussion is happening on the SIDR list, given that it's a deployment decision between an CA and its customers.  

--Richard

---

From: Arturo Servin <[email-masked]>
Date: Thu, 06 Dec 2012 20:54:39 -0200

Eric

	Chris said much better than me. Hosted rpki is like the "go-daddy" of
RPKI. It is intended as a bootstraping solution to take rpki up.

	The hosted solution is not aimed to everybody, it is aimed to
small/medium operators that otherwise would struggle to sign their
resources, run a CA, and run a repository.

	Large operators DO NOT need to use the hosted solution, they can if
they want but they can run their own CAs and they should.

	If a hosted ISP wants to go to up/down and run their own CA they can do
that with a key-rollover and a "make before break" strategy.

	About the EE certs for router I didn't explain correctly. Hosted
solution do not have it now because they haven't been defined yet (we
are still arguing about BGPSEC specs). However, in the moment that the
specs are ready router certs will be supported.

Regards,
as

---

From: Eric Osterweil <[email-masked]>
Date: Thu, 6 Dec 2012 17:57:17 -0500

> DNSSEC requires you to have up-to-date information on all the things you care about.  It's just that in BGPSEC, everyone cares about everything :)

Uh... big difference.  DNSSEC doesn't require you to care about anything before you need it (on demand).  RPKI is prefetching...  I can't really outline the architectural difference better than that.

> W.r.t. (a) -- Doesn't this argue in favor of hosting?  Isn't the "all" less burdensome if there are fewer things to keep alive?  I know I would rather have my web site on Dreamhost than on my on ho…

Nope, it doesn't.  It actually points out that hosting's myopic benefits are fools-gold, and they come with a much more dangerous price tag.  Your choice of dreamhost is a _choice_, not a mandate from a set of 5 options.  Fundamentally different.

> W.r.t. (b) -- Again, this seems to argue in favor of the hosted model, since you have to crawl fewer repos.  

Again, I fear you've missed the point.  This points out that there is a _fundamental_flaw_ in the architecture.  You inherit the instability and latency of all hosted environments all over the world, and you are captive until it is resolved.  This is an architectural issue.

> And it's not really true that you have to crawl the whole tree before you can do anything.  If you have a partial tree, then you can validate part of the ROAs.  Especially if you crawl intelligentl…

So, if I need _you_ (an RP) to get my ROA because I've just come under attack, do you think it's OK to just schedule me for later?  That's not going to help you reach the critical services that I might be offering.  I'd call that, seriously suboptimal engineering. ;)  Architecturally, it would be better to reach out for information when you need it... In fact, that's kinda how other name mapping systems that you might have mentioned earlier do it... ;)

> Not sure where you're going with this.  I thought you were arguing that the hosting model in general was a bad technology, which would not depend on the coefficient or which entities were doing the…

I am, and it doesn't.  I was, and am, illustrating for you how inapt the comparison is.

> Different services have different values for different networks.  There are certainly lots of stakeholders for whom loss of DNS reachability would be as bad as loss of IP reachability, as would see…

Sorry, a quick look at my dayjob might inform you of the fact that I put a high price on the importance of DNS... ;)  That said, I have to say that the cybercime analogy is pretty weak.  How about, the angle where cyber criminals DDoS a company and they come to someone (maybe us?) for DDoS remediation _through_BGP_?  That work fer ya? ;)

> To be clear, I have no particular dog in this fight.  I'm not even really clear on why this discussion is happening on the SIDR list, given that it's a deployment decision between an CA and its cus…

This is a deployment discussion about the rpki infrastructure, which has quite a bit to do w/ sidr.

Eric

---

From: Eric Osterweil <[email-masked]>
Date: Thu, 6 Dec 2012 18:03:18 -0500

> RPKI. It is intended as a bootstraping solution to take rpki up.

Maybe you can help me answer the questions I posed in my last email then?  :-P

> resources, run a CA, and run a repository.

How do they get their private keys from you?  This is important to think through _now_ before it becomes an operational blackhole... Also, what happens if you get DDoS'ed and I need your services?  In DNS, there are a lot of registrars to choose from, and no single point of failure... The RIRs are not as plentiful in numbers as them, so you are a higher value target this way...

> they want but they can run their own CAs and they should.

Anyone who uses you would need these services, it seems like it would be worth working out the ``what ifs,'' no?

> specs are ready router certs will be supported.

If we've enshrined operations and practices (i.e. the hosted/HSM model) because we didn't think through the complexities, and that later impedes our needs, then we have been negligent.  This seems like a pretty serious problem and it ought to be worked out now before we decide people should be doing the hosted anything.

Eric

---

From: Russ White <[email-masked]>
Date: Thu, 06 Dec 2012 18:35:01 -0500

> And it's not really true that you have to crawl the whole tree before you can do anything.  If you have a partial tree, then you can validate part of the ROAs.  Especially if you crawl intelligentl…

The simplest way to explain this is that you've created a three way
dependency here --the routing system is dependent on the RPKI, which is
in turn dependent on the hosting service, which is in turn dependent on
the routing system, but you have the third interaction with the RPKI itself.

You've gone from three moving parts on the protocol side to three, which
makes it more complex. On the human side, you've added the complexity of
yet another contractural relationship, which makes things more complex
there, adding more places to make mistakes.

So adding another moving piece makes things more complex, which will
make an already fragile system more fragile.

:-)

Russ

-- 
<><
[email-masked]
[email-masked]

---

From: "Montgomery, Douglas" <[email-masked]>
Date: Thu, 6 Dec 2012 19:00:42 -0500

On 12/6/12 5:57 PM, "Eric Osterweil" <[email-masked]> wrote:

>> DNSSEC requires you to have up-to-date information on all the things
>>you care about.  It's just that in BGPSEC, everyone cares about
>>everything :)
>
>Uh... big difference.  DNSSEC doesn't require you to care about anything
>before you need it (on demand).  RPKI is prefetching...  I can't really
>outline the architectural difference better than that.

So this seems to be about sub-system behavior in various transient states.
 Cold start of a relying party, when there are no other "hot" instances in
contact.  While some will argue that one can engineer redundant systems,
and smart RPs (e.g., that initialize with last running check-pointed state
at boot),  still I will admit that one could envisions firing up a new RP,
just out of the box, for the first time and it taking time to load its
initial state.

Won't a demand driven system will experience something very similar when
first fired up and and a full table dump comes across the wire in BGP?
If a demand driven system wasn't smart enough to use a hot standby with
significant cached state, it too will suffer the latency of pulling a
significant portion of the data it needs instantly.

While I see the architectural differences in those two, it is not clear to
me that the end result to a running BGP system that uses them is all that
different.

Once either approach has achieved steady state, assuming there was some
caching done in the demand based system, if the cache holding time of
demand queries was the same as the poling interval of the RP, do you think
the responsive ness of a change of authoritative info is all that
different?

Or do you not assume there will be any caching in a demand based system?
And if so, would you be concerned about the peers * full_table number of
queries that would result from a router reboot?

Truly just asking to understand ...
Dougm

---

From: [email-masked]
Date: Thu, 6 Dec 2012 23:59:11 -0200 (UYST)

Inline.

> Maybe you can help me answer the questions I posed in my last email then?

I tried, but may be I was not clear.

> How do they get their private keys from you?  This is important to think
> through _now_ before it becomes an operational blackhole...

"If a hosted ISP wants to go to up/down and run their own CA they can do
that with a key-rollover and a "make before break" strategy."

In other words, you cannot. You can generate your own and
make-before-break by publishing your new shiny cert with your own CA and
then revoking the old one.

>Also, what
> happens if you get DDoS'ed and I need your services?  In DNS, there are a
> lot of registrars to choose from, and no single point of failure...

There could be more hosted providers. RIRs are by not means the only ones.
In fact, it would be very similar to the registry-registar model.

>The
> RIRs are not as plentiful in numbers as them, so you are a higher value
> target this way...

The whole scenario does not seem to different to me if you host your DNS
services and your registrar/hosted are DDoSed and you cannot change your
NS records.

And in the same fashion to DNS hosted solutions, RPKI-hosted are aimed to
specific organizations. If you want to be independent from any hosted
solution, run your own DNS servers or your own CA.

The fact that hosted solutions are not DDoS resistant today does not mean
that they cannot be in the future.

> worth working out the ``what ifs,'' no?

This (RPKI in general, repositories, hosted solution) is very new and
still evolving. I take your concerns very seriously and a value input for
improvement.

> If we've enshrined operations and practices (i.e. the hosted/HSM model)
> because we didn't think through the complexities, and that later impedes
> our needs, then we have been negligent.  This seems like a pretty serious
> problem and it ought to be worked out now before we decide people should
> be doing the hosted anything.

I found a bit hard to engineer a system without specs. I thought that this
thread was about precisely about BGPSEC specs. Once we have the specs,
then hosted-rpki solutions can incorporate them.

Regards,
/as

---

From: [email-masked]
Date: Fri, 7 Dec 2012 00:09:06 -0200 (UYST)

Russ,

I think you are twisting the facts on your own convenience.

RPKI != RPKI-hosted solutions. And RPKI is not dependant on the
RPKI-hosted solutions in the same way that DNS is not dependant in the
DNS-hosted solutions.

Regards,
.as

>
>> And it's not really true that you have to crawl the whole tree before
>> you can do anything.  If you have a partial tree, then you can validate
>> part of the ROAs.  Especially if you crawl intelligently, e.g., trying
>> to avoid missing links in a cert chain. I believe that RPSTIR does
>> something like this.
>
> _______________________________________________
> sidr mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/sidr
>

---

From: Eric Osterweil <[email-masked]>
Date: Thu, 6 Dec 2012 22:03:12 -0500

> just out of the box, for the first time and it taking time to load its

I was hoping we could all see from my quoted text above that this latest discussion is about the _architectural_ difference between the on-demand soft-state DNS system, and the prefetching replicated state machine of RPKI.  These two are fundamentally very different architectural models.  Your comments about boot states are interesting, but somewhat off topic to this post, imho.

> first fired up and and a full table dump comes across the wire in BGP?

Ask any of the operators on this list how they feel about full table dumps and routers refreshing table state, configs, etc when coming online.  If you're saying that RPKI is just like that, I think you just sank your position.

> significant portion of the data it needs instantly.

Again, we were discussing the architectural difference, not polishing the chrome on the titanic. :)

> me that the end result to a running BGP system that uses them is all that

Hmm.. Well, re: my comment above, we have very different opinions.  I (literally) defer to operators to answer my above question, and if I'm rebuffed on that, then so be it.

> the responsive ness of a change of authoritative info is all that

I honestly have to say that I don't know how I could tell at this point.  I think it would be good for someone to put forward a model, do some simulation/measurements and present some analysis.  I don't know when BGP is in ``steady state'' or what you might mean by that, but it seems like a good time for us to be more quantitative and less qualitative.

> queries that would result from a router reboot?

I honestly don't understand this last part, but I'm hoping my comments and questions (above) address it?

Eric

---

From: Eric Osterweil <[email-masked]>
Date: Thu, 6 Dec 2012 22:10:02 -0500

> In fact, it would be very similar to the registry-registar model.

Well, I am more partial to your first comment (not just RIRs), but I think it is very _very_ different than the registrant, registrar, registry model.

> services and your registrar/hosted are DDoSed and you cannot change your

If _your_ registrar gets DDoS'ed, but _mine_ does not, I'll be fine.  THe interface between registrars and the registry is not open for others to bang on.  It's out there, but the front door is not in broad view of the street.

> solution, run your own DNS servers or your own CA.

Actually, this was my point.

> that they cannot be in the future.

The fact that we're painting the floor doesn't mean we'll paint ourselves into a corner either.  Yet, if we aren't careful...

> still evolving. I take your concerns very seriously and a value input for

Thanks, I appreciate that.

> then hosted-rpki solutions can incorporate them.

Well, we _ought_ to be talking about requirements, and that ought to shape the specs.  That said, I think it is plain to all that BGPsec is planning to put router EE certs into the RPKI, and that caches will need to fetch them.  The rest... well, that's a separate thread about requirements. :)

Eric

---

From: Tony Tauber <[email-masked]>
Date: Thu, 6 Dec 2012 23:17:23 -0500

Hello,

Joe Operator here.

I don't want to interrupt the tiresome food-fight, but would like to more
clearly understand how this alternative system would work.
I may have lost the thread in the roiling sea of presumptions that's got
this discussion bouncing hither, yon, and back again.

I didn't change the subject line, but feel free to do so if needed.

Tony

> Ask any of the operators on this list how they feel about full table dumps
> and routers refreshing table state, configs, etc when coming online.  If
> you're saying that RPKI is just like that, I think you just sank your
> position.
>
> Hmm.. Well, re: my comment above, we have very different opinions.  I
> (literally) defer to operators to answer my above question, and if I'm
> rebuffed on that, then so be it.
>

---

From: "Montgomery, Douglas" <[email-masked]>
Date: Thu, 6 Dec 2012 23:20:02 -0500

On 12/6/12 10:03 PM, "Eric Osterweil" <[email-masked]> wrote:

>I was hoping we could all see from my quoted text above that this latest
>discussion is about the _architectural_ difference between the on-demand
>soft-state DNS system, and the prefetching replicated state machine of
>RPKI.  These two are fundamentally very different architectural models.
>Your comments about boot states are interesting, but somewhat off topic
>to this post, imho.

IMHO they address the architectural issue, by pointing out that while the
there are architecturally different in the abstract, for the purpose they
are being designed/discussed, this architectural difference makes very
little difference in performance.

>Ask any of the operators on this list how they feel about full table
>dumps and routers refreshing table state, configs, etc when coming
>online.  If you're saying that RPKI is just like that, I think you just
>sank your position.

No you completely missed my point, what I am saying is that when a demand
based system (say DNS) first comes on line with no cached state, and a
router with a full BGP table, it will need to do O(500K) queries
immediately.   Lets say DNS is the model for a query based system.  O(10s
to 100s msec) per query?   Lets look at a range of a reverse DNS query
(with DNSSEC?) taking from 10msec (very optimistic) to 500msec.  That
means a cold starting DNS based system would take between 1.4 hours and 69
hours to gather enough origin mappings to support a running DFZ router.
Note that is dangerous not to have all the state necessary to verify a
full RIB because of "prefer valid" like policies, where the relative
validity state of multiple entries in the RIB matter.  I.e., you better
stall the BGP decision process until you have the validation state for all
entries.

So if the query model takes 69 hours to gain enough state to be useful,
why is it architecturally interesting that it did that through 500K
individual queries instead of batch downloads?

>chrome on the titanic. :)

Not sure the pithy digs with smilies helps the clarity of these
discussions ... But OK I will give it a shot... Just because the Spruce
Goose's architecture did not sink, it is not clear to me it was any better
suited for the job it was designed for that the Titanic. :^)    .... Nah,
I still don't think these help.

>rebuffed on that, then so be it.

Seem my comment above.   I suspect their behavior relative to the
architectural differences you point out ... Will be of no operational
significance in the application they are designed to support.

>I honestly have to say that I don't know how I could tell at this point.
>I think it would be good for someone to put forward a model, do some
>simulation/measurements and present some analysis.  I don't know when BGP
>is in ``steady state'' or what you might mean by that, but it seems like
>a good time for us to be more quantitative and less qualitative.

I was talking about when the systems designed to support the distribution
of authorization information (e.g., RP/RPKI or some DNS based system?)
were in steady state ... I.e., they have booted up and done their initial
data loads.

>and questions (above) address it?

If you don't locally cache the results of querying some system for each
prefix origin pair from a neighbor ... You will have to multiply the time
to query  500K such lookups for each peering session that gives you a full
table.   If you do cache them for some amount of time.  Set the polling
interval of a batch pull based system to that same time span.   Now, how
fast do each react to changes in the authoritative data?

Dougm

---

From: Christopher Morrow <[email-masked]>
Date: Fri, 7 Dec 2012 00:34:39 -0500

> were in steady state ... I.e., they have booted up and done their initial

somewhere along the train my comment about dns got moved to 'just like
dnssec' (which I was intentionally not referring to, maybe I should
have used email as an example instead).  Then in this part of the
messages I think my DNS comment got moved into 'if you distribute the
rpki data via dns instead of up/down/left/right/a-b/a-b... the
repository -> cache protocol'.  I didn't mean it in that sense either
:(

I think somewhere 5-8 messages back Arturo's note that:
  1) hosted model is just a crutch
  2) hosted model isn't intended for everyone to use
  3) most large ISP or large operations groups are expected to run their own CA

coupled with eric's notes that:
  1) hosted seems fragile for lots of operations
  2) people should think long and hard about using the hosted model of
controlling their own fate

gets the general gist of my point: "If you use the hosted model you
are equivalently outsourcing your Mail/SMTP infrastructure to another
person, be sure you want to do that..."

apologies for the confusing example :(
-chris

---

From: Russ White <[email-masked]>
Date: Fri, 07 Dec 2012 07:25:31 -0500

> I think you are twisting the facts on your own convenience.

No, I am merely pointing out that adding a hosted service to an already
complex system increases it's complexity. I don't see how that's
"twisting facts." Would you care to explain which fact I've twisted, and
how?

> RPKI-hosted solutions in the same way that DNS is not dependant in the

So you're saying that the hosted RPKI system is not reliant on routing?
That it can be reached no matter whether or not routing is available?

There was, just recently, a large amount of discussion over the problems
with a DNS based system to provide origin authentication because "DNS
relies on routing, and now you're making routing rely on DNS!" Ignoring
the dependence of virtually _any_ distribution system on routing to one
degree or another, how can the same criticism not apply to a hosted RPKI
system, and the customer's ability to reach that system?

And how does adding another system that relies on the system it's
supposed to be protecting reduce (or even "not add") complexity?

Or am I "twisting facts to my convenience" again?

:-)

Russ

---

From: "Montgomery, Douglas" <[email-masked]>
Date: Fri, 7 Dec 2012 07:56:14 -0500

I just want to point out that I was on a different train.  Discussing the
supposition that query&cahce arch's behave differently that batch pull
archs in either startup or steady state performance.  I was not commenting
on the whole trust/business model discussion.

I was replying to Eric's suggestion that DNS-like systems are an example
of the on-demand architectural choice.

>Your comments about boot states are interesting, but somewhat off topic

Those two choices seem moot from a back of the envelope performance /
behavior perspective (although I would like to run some tests on a 500K
query of a signed reverse DNS ... I don't expect my estimates to get
better) ... With either you had better engineer well with redundant
"gatherers/resolvers", that don't fate share infrastructure, etc ...
Otherwise the hour glass is going to spin for a long time when a DFZ
router tries to validate its RIBs.

Now I return you to your trust/business model discussion ...
Dougm

---

From: "Carlos M. martinez" <[email-masked]>
Date: Fri, 07 Dec 2012 11:21:29 -0200

Hello R.

> That it can be reached no matter whether or not routing is available?

If by 'hosted RPKI' you are referring to the 'web page', well, the
system does need the webpage to function. All repository generation and
upkeep will still be running even if the webpage is DDoSed. True, you
lose the ability to introduce changes while this supposed DDoS is
operating, but routing will not get plastered. All hosted RPKI
implementations implement a split system were this upkeep is run by
servers not exposed directly to the Internet.

If by 'hosted RPKI' you are referring to repository operators hosting
all crypto material, yes, you are depending on routing. But, then, if
you host your own CA RPs still depend on routing to fetch objects from
your repo. If you put objects in the DNS, you depend on routing AND on
the DNS to fetch them (that's two dependencies instead of one).

RPKI repo fetch could be made fully independent from DNS by adopting
something like multiple-pub-points.

RPKI repositories could be massively cached by CDNs by adopting HTTP as
an additional repository fetching protocol.

Yet all of these depend on routing up to some point, it doesn't matter
if the repos are hosted by third parties or hosted by the end users
themselves. If you have a design for a distribution system which in no
way depends on routing and doesn't involve the UPS guy, that'd be awesome ;)

> Or am I "twisting facts to my convenience" again?

I think you're describing a dependency which is not critical and
presenting it as if it were.

---

From: "Carlos M. martinez" <[email-masked]>
Date: Fri, 07 Dec 2012 11:31:55 -0200

Hey Chris et al,

On 12/07/2012 03:34 AM, Christopher Morrow wrote:
... snip ...

>   3) most large ISP or large operations groups are expected to run their own CA

The hosted model is just about right for probably around ~80% of the
routing participants out there. I don't think it's just a crutch, I
think it answers a real community (or should I rather say, market)
demand. It might have started as one, but IMO it hit a nail in the head.

Let's face it: Most sites out there are not as technology-literate as we
might light them to be. The dangers of screwing their own CA, losing
their keys and mismanaging their publications are much higher if their
host their own than if they trust a specialized third party.

Hosted fills a gap. Sure, I don't think France Telecom, Verizon or
Telmex should use hosted, they can do better. But the folk holding one
AS, a /48 and a couple of IPv4 /22s, well, they are mostly better off
using hosted.

The good thing here is that hosted is _optional_ (I think we are
forgetting about this fact in all of this discussion). If you don't want
it and want to get your feet wet, please go up/down.

> person, be sure you want to do that..."

Agree, people are still responsible for making conscious decisions.
Then, taking your example, how many small ops right now host their own
email? Probably not many.

Warm regards,

~Carlos

---

From: Russ White <[email-masked]>
Date: Fri, 07 Dec 2012 09:10:58 -0500

> way depends on routing and doesn't involve the UPS guy, that'd be awesome ;)

There was once a system proposed.... But that was a long time ago, in a
galaxy far away.

> presenting it as if it were.

I was presenting a dependency which exists. The question isn't whether
or not it's critical, it's whether or not it adds complexity.

In other words, even if you classify it as "non-critical" (which I'm not
certain I agree with without a lot more thought on the matter), I would
still argue that the non-critical parts of a system still add
complexity. In fact, to go farther, it is generally true that the
non-critical parts of a system add more complexity than the critical
ones, particularly in routing.

So, I'm still looking for this poor "fact," I've left twisted and
horribly abused on the floor.

:-)

Russ

---

From: Eric Osterweil <[email-masked]>
Date: Fri, 7 Dec 2012 10:28:41 -0500

> little difference in performance.

Seriously Doug, this totally misses the point.  If there is a fundamental difference, all the engineering in the world will only be masking the fact that the architecture is different, has different scaling properties, different usage models that it is suited for, etc.  Can a model that is inapt for certain requirements be buffed up to service them anyway?  Sure.  Does that make it ideal?  Doubtful, but ymmv...

> individual queries instead of batch downloads?

I think I see the disconnect here...  I don't know why you'd presume that _I_ think loading a router's RIB from DNS queries is a good idea.  I certainly don't recall saying that, and I am certainly not advocating that.

> I still don't think these help.

But the Spruce Goose was killed by yellow journalism, and the Titanic just couldn't manage the open seas with such a tiny rudder (point, iceberg)... ;-}

> significance in the application they are designed to support.

*sigh*  Apparently, this will just be a point where we have to agree to disagree.  I feel that (as engineers) we should strive to find the right tool for the right job, but ymmv.

> were in steady state ... I.e., they have booted up and done their initial

Right, and my response was that (even though I have not done any specific analysis on this, in this context), BGP tends to defy the notion of being in steady state, so without a precise comment like the above, it is hard to agree.  Also, I'd argue that after a router boots, I've _heard_ they tend to face a lot of churn that isn't steady.  I was suggesting a study be done, but that's just my 0.02.

> fast do each react to changes in the authoritative data?

Yes, caching is good, but I also think the model that the caching is overlaid on matters.

Eric

---

From: Eric Osterweil <[email-masked]>
Date: Fri, 7 Dec 2012 10:30:12 -0500

>  3) most large ISP or large operations groups are expected to run their own CA

I totally agree, and I hope I was clear in ack'ing that this was something that I agreed with.

> person, be sure you want to do that..."

Yup.  Agreed.

Eric

---

From: Tim Bruijnzeels <[email-masked]>
Date: Fri, 7 Dec 2012 16:30:55 +0100

Hey all,

>>  3) most large ISP or large operations groups are expected to run their own CA

Hosted model is ambiguous.

Please differentiate between hosted CAs and centralised repository server. Some people, like Carlos wrt UI vs publishing, do, but not everywhere in this thread..
The pros and cons for hosted CAs and 'hosted' repositories are different.

Current hosted solutions combine the two, but this is not necessarily the case in future. E.g. an organisation might choose to run their own non-hosted CA, but publish in one, or more (yeay..) central repositories.

> demand. It might have started as one, but IMO it hit a nail in the head.

+1 from our experience rolling this out.

> AS, a /48 and a couple of IPv4 /22s, well, they are mostly better off

In a way people are used to this. Lots of sites are already used to 'hosted' whois, IRR etc.

> it and want to get your feet wet, please go up/down.

Non-hosted is an option in our pilot, but so-far we say very little demand for it. We have four members playing with non-hosted in our pilot environment vs 1200+ using the the hosted model in production. This includes big organisations.

> email? Probably not many.

Benefits of non-hosted CA:
- you have your own keys
  => in hosted a security breach can in principle let others do things with your keys like issue/revoke ROAs
- little risk of *your* ui being DDoS-ed

Benefit of hosted:
- low cost, easy to enable, no need to run your own (complex) systems

@Erik's remark regarding hosted HSMs. As Carlos said: we would also allow people to migrate between hosted and non-hosted: temporarily signing certs to both CAs, just for the time of transition.

Non-hosted CA management may well pick up one day, but imho this depends on uptake of RPKI for secure routing, and the availability of easy-to-use non-hosted solutions. Present day the benefit of doing this yourself is not clear enough to warrant the cost. If benefit goes up and cost goes down this might change. Comparing to the DNSSec world, imagine appliance boxes that network operators can put in a rack that let them do this.. Without this I seriously doubt there will be significant uptake of non-hosted. And without uptake of hosted I see no business case for vendors to build these things..

With regards to non-hosted publication….

In the current RPKI & sidr model (and I haven't seen a comprehensive alternative -- echo comment by Tony Tauber) objects will have to be downloaded. Having many small self-managed repositories does not seem the best way forward to me. It's difficult for relying party tools, it's likely that many of them will be poorly run and/or easy targets for more localised attacks.

I see more in having a number of large repositories and allowing CAs to publish in more than one place to mitigate the risks of one of them being unavailable. Plus of course a set of 2.0 repository standards that give these repositories a fighting chance to achieve high-availability without having to re-invent the wheel here, and roll out 10s-100s of rsync servers across the globe. Think using tested and proven industry technology.. Like using immutable data, http, CDNs etc. When is the last time your googlebookcnnbbc website was completely unreachable?

Tim

---

From: Eric Osterweil <[email-masked]>
Date: Fri, 7 Dec 2012 10:32:08 -0500

> of the on-demand architectural choice.

Doug, this is getting silly.  If you have a system that _requires_ full state of all data producers in the entire world before it can begin, then you are talking about a _much_ different architecture than one that only queries for _relevant_ information when it becomes relevant.  If you use the latter for the former's goals, or the former for the latter's goals then so be it, but they are designed with different precepts.

Eric

---

From: Andy Newton <[email-masked]>
Date: Fri, 7 Dec 2012 16:47:28 +0000

On 12/6/12 5:57 PM, "Eric Osterweil" <[email-masked]> wrote:

> Nope, it doesn't.  It actually points out that hosting's myopic benefits
>are fools-gold, and they come with a
> much more dangerous price tag.  Your choice of dreamhost is a _choice_,
>not a mandate from a set of 5
> options.  Fundamentally different.

Just because the RIRs are the only organization offering hosting services,
doesn't mean that they are the only ones that can offer hosting services.
Certainly an enterprising Internet company with DDOS mitigation services
could offer RPKI CA hosting or simply hosting of RPKI repositories for
DDOS mitigation.

On 12/6/12 6:03 PM, "Eric Osterweil" <[email-masked]> wrote:

>through _now_ before it becomes
> an operational blackhole... Also, what happens if you get DDoS'ed and I
>need your services?  In DNS,
> there are a lot of registrars to choose from, and no single point of
>failure... The RIRs are not as
> plentiful in numbers as them, so you are a higher value target this
>way...

Just to reiterate the points Carlos made regarding DDOS of RPKI
repositories (which is the issue, not the hosting services): 1) if DDOS of
the repositories is in question, we should look at the
multiple-publication points proposal, and 2) switching to an HTTP fetching
protocol would broaden the available base of organizations that might wish
to distribute RPKI repositories.

-andy

---

From: Eric Osterweil <[email-masked]>
Date: Fri, 7 Dec 2012 12:01:23 -0500

> could offer RPKI CA hosting or simply hosting of RPKI repositories for

Yeah, while I'm not proposing that, my whole point was triggered off the assertion that there will be 5 repos, and that the hosted model fits the eventual needs.  That thinking ignored too much grey area for me.

> to distribute RPKI repositories.

For 1) I think considering the results of DDoS'ing the repos should be part of the thinking.  For 2) I (personally) haven't thought that through enough to adopt a position.

Eric

---

From: Eric Osterweil <[email-masked]>
Date: Fri, 7 Dec 2012 12:04:54 -0500

> Yeah, while I'm not proposing that, my whole point was triggered off the assertion that there will be 5 repos, and that the hosted model fits the eventual needs.  That thinking ignored too much gre…

Sorry, meant to say fits the eventual needs _of_everyone_...

---

From: "Montgomery, Douglas" <[email-masked]>
Date: Fri, 7 Dec 2012 12:28:09 -0500

On 12/7/12 10:32 AM, "Eric Osterweil" <[email-masked]> wrote:

>Doug, this is getting silly.  If you have a system that _requires_ full
>state of all data producers in the entire world before it can begin, then
>you are talking about a _much_ different architecture than one that only
>queries for _relevant_ information when it becomes relevant.  If you use
>the latter for the former's goals, or the former for the latter's goals
>then so be it, but they are designed with different precepts.

What is silly is our failure to communicate.

If the initial immediate requirement of a demand driven system is to
individually query for 90%+ of the total information in the system, how is
that different that pre-fetching it all?

DFZ Router with 500K unique routes needs to validate its RIB using a
demand-based validator that just came up.

How would you like the demand system to perform those resolutions?  What
is your time estimate for how long the cold started demand driven
"validator" will take to validate all of those entries?

You are right that they are different architectural models.  No argument
there.   Their use for the problem we are discussing, will result in that
difference being moot from a performance scaling perspective.

No argument that the architectures are different.

We seem to differ on that difference is important ... And probably on
which would scale better.

dougm

---

From: "Montgomery, Douglas" <[email-masked]>
Date: Fri, 7 Dec 2012 12:35:45 -0500

On 12/7/12 10:28 AM, "Eric Osterweil" <[email-masked]> wrote:

>Seriously Doug, this totally misses the point.  If there is a fundamental
>difference, all the engineering in the world will only be masking the
>fact that the architecture is different, has different scaling
>properties, different usage models that it is suited for, etc.  Can a
>model that is inapt for certain requirements be buffed up to service them
>anyway?  Sure.  Does that make it ideal?  Doubtful, but ymmv...
>
>I think I see the disconnect here...  I don't know why you'd presume that
>_I_ think loading a router's RIB from DNS queries is a good idea.  I
>certainly don't recall saying that, and I am certainly not advocating
>that.

The rest we have beaten to death so I won't try again.  I was not
suggesting/discussing loading a RIB from DNS queries.   I was thought we
were discussing information systems that might allow me to validate the
origin of an router's RIB.   That problem is O(500K) at time zero.

>But the Spruce Goose was killed by yellow journalism, and the Titanic
>just couldn't manage the open seas with such a tiny rudder (point,
>iceberg)... ;-}

Actually the Spruce Goose floated fine, but was completely ill designed /
engineered to fly.   The Titanic sank because of the poor, under spec
metal quality used in its rivets.  A NIST metallurgist proved this some
number of years back and got a lot of press for it.

>*sigh*  Apparently, this will just be a point where we have to agree to
>disagree.  I feel that (as engineers) we should strive to find the right
>tool for the right job, but ymmv.
>
>Right, and my response was that (even though I have not done any specific
>analysis on this, in this context), BGP tends to defy the notion of being
>in steady state, so without a precise comment like the above, it is hard
>to agree.  Also, I'd argue that after a router boots, I've _heard_ they
>tend to face a lot of churn that isn't steady.  I was suggesting a study
>be done, but that's just my 0.02.
>

---

From: Arturo Servin <[email-masked]>
Date: Fri, 07 Dec 2012 15:42:18 -0200

You are twisting the fact when you are mixing hosted-rpiki and
rpki-repositories as the same thing.

> So you're saying that the hosted RPKI system is not reliant on routing?

	No, I am saying that rpki hosted (the web interface) is not critical
for RPKI.

> Or am I "twisting facts to my convenience" again?

	A bit, you are (in my understanding) are still mixing hosted-rpiki and
rpki-repositories.  ;-)

	Now, how to make the rpki-repositories more robust, there are people
already working on that. 	

Regards,
as

---

From: Russ White <[email-masked]>
Date: Fri, 07 Dec 2012 13:24:04 -0500

> rpki-repositories as the same thing.

Please feel free to point to the place in any message when I actually
said they are the same thing. They are part of the same system, not the
same thing --and I don't think I ever claimed otherwise.

> 	No, I am saying that rpki hosted (the web interface) is not critical

Depends on who's ox is being gored, I suppose. For the provider/RIR,
it's not so crucial. For the customer, it seems like it might be
(depending on the customer, etc.). I would think that if I were losing x
million of dollars per minute because my site is unreachable, I might
consider being able to reach the services required to bring my site back
onto the web to be somewhat important... Along with the time required to
propagate information through that system, etc.

> 	A bit, you are (in my understanding) are still mixing hosted-rpiki and

No... There is an entire ecosystem here, not just one "thing." There is
the certificate structure, four or five kinds of certificates
themselves, the data replication service (possibly multiples), a set of
protocols for getting the information between the replicating server and
the router, and now, from what I understand, a hosted service as well...

The point Eric was making is that each piece of the system adds
complexity. The counter was, "what complexity?" My answer is --each and
every piece adds complexity, no matter whether each particular piece is
considered "crucial," etc.

More moving parts == more complexity

Sometimes internal complexity is a good tradeoff for external
simplicity, but that doesn't make the complexity issues, along with the
brittleness, ossification, and systemic dependencies go away, it just
hides them.

:-)

Russ

---

From: Christopher Morrow <[email-masked]>
Date: Fri, 7 Dec 2012 16:54:35 -0500

> origin of an router's RIB.   That problem is O(500K) at time zero.

backing up a bit in the thread, and I hope/think setting some things
up a bit better for the conversation... or attempting to :)

If we look at the whole system (or a bunch of it) in SIDR/BGPSEC/RPKI,
there are likely these moving parts:

  o RPKI repositories (some number, let's say 1/ASN for simple numbers)
  o RPKI TAL/TA/'Root' bits (say 5 today, hopefully 1 tomorrow which then
    lets you walk down the tree to find all the actors)
  o network operators running networks (again 1/ASN)
  o gathering hosts/systems at each of the above would talk to all
repositories and
     gather the content for local use/distribution (again 1/ASN at
least, probably safe
     to assume 2/ASN at least)
  o cache systems inside each ASN, more than 1, less than 1/router seems sane?

In the end, the last item is completely up to the AS operator in
question. They may choose to run 1 cache/router, or one for their ASN,
they are responsible (according to the docs) to keep their cache's
semi-coherent, or as close to coherent as they can.

So, looking at timing information there's a base time for: "Make a
ROA/EE/etc change to the local repository", that timing is almost up
to the local operator, then things beyond that are about automatic...
first gatherers get data, then local-caches will get sync'd and
distribute to the routers the updates required. It's important to note
that a smart solution would only pass updates to the caches, or rather
the caches would update from some point-in-time that the gatherers
kept. (again, this is likely dependent upon the local operator's
timing requirements/design).

One point that's brought up a bunch on the thread so far is 'cold
start'. There are many forms of this:
  o for a router
  o for a cache
  o for a gatherer
  o for an ASN
I think for some the answer is 'easy', and the timings are 'fast'. For
others though the timing is longer.

For instance, a router cold-start (presuming no special knob request) is:
  1) boot
  2) load-os/config
  3) prefer internal/igp
  4) load cache-data
  5) bring up bgp (e and i) sessions
(it's probably harder to determine if 3 and/or 5 happen before 4
today, but that seems like a vendor tweak to me)
Loading the cache data should be essentially lan-speed limited... or
at least limited by the cache deployment that the operator picks.

A gatherer cold-start is: "Fetch all objects from all remote
repositories" and is likely bounded by times calculated in eric's
paper... or similar. 1/asn links with X average time to
connect/download/digest...

An ASN cold-start is ... actually pretty simple, except that they rely
upon everyone else finding them, so they are likely bounded on start
by the time it takes all remote-asns to walk the system and find them
(call it 4hrs based on the timings in the ops-docs?).

I think the discussion so far has centered around 'all' of the system,
but has variously talked about only 1-2 parts (really) when it comes
to timings. Could we think about the problem-space and timings in the
above framework? or alter the above to something we can all agree
upon?

-Chris

---

From: Christopher Morrow <[email-masked]>
Date: Fri, 7 Dec 2012 16:56:34 -0500

>> rpki-repositories as the same thing.

(*note quoting russ's message, but not actually aiming at russ in particular)

could we focus a bit of the conversation on the issues of scaling and
timing and not on this one particular bit of the puzzle which is
optional anyway? (hosted repositories) I don't think it helps to focus
on 'to host or not' since really the problem of timing is not about
where the repository is (as much, corner cases aside) as that there is
a repository.

thanks!
-chris

---

From: "Carlos M. Martinez" <[email-masked]>
Date: Fri, 14 Dec 2012 15:07:39 -0200

Hi Russ,

> Modulo the propagation time...

True, but that's hardly the fault of the hosted model IMO. It's the
whole architecture.

> advertised into the ROA system?

Maybe. But that doesn't disqualify hosted mode as a viable model for
some users. The key resides in balancing risks and benefits. As I said
in an earlier email, I am convinced that hosted RPKI provides just what
a large slice of the community needs.

This doesn't mean hosted is perfect, or even that it's the 'best' for
everyone.

> and convenience here, right?

There definitely is and I can live with it :)

Taking your example, banking is probably more insecure now that you can
bank online, but who's parting with the convenience ? I think tradeoffs
are everywhere and dealing with them appropriately is part of doing good
system design.

Again, if you are small / medium, hosted is probably right for you. If
you are a large ISP, definitely it is not.

And, the good thing above all, hosted is optional. If you are small /
medium but still want to take the matter into your own hands, you are
more than welcome.

> involved be documented someplace?

By all means. But, regarding the whole system, what personally worries
me the most is the full fetch model... I think we need to work on that.
Hosted really doesn't bother me at all.

~Carlos

---

From: Eric Osterweil <[email-masked]>
Date: Mon, 17 Dec 2012 14:35:41 -0500

Hi all,

I mentioned in an earlier email that we have been preparing a rather major revision to our initial sizing tech note.  As we mentioned in this thread, and in the tech note itself, we are very much interested in feedback,.  To that end, a number of comments from people (on and off list) were very helpful in correcting some of our thinking, and informing some of our analyses, thank you!

With apologies for the protracted delay, the new tech note is now available at:
	http://techreports.verisignlabs.com/tr-lookup.cgi?trid=1120005&rev=2

Some of the more substantial changes are:
- Our core formulation for estimating the number of objects changed according to feedback from people on the list.
- We added a treatise on ``deadline'' scheduling (i.e. what do things look like if RPKI crawls have to be done in $t$ seconds).
- We removed the x2 factor from our introduction and conclusions (as people on the list felt that this was not well supported).
- We broach the almost doubling of time estimates that the RPKI will undergo during algorithm rollover periods (``months or years'' in duration).
- We summarized our findings a bit more.

There are other changes throughout, but we continue to feel that a living document like this one is critical to understanding the way in which a global RPKI may be likely to perform, and informing the community.

Ultimately, with the measurements that we have seen today, we still calculate crawl times taking between about 15 and 30 days.  The section on deadline scheduling essentially finds that if we are to be able to reliably count on crawls from all RP caches (throughout the Internet) being done by 24 hours, the average repository in the global RPKI has to be orders of magnitude faster than the measurements we have seen.  

Thanks,

Eric

PS - In the spirit of full transparency, I have attached the _incredibly_ complicated script that generated our Figure 1... ;)