Subject: Re: [Simple] WGLC: http://www.ietf.org/internet-drafts/draft-ietf-simple-interdomain-scaling-analysis-03.txt

From: Avshalom Houri <[email-masked]>
Date: Thu, 21 Feb 2008 14:23:20 +0200

Working on version 4. Following is some feedback on what I plan to do.
Appreciate if you can give me feedback ASAP (deadline for drafts this IETF 
is Monday 25th).

--Avshalom

Peter Saint-Andre <[email-masked]> wrote on 17/01/2008 01:18:58:

> Robert Sparks wrote:
> > This is a SIMPLE Working Group Last Call for Comments on
> > http://www.ietf.org/internet-drafts/draft-ietf-simple-interdomain-
> scaling-analysis-03.txt
> > 
> > Please review this draft carefully. Provide comments to the editor 
> > and/or the list no later than
> > Monday 21 January 2008.
> 
> I think this draft needs further work before it is ready for IETF Last 
> Call. As a point of comparison, I have just updated a similar analysis 
> for XMPP:
> 
> http://www.ietf.org/internet-drafts/draft-saintandre-xmpp-presence-
> analysis-03.txt
> 
> Herewith some comments.
> 
> 1. As I previously expressed on this list, I think it is not a good idea 

> to compare two different technologies in such a document. Therefore I 
> have removed any mention of SIP presence from the XMPP analysis (other 
> than to point to draft-ietf-simple-interdomain-scaling-analysis from my 
> I-D), and I suggest that the authors of the SIP/SIMPLE analysis do the 
> same. There are just too many differing assumptions and possible 
> scenarios to accurately compare two presence technologies in a single 
> document (let alone more than two technologies). Therefore I strongly 
> suggest the following:
> 

Will remove explicit references to XMPP in the text.

I am not sure about the removal to "other protocols" in the text.
I do not want to mean XMPP specifically but want to say that even with
all the optimizations that we can of (given SIP base) what should we 
expect.

> 1a. Modify the second paragraph of Section 1 ("Introduction") to remove 
> references to other presence technologies.
> 
> 1b. Remove Section 2.10 ("Other Protocols"). In particular, the 
> calculations in this section use SIP terminology, which patently does 
> not apply to non-SIP systems.

I think that it is important. I may need to change the language to make
it more eplicit that what we mean here that with all optimizations that
we can think of we still get X messages and Y bytes.

> 
> 1c. Remove the fourth bullet-point in Section 7 ("Conclusions"). In 
> particular, during discussion on this list the claims in this paragraph 
> were shown to be seriously misleading or false.

Agree. Will rephrase and remove XMPP. I probably should not have put it 
there in the
first place.

What do you think about adding the following paragraph somewhere in the 
text:

"The calculations here are based on the assumption that all data is 
carried in bound of the
protocol and no optimizations that enable getting the presence information 
via out bound means
are taken into account. These optimizations may improve the number of 
messages and number of bytes
significantly but they are out of scope for this document"

> 
> 1d. In several places the analysis claims to apply not just to 
> SIP/SIMPLE presence systems but to presence systems built using any 
> possible presence technology. Extreme care should be taken when making 
> such statements.

OK

> 
> 2. While working on the XMPP presence analysis just now, I attempted to 
> incorporate the fact that the number of users who are registered with a 
> presence service is typically much greater than the number of users who 
> have an active presence session with the service. In my experience 
> helping to run a fairly large (~300,000 user) consumer-oriented presence 

> and messaging service, typically at most 5% of the users are "online" at 

> any one time. The number of online users within an enterprise deployment 

> is typically higher, on the order or 40% online. To be conservative, in 
> the XMPP analysis I have assumed that 10% of the users at a service 
> provider deployment are online at any one time and that 50% of the users 

> at an enterprise deployment are online at any one time. I do not see 
> that this factor has been taken into account in the SIP/SIMPLE analysis 
> (though perhaps I am missing something). I think it is extremely 
> important to take the registered vs. online factor into account because 
> this is the usage pattern that presence systems experience in the real 
> world and those who read these presence analyses will quickly be scared 
> off if we don't do so (they will just look at the number of users and 
> assume that is the number of concurrent presence sessions).

Having 40K watchers between two domains or 20M watchers between two
very big domains does not seem to be that high. Getting official
numbers of number of users and rate of usage is almost impossible so
we did some educated guesses. Recall also that we are not modeling the
usage within a single domain but the numbers between multiple domains.
I think that the numbers are reasonable. Are there specific numbers that
seem to be too high?

> 
> 3. The real-world numbers in Section 2.2 ("Assumptions") are helpful, 
> but they are incomplete:
> 
> 3a. What are the numbers of registered users and concurrent users for 
> the "large consumer network"?
> 
> 3b. Are the 110 users (consumer network) and 200 users (enterprise) in 
> the watch list total users or federated users? Only the number of 
> federated users is important for this analysis.
> 
> 3c. Why is the peak number of instant messages mentioned for the 
> consumer network? This analysis deals only with presence.
> 
> 3d. It is mentioned that about 50% of the enterprise users are online at 

> any one time. What percentage of the users at the consumer network are 
> online at any one time? And (as mentioned) is this kind of percentage 
> factored into the calculations?

Numbers here are more for getting the sense of things. I was not able to
get the numbers or everythings since some of these numbers are considered
confidential. I will try to clarify.

> 
> 4. The inclusion of full vs. partial notifies, dialog optimizations, 
> NOTIFY optimizations, and so on makes the calculations very involved and 

> confusing to read (e.g., I have not yet had time to review them all in 
> detail). Can they be simplified somehow?

We need to show the effect of various optimizations in SIP. I am not sure
how to simplify this. Will try.

> 5. I think it might help to make the scenarios a bit more realistic, at 
> least by explaining how a particular scenario might apply in the real 
> world (e.g., "this scenario is similar to what might be observed in the 
> context of federation between two large service providers"). I have 
> tried to do that in version -03 of the XMPP analysis, but we could even 
> go farther.

OK

> 
> 6. When I attempted to address the issue of real-world relevance, I 
> found that several of the scenarios in the XMPP analysis diverged from 
> the scenarios in the SIP analysis. I would be happy to work directly 
> with the authors of draft-ietf-simple-interdomain-scaling-analysis to 
> sync up on the scenarios.

Can you elaborate more on this? Maybe we can discuss off-list.

> 
> That's it for now.
> 
> Peter
> 
> -- 
> Peter Saint-Andre
> https://stpeter.im/
> 
> [attachment "smime.p7s" deleted by Avshalom Houri/Haifa/IBM] 
> _______________________________________________
> Simple mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/simple
_______________________________________________
Simple mailing list
[email-masked]
http://www.ietf.org/mailman/listinfo/simple

---

From: Peter Saint-Andre <[email-masked]>
Date: Thu, 21 Feb 2008 11:07:50 -0700

> Appreciate if you can give me feedback ASAP (deadline for drafts this IETF 

I'm boarding a plane soon so I have only a few minutes to reply...

> all the optimizations that we can of (given SIP base) what should we 

So IMHO that is best expressed as "here is how a fully optimized SIP
presence technology would perform", not as "here is how some other
unmentioned technology would perform". Let the developers of each
presence technology try to specify how their technology meets the
presence scalability challenge. I think it muddies the waters to address
that in this document.

> we can think of we still get X messages and Y bytes.

Right. So let's describe this as "fully optimized SIP presence with all
the improvements we currently envision", not as the performance of any
other given presence technology no matter how that technology is
defined. I agree that there are probably general issues that apply to
any presence technology (since presence is a matter of pushing out
fairly frequent updates to a relatively large number of interested
parties), but I think those issues should be abstracted from the
treatment of each presence technology.

> Agree. Will rephrase and remove XMPP. I probably should not have put it 

Well I made the same mistake in version -01 of the XMPP analysis, so
don't worry about "should not have".

> significantly but they are out of scope for this document"

That seems reasonable.

>> possible presence technology. Extreme care should be taken when making 

I think I expressed this concern more clearly above. It's a matter of
abstracting from many instances of presence technologies (SIP/SIMPLE,
XMPP, IMPS, all the IM services out there, etc.), not assuming that what
applies to one of those technologies applies to all. And note that many
people with significant experience with presence technologies don't even
pay attention to this list (or at least I never see posts from folks at
Skype, Yahoo!, MSN, AIM, etc.), so generalizing too quickly might not be
a good idea. :)

> I think that the numbers are reasonable. Are there specific numbers that

My concern is that we need to address the fact that only a certain
percentage of the users are online at any one time. If someone who is
not deeply familiar with the terminology of watchers etc. looks at these
documents, they may think "oh I suppose a watcher is a registered user
instead of a connected users". So I think we want to make that a bit
clearer. Basically I'm just trying to forestall potential confusion.

> confidential. I will try to clarify.

Well, the numbers don't have to be exactly what a given service sees,
only realistic numbers for that type of service.

> how to simplify this. Will try.

Yes, it's difficult. I'm sorry that I don't have more helpful suggestions.

> Can you elaborate more on this? Maybe we can discuss off-list.

Please see the most recent version of the XMPP analysis I performed.
Unfortunately I am out of time so elaborate more fully, I'll try to do
that on-list or off-list soon.

Peter

-- 
Peter Saint-Andre
https://stpeter.im/