Subject: [apps-discuss] Aggregated service discovery

From: Cyrus Daboo <[email-masked]>
Date: Tue, 22 May 2012 11:00:22 -0400

Hi folks,
Today many protocols define their own "service discovery" protocols (e.g., 
<http://tools.ietf.org/html/rfc6186>, 
<https://datatracker.ietf.org/doc/draft-daboo-srv-caldav/>, 
<http://tools.ietf.org/html/rfc6125>).

>From a client perspective, these each work fine individually. But more 
often than not, a client device actually wants to be able to discover all 
services a "service provider" has available or provisioned for the user. 
i.e., a user expects email, calendar, contacts, IM, directory etc to be 
setup in one step by the client, rather than having to go and setup each 
service separately. Whilst a client can present a single UI for such an 
"aggregated service discovery" it still has to go use separate discovery 
protocols for each service. This is expensive - lots of separate DNS 
lookups, etc.

Several proprietary systems offer and "aggregated service discovery" 
protocol - in its simplest form a GET on a well known URI that returns an 
XML document listing available services and other useful configuration 
information.

It is time we had such a protocol for the IETF standard suite of protocols. 
In particular implementors involved in the Calendaring and Scheduling 
Consortium are very keen to see a good solution to this problem. So I am 
starting a discussion on this here to solicit some ideas about how best to 
approach this, with a view to writing a draft.

The obvious, and simplest approach, is the HTTP GET on a .well-known URI 
returning an XML or JSON document with a standard "schema".

Another possibility is to leverage the webfinger work. I'd like to hear 
from webfinger experts as to whether a use case like this would be a 
reasonable solution. Some concerns I have are the security "scope". 
Obviously service discovery is something that a user does for themselves 
and their service information should be private, which seems somewhat 
contrary to the primary user case for webfinger whether other users are 
looking up the information.

Security, simplicity and efficiency are the key goals for this protocol.

Comments, ideas?

-- 
Cyrus Daboo

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 23 May 2012 16:29:24 +1000

If they're HTTP - <https://datatracker.ietf.org/doc/draft-nottingham-json-home/>

Not yet convinced that well-known is needed for this yet; it's effectively substituting a hostname for a full URL.

Cheers,

On 23/05/2012, at 1:00 AM, Cyrus Daboo wrote:

> Hi folks,
> Today many protocols define their own "service discovery" protocols (e.g., <http://tools.ietf.org/html/rfc6186>, <https://datatracker.ietf.org/doc/draft-daboo-srv-caldav/>, <http://tools.ietf.org/html/rfc6125>).
> 
> often than not, a client device actually wants to be able to discover all services a "service provider" has available or provisioned for the user. i.e., a user expects email, calendar, contacts, IM, directory etc to be setup in one step by the client, rather than having to go and setup each service separately. Whilst a client can present a single UI for such an "aggregated service discovery" it still has to go use separate discovery protocols for each service. This is expensive - lots of separate DNS lookups, etc.
> 
> Several proprietary systems offer and "aggregated service discovery" protocol - in its simplest form a GET on a well known URI that returns an XML document listing available services and other useful configuration information.
> 
> It is time we had such a protocol for the IETF standard suite of protocols. In particular implementors involved in the Calendaring and Scheduling Consortium are very keen to see a good solution to this problem. So I am starting a discussion on this here to solicit some ideas about how best to approach this, with a view to writing a draft.
> 
> The obvious, and simplest approach, is the HTTP GET on a .well-known URI returning an XML or JSON document with a standard "schema".
> 
> Another possibility is to leverage the webfinger work. I'd like to hear from webfinger experts as to whether a use case like this would be a reasonable solution. Some concerns I have are the security "scope". Obviously service discovery is something that a user does for themselves and their service information should be private, which seems somewhat contrary to the primary user case for webfinger whether other users are looking up the information.
> 
> _______________________________________________
> apps-discuss mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/apps-discuss

--
Mark Nottingham   http://www.mnot.net/

---

From: Michiel de Jong <[email-masked]>
Date: Wed, 23 May 2012 10:33:34 +0000

IMO, webfinger/swd is the way to go. they are currently being merged
into one. All discovery paths should use webfinger/swd as the first
step, and then do other stuff (including requiring credentials) in
documents linked from there. There are cases where a service is
specific to a domain, but not to a user, but I think they should still
also be announced from the same first starting point (which is
/.well-known/host-meta).

how to deal with private information (meant only for the user
themselves), is not very well documented. the webfinger/swd spec
basically leaves it out of scope. Basically what you would do IMO is,
for a user "<user>@<host>", announce a first starting point at
https://<host>/.well-known/host-meta, and then use "follow your nose"
to discover everything else. That includes discovering the home-pages
of any domain-specific APIs, as well as caldav, BrowserID, OpenID,
ActivityStreams, foaf, PoCo, remoteStorage, email addresses, avatars,
and everything else. The first starting point should be available
without credentials, publicly, and with CORS headers on there. Then as
you follow the links to all these services, you will find barriers
where maybe a bearer token or a client-side certificate or something
else is needed to retrieve the next bit of information.

But the first starting point should always be public, on
/.well-known/host-meta and with CORS headers on there. Even if it's
just to say "nothing to see here unless you can give me credentials of
type X" (IMO, OAuth end-point discovery can itself serve here as a
syntax for expressing that, although i think announcing
credentials-requirements is still a relatively under-explored part of
discovery best practices).

Cheers!
Michiel

---

From: Cyrus Daboo <[email-masked]>
Date: Wed, 23 May 2012 08:55:22 -0400

Hi Mark,

--On May 23, 2012 4:29:24 PM +1000 Mark Nottingham <[email-masked]> wrote:

> If they're HTTP -
> <https://datatracker.ietf.org/doc/draft-nottingham-json-home/>

Of course we do want service discovery for non-HTTP protocols, LDAP, IMAP, 
Submission, POP3, XMPP etc. Now I could almost imagine having non-http 
resource URIs in the json-home document for these other protocols, but I 
think that would be over stepping the intent of json-home.

> Not yet convinced that well-known is needed for this yet; it's
> effectively substituting a hostname for a full URL.

Not sure what you mean by that. Obviously it is important to have just one 
jumping off point for this.

-- 
Cyrus Daboo

---

From: Graham Klyne <[email-masked]>
Date: Wed, 23 May 2012 14:26:18 +0100

On 23/05/2012 13:55, Cyrus Daboo wrote:
> Submission, POP3, XMPP etc. Now I could almost imagine having non-http resource
> URIs in the json-home document for these other protocols, but I think that would
> be over stepping the intent of json-home.

I don't see why it would (assuming you mean the URIs obtained via the URI 
template in the home document).  HTTP would still be the discovery protocol, but 
it could easily provide reference to a non-HTTP service.

Indeed, I think web architecture principles would suggest that non-HTTP services 
should be discoverable (by virtue of not restricting the kind of URI that can be 
returned).

#g
--

---

From: Cyrus Daboo <[email-masked]>
Date: Wed, 23 May 2012 14:25:44 -0400

Hi Michiel,

> discovery best practices).

Efficiency is one of the key goals for this. If the ASD ("aggregated 
service discovery") app has to follow multiple links then the benefit 
rapidly decreases over simply using the current per-service mechanisms. I 
do recognize that there may be a need to do that (e.g., services whose 
administration is delegated within a domain) but it would really be better 
to get all the required information in a single request. The other aspect 
of that is also avoiding getting redundant information. The ASD app may 
well know precisely which services it can support so being able to have the 
query filtered for just those would be good to avoid extraneous information 
being sent in a large document.

Whilst host-meta might be a jumping off point for this, its worth 
remembering that we are dealing with more than just "web" services here.

-- 
Cyrus Daboo

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 24 May 2012 10:27:09 +1000

On 23/05/2012, at 10:55 PM, Cyrus Daboo wrote:

> Of course we do want service discovery for non-HTTP protocols, LDAP, IMAP, Submission, POP3, XMPP etc. Now I could almost imagine having non-http resource URIs in the json-home document for these other protocols, but I think that would be over stepping the intent of json-home.

As long as the relationship is expressed as a well-defined link relation, I don't see a problem. Of course, some of the HTTP-specific bits are going to be irrelevant, but there isn't anything requiring their use (maybe I should call it "http-hint"?).

> Not sure what you mean by that. Obviously it is important to have just one jumping off point for this.

1) What's the use case driving having ONE location for it? Some of the advice in .well-known is to lump things that share a use case into a single URL, but to avoid having "kitchen sink" well-knowns, because they become unwieldy. Is there a strong use case for someone discovering ALL of the various (and often totally unrelated) services a site offers in one request?

2) Even if it's so, the question I'm asking is why that ONE identifier is a hostname instead of a URL. Are there some hidden UI requirements at work here, perhaps?

Cheers,

---

From: Andrew McMillan <[email-masked]>
Date: Thu, 24 May 2012 16:54:31 +1200

On Thu, 2012-05-24 at 10:27 +1000, Mark Nottingham wrote:
> > Of course we do want service discovery for non-HTTP protocols, LDAP,
> IMAP, Submission, POP3, XMPP etc. Now I could almost imagine having
> non-http resource URIs in the json-home document for these other
> protocols, but I think that would be over stepping the intent of
> json-home.
> 
> As long as the relationship is expressed as a well-defined link
> relation, I don't see a problem. Of course, some of the HTTP-specific
> bits are going to be irrelevant, but there isn't anything requiring
> their use (maybe I should call it "http-hint"?).
> 
> 
> 1) What's the use case driving having ONE location for it? Some of the
> advice in .well-known is to lump things that share a use case into a
> single URL, but to avoid having "kitchen sink" well-knowns, because
> they become unwieldy. Is there a strong use case for someone
> discovering ALL of the various (and often totally unrelated) services
> a site offers in one request?

Here's some real background, that I hope exposes some of the value that
this could offer...

I started working in a new company on the 2nd of May.  On arrival they
issued me with a username and a password that applied to the following
services:

* IMAP
* CalDAV
* CardDAV
* LDAP
* Wiki
* Continuous Integration Testing system
* Problem tracking system
* SMTP
* SIP
* XMPP
* Wireless LAN

and quite possibly a few more that I have yet to discover.  This is only
a relatively small company of about 150 staff, so I think it's pretty
stellar that they've managed to get all of these things singing and
dancing to the same username and password.

What they have not done, and what they *cannot* do, is to leave me to it
at that point, and I (or my software) will be able to figure everything
out for itself.

What I (and Cyrus and others from CalConnect) would like to see would be
a unification of the process of discovering these things.

Currently IANA maintains a registry of well-known ports, more recently
this has been extended to include SRV details, and there is also a
registry of .well-known URLs.  What is missing is a registry of any
parameters which apply to a service.

For example, to configure my SMTP service, I needed to know that it was
using port 465, that SSL was required, and that plain text
authentication was used in the SSL channel.  I have other mail servers I
connnect to on port 8487, with TLS, and although these variations on the
protocol are quite well-understood by the server and the client software
they tend to require a lot of poking and prodding from the client to
discover what is the right answer.  And without any ability to control
what is the preferred answer.

Having configured my SMTP service I then had to configure my IMAP,
CardDAV and CalDAV all in the one program.  Obviously that's a pretty
bad UI in that particular program, but it's far from unique.

I would like to see the sourcing of this information become the kind of
service that the operating system can offer to programs.  So that I can
configure the absolute minimum necessary information about my employer
(perhaps a domain name, a username and some authentication details) and
then as I configure (e.g.) my new XMPP application it offers me a list
of the ones it knows about by querying this central source "which
accounts have an associated XMPP service?".  When I select the account I
need enter nothing more, because the people who knew enough to set it up
in the first place have provided all of the relevant information as
client configuration parameters for that service, and my device has
discovered them.

If we are ever going to unify these kinds of things into a single place
in the operating system then having standards for a registry and a
framework for such service parameters will be what makes it possible.

Microsoft currently have an autodiscovery document which some people are
also using to provide information for non-Microsoft services in
non-standard ways.  I believe that Apple also have some internal
standards for retrieving centralised service parameters.

Neither of these approaches is a standard, and from what I have seen of
them they are somewhat built of duct tape and necessity and are not
appropriate for use other than to inform us of some of the parameters
that will be needed for some of the services.

In this sense I think that json-home would be an acceptable transport
for such information, but I would be uncomfortable (to say the least)
with trying to impose a pre-defined structure that would suit the
definition of all current and future services.  For this reason I think
that a per service registration of parameter names (perhaps with some
lightweight definition of associated data structures) would be the best
solution.

> 2) Even if it's so, the question I'm asking is why that ONE identifier
> is a hostname instead of a URL. Are there some hidden UI requirements
> at work here, perhaps?

The UI would be that I have three 'facts' and I would like to use these
to configure my $SERVICE.  I'd like to be able to supply the same three
facts for any service and hope that my software was able to hide further
complexity from me, ideally supplying them only once, and authorizing
each client on first use.

I believe those facts should be:

* A domain name
* A username
* Some authentication token(s)

Of course the domain name could be a URL, but in providing a standard
simplification a domain name offers the advantage that it is shorter
than a URL, and if we then define a set of standard transformations to
that domain name (such as an SRV lookup to discover protocols, hosts and
ports, and appending some standard path) we can save my mother a lot of
laborious and error-prone data entry on her smartphone.

Regards,
					Andrew McMillan.

-- 
------------------------------------------------------------------------
http://andrew.mcmillan.net.nz/                     Porirua, New Zealand
Twitter: _karora                                  Phone: +64(272)DEBIAN
        Life is knowing how far to go without crossing the line.
------------------------------------------------------------------------

---

From: Michiel de Jong <[email-masked]>
Date: Thu, 24 May 2012 14:11:04 +0000

> remembering that we are dealing with more than just "web" services here.

sure, so for xmpp service it makes sense to use the existing xmpp
disco, which is non-http. but the question is whether it makes sense
to have a way to /list/ services, rather than relying on the app to
know what to ask for.

if you're going to provide a list, then that has to be on 1 specific
platform. http seems a good choice for that.

it's a choice between server-side aggregation or client-side
aggregation of this list.

---

From: Michiel de Jong <[email-masked]>
Date: Thu, 24 May 2012 14:30:03 +0000

> laborious and error-prone data entry on her smartphone.

i find this a super interesting question. i have been thinking about
this as well, and i think we don't have a good solution for this right
now, so it would be cool to change that. aspects i see:

1: Existence - listing which services even exist
2: Version - which version of the spec the instance claims to be compliant with
3: End-points - this is service-specific
4: Options - which types of encryption, signature, encodings, etc. are supported

i think these 4 types of discoverable information make sense for most
services. And i think if we combine webfinger/swd with json-home, then
we probably have enough expressive power to announce these 4 types of
information in a sensible way.

but the main objection people would have to doing this is i think
privacy/security. you don't want to announce the exact details of all
your services publically, because:
1) it makes it easier for an attacker to know where to attack your systems
2) it may reveal non-public information about your users unnecessarily.

Given that the user story Andrew describes (and i think this is the
correct thing to be looking at), starts out with a domain identifier,
a user identifier within that domain, and some authentication
token(s). If we have a trusted client then we can already use those
tokens during the discovery. In other cases we would have to use maybe
OAuth or similar to get derived tokens for doing the actual discovery.
If we do that, then we need to announce that that is the case. In the
case where you use resource owner password credentials, then i can see
how you could discover everything you want in 1 round trip. if you use
derived credentials (e.g. implicit grant), then i think 3 round-trips
are unavoidable: one to discover the discovery and OAuth end-points,
one to obtain the derived token, and one to do the actual lookup with
it.

My 2ct,
Michiel

---

From: "Paul E. Jones" <[email-masked]>
Date: Thu, 24 May 2012 12:19:10 -0400

Cyrus,

I apologize for not replying sooner.

I do believe WebFinger could be used to locate these services.  I wasn't
aware of RFC 6186 (and rushed right out to add it to my domain), but
WebFinger could be used to provide an even better level of granularity.  It
exists now, so I'm not sure if one would want to migrate to WebFinger, but
if it did I can imagine something like this:

GET /.well-known/host-meta?resource=mailto:[email-masked]

Returning...

{
  "subject" : "mailto:[email-masked]",
  "links" :
  [
    {
      "rel" : "config-email",
      "href" : "http://www.packetizer.com.com/config/email/?user=paulej"
    }
  ]
}

The above document would include a link relation that refers to mail server
configuration.  Querying the /config/email/ URI might return a JSON document
like this:

  {
    "imaps" : "cluster23.imap.packetizer.com",
    "smtp-submission" : "smtp.packetizer.com"
  }

(Note: this is a trivial example, whereas we'd probably need to indicate use
of SSL, TLS, login requirements, etc.)

It would accomplish something similar to SRV records, but provide a finer
level of granularity to allow domain owners to assign users to different
machines and specific configurations.

Paul

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]]
> On Behalf Of Cyrus Daboo
> Sent: Tuesday, May 22, 2012 11:00 AM
> To: [email-masked]
> Subject: [apps-discuss] Aggregated service discovery
> 
> Hi folks,

---

From: Peter Saint-Andre <[email-masked]>
Date: Thu, 24 May 2012 10:23:48 -0600

On 5/24/12 8:11 AM, Michiel de Jong wrote:
> disco, which is non-http.

Yes, we've had service discovery methods in XMPP for over ten years (in
fact the following spec superseded an older method in use since 1999):

http://xmpp.org/extensions/xep-0030.html

Because XMPP native includes presence "push", we also have a dynamic
profile of service discovery for real-time capability changes:

http://xmpp.org/extensions/xep-0115.html

Peter

-- 
Peter Saint-Andre
https://stpeter.im/

---

From: Alessandro Vesely <[email-masked]>
Date: Fri, 25 May 2012 10:15:15 +0200

> 2) it may reveal non-public information about your users unnecessarily.

Requiring authentication in order to discover the services would seem
to be a relevant functional difference w.r.t. SRV records.  I, for
one, don't use SRV records because of those two reasons.

Of course, directing all mass, blind dictionary attacks toward a
single entry point will call from some savvy implementation advice.
For example, centralized discovery could count failed attempts and
block a user when that number becomes comparable to her password's
entropy.  She won't be able to install new client devices for a while,
but that is much less disruptive than blocking IMAP access.

jm2c

---

From: Andrew McMillan <[email-masked]>
Date: Sun, 27 May 2012 15:43:53 +1200

> one, don't use SRV records because of those two reasons.

I can definitely understand that reasoning, and it's a good point in
favour of a more securable alternative.  You could also make the
relevant SRV records available only on your private networks, but then
that can result in a complex and fragile DNS setup.

> but that is much less disruptive than blocking IMAP access.

I can see plenty of value in an administrator putting the discovery
point inside a corporate (or ISP) firewall in order to reduce the value
to automated external attackers.  Or having alternative representations
of the discovery with only limited visible services for external
visitors.  Analogously, a manual system of providing these configuration
parameters could well place them inside a firewall on internal
documentation.

I would like to see a scheme which had significant utility with only a
single static document providing the information.  More advanced
implementations may well produce that document dynamically, but these
parameters are not expected to be very changeable values in most
installations.

Regards,
					Andrew.
-- 
------------------------------------------------------------------------
andrew (AT) morphoss (DOT) com                            +64(272)DEBIAN
        Open Source: the difference between trust and antitrust
------------------------------------------------------------------------

---

From: Michiel de Jong <[email-masked]>
Date: Tue, 29 May 2012 14:22:59 +0200

in the remoteStorage spec we are using webfinger to discover a
personal data service. relying on network topology for security is not
an option here, and nor are SRV records, because we want to be able to
link any unhosted html5 app to any remoteStorage server. This is still
very much a draft, and we're still changing the link format around,
but our current proposal is this:

http://www.w3.org/community/unhosted/wiki/Pds

Basically, our example (publicly) announces the following service
attributes for a Read-Write Web (rww) server over public
webfinger+CORS:

- service base address
- service API version
- auth method
- auth end-point

I would love to get feedback on they way we're doing that discovery
there, and if anybody knows a more standard way to achieve the same
thing.

Cheers!
Michiel.