Subject: Re: [domainrep] At long last, some drafts and a charter

From: Nathaniel Borenstein <[email-masked]>
Date: Sun, 17 Jul 2011 11:00:35 +0100

I have a few belated comments, catching up in preparation for QC:

-- The concern over meta-reputation is, I suspect, overblown.  By analogy, consider credit ratings:   A few well-known entities are generally trusted (to a point), their reputations broad enough and their number small enough to obviate the need for an automated mechanism to assess them.  This is not to say that it wouldn't be useful to find out if people prefer TransUnion to Experian, merely that it isn't necessary to have a lot of infrastructure to know that they're both fairly well respected, and that it's not a huge mistake to set up a direct relationship with them.  I've been working on the assumption that a few fairly well-known reputation authorities will suffice to provide reputation information about a much larger world.  I agree with Murray that it would be possible, in the current proposed framework, to set up "a reputation vocabulary for describing reputation services" but I doubt that it's necessary.  For purposes of preventing phishing in the bank industry, for example, it would be reasonable to simply identify and trust a banking industry group to list the entities that are known to be banks (citibank yes, cit1bank no).

-- John makes a good point when he writes

> I suspect that people would be happier with a ZIP-CODE field with a five digit value rather than 100,000 assertions like IS-ZIP-90210. 

My guess is that there's not much point in using reputations -- positive OR negative -- for matters that aren't controversial, or where people have no incentive to lie.  

-- My first response to the issue of scaling from 0 to 1 vs -1 to 1 was a big yawn, because they're obviously functionally equivalent, but then I realized we're arguably talking about two different ways of thinking about it.  Peter's suggestion implies that a reputation starts at 0 and moves up or down based on more data, while the 0-1 model is simply envisioning a scale from bad to good; 0.5 on that scale is functionally equivalent to 0 on Peter's scale, but doesn't as obviously carry with it any assumptions about how much data you have.  We tried to address that, in the drafts, by giving the number of data points.  I like the explicitness of that, but it may not be as clear.  If you're trying to express a neutral reputation, 0 seems more obvious than 0.5.

I look forward to seeing many of you in Quebec -- travel safely!  -- Nathaniel

---

From: Alessandro Vesely <[email-masked]>
Date: Wed, 20 Jul 2011 13:06:37 +0200

On 17/Jul/11 12:00, Nathaniel Borenstein wrote:
> -- My first response to the issue of scaling from 0 to 1 vs -1 to 1
> was a big yawn, because they're obviously functionally equivalent,
> but then I realized we're arguably talking about two different ways
> of thinking about it.  Peter's suggestion implies that a reputation
> starts at 0 and moves up or down based on more data

While they may seem functionally equivalent at a purely numerical
glance, they are not.  The point with negative ranges is that they are
used as synonym for blacklisting.  A domain-based reputation system
cannot be used as an effective blacklist:  Otherwise, spammers can use
it as an adviser for changing domain names at the right time.  Hence
the blacklist won't block them, although we hypothesized it is
effective.  Contradiction.

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Wed, 20 Jul 2011 10:31:32 -0700

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of Alessandro Vesely
> Sent: Wednesday, July 20, 2011 4:07 AM
> To: [email-masked]
> Subject: Re: [domainrep] At long last, some drafts and a charter
> 

This seems like a blanket statement to me.  What a negative value means really depends on with what assertion it is being associated.

I suspect if consensus is to allow [-1, 1] as the range, then the declaration of a new assertion will need to explain how a consumer is expected to apply a negative value or, indeed, if they are allowed at all for that assertion.  For example, I suggest that in general we say a value of 1 agrees strongly with the assertion being made, 0 disagrees with it, and -1 disagrees and asserts the opposite.

SENDS-HIGH-VOLUME might be one for which negative numbers make sense, where values tending towards 1 mean "yes, this sender sends a huge amount of mail", 0 means "this sender sends an average amount of mail", and -1 means "no, this sender actually sends very little mail".

On the other hand, an assertion of SENDS-VIRUSES is one where I can't imagine what a negative assertion might mean (i.e., what's the opposite?).

---

From: Peter Saint-Andre <[email-masked]>
Date: Wed, 20 Jul 2011 11:41:01 -0600

<hat type='individual'/>

> On the other hand, an assertion of SENDS-VIRUSES is one where I can't imagine what a negative assertion might mean (i.e., what's the opposite?).

Who defines the assertions? How do we avoid overlap and confusion 
between assertions?

For, say, an SMTP or XMPP server I could see assertions like this:

- SUPPORTS TLS USING A PKIK CERTIFICATE
- HAS A CA-ISSUED CERT
- HAS A SELF-SIGNED CERT
- HAS AN EXPIRED CERT

It's not clear how to usefully interpret the results if all of those 
assertions are in play. :)

It also seems that in some systems people might think they can aggregate 
scores on particular dimensions into general scores. That might be a 
challenge if the measurements are overlapping or not commensurate.

To feel free to tell me this is already defined in a document I haven't 
had time to read yet...

Peter

-- 
Peter Saint-Andre
https://stpeter.im/

---

From: Nathaniel Borenstein <[email-masked]>
Date: Thu, 21 Jul 2011 09:04:43 +0100

> On the other hand, an assertion of SENDS-VIRUSES is one where I can't imagine what a negative assertion might mean (i.e., what's the opposite?).

I think the idea is that 1 would mean definitely yes, -1 would mean definitely no (doesn't send viruses), and 0 would mean the rater has no clue, while 0.5 would mean the rater has seen some evidence of sending viruses but it's not overwhelming.   (Sending one virus can easily be an accident, ask your clueless users.)

While I still think it's (obviously) mathematically equivalent to a [0,1] scale, the [-1,1] scale may actually be more comprehensible.  But it feels like the kind of thing that's so fundamentally minor that we will end up fighting over it tooth and nail.  :-(

On Jul 20, 2011, at 6:41 PM, Peter Saint-Andre wrote:

> Who defines the assertions? How do we avoid overlap and confusion between assertions?

The assertions are defined as part of the vocabularly, which is defined for each application/instantiation of the protocol.  So in your examples, you'd be getting the assertions from an SMTP vocabulary, an XMPP vocabulary, etc.  -- Nathaniel

---

From: Alessandro Vesely <[email-masked]>
Date: Fri, 22 Jul 2011 10:03:38 +0200

On 21/Jul/11 10:04, Nathaniel Borenstein wrote:
> I think the idea is that 1 would mean definitely yes, -1 would mean
> definitely no (doesn't send viruses), and 0 would mean the rater
> has no clue, while 0.5 would mean the rater has seen some evidence
> of sending viruses but it's not overwhelming.   (Sending one virus
> can easily be an accident, ask your clueless users.)

Oh well, I'm sorry for my "blanket statement", I thought we were
talking about executive results.

> While I still think it's (obviously) mathematically equivalent to a
> [0,1] scale, the [-1,1] scale may actually be more comprehensible.
> But it feels like the kind of thing that's so fundamentally minor
> that we will end up fighting over it tooth and nail.  :-(

For basic assertions, I don't think scaling makes much sense.  For
example, one can scale the ESTIMATED-USERS by 7G so as to obtain a
probability in the [0, 1] range, but such practice alters the nature
of the original statement somewhat.  Another example is dates, useful
to express qualities such as REGISTERED-ON, FIRST-HAM-EVER-SENT.

It may be preferable to have a few homogeneous types, identified by
ranges --not necessarily numeric--, and leave any scaling to the
"secret sauce" recipes.

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Fri, 22 Jul 2011 01:26:36 -0700

> Sent: Friday, July 22, 2011 1:04 AM
> To: [email-masked]

I don't think those are qualities that lend themselves to metrics of reputation.  Those are absolutes, rather than expressions of opinion about an entity.

The former would be something I'd expect to get out of WHOIS, for example.

---

From: Alessandro Vesely <[email-masked]>
Date: Fri, 22 Jul 2011 12:26:40 +0200

On 22/Jul/11 10:26, Murray S. Kucherawy wrote:
>> From: domainrep On Behalf Of Alessandro Vesely
>> 
> I don't think those are qualities that lend themselves to metrics of reputation.

How do you define "metrics of reputation"?

> Those are absolutes, rather than expressions of opinion about an entity.
> 

Yes, there's some overlapping.  It is not clear to what extent WHOIS
information is going to be complemented and not rate limited.  The
existence of an abuse mailbox and, possibly, the number of complaints
sent to it may also be interesting.

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Fri, 22 Jul 2011 07:04:52 -0700

> Sent: Friday, July 22, 2011 3:27 AM
> To: [email-masked]

The dictionary defines reputation as "the estimation in which a person or thing is held, especially by the community or the public generally".  It thus seems to me that your two suggested qualities might be things one uses in computing a reputation, but are not themselves reputations.

You might use a domain registration date as part of the overall data set you use to assert a SUSPECT reputation, for example, where that value tends towards 1 as the registration date is more recent.

---

From: "David F. Skoll" <[email-masked]>
Date: Fri, 22 Jul 2011 10:11:06 -0400

On Fri, 22 Jul 2011 07:04:52 -0700
"Murray S. Kucherawy" <[email-masked]> wrote:

> The dictionary defines reputation as "the estimation in which a
> person or thing is held, especially by the community or the public
> generally".  It thus seems to me that your two suggested qualities
> might be things one uses in computing a reputation, but are not
> themselves reputations.

Well, of course.  The only things you should collect about a system are
objective facts.  In the email realm, it is objectively true that
machine 10.1.2.3 tried to send a message to a nonexistent recipient,
or that machine 10.2.3.4 send an email that was classified as spam by
a human.

The subjectivity comes when you combine knowledge of facts to form an
opinion about the system.  I don't believe your charter gives any
guidance about how to do this (and it probably shouldn't.)

Regards,

David.

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Fri, 22 Jul 2011 07:26:33 -0700

> From: [email-masked] [mailto:[email-masked]] On Behalf Of David F. Skoll
> Sent: Friday, July 22, 2011 7:11 AM
> To: [email-masked]

I agree, although defining "reputation" by citing the dictionary might actually be helpful in framing the work and providing guidance about assertions that are reasonable to pursue.