Subject: [perpass] What problems does perpass need to solve?

From: Dave Crocker <[email-masked]>
Date: Thu, 12 Sep 2013 13:18:18 -0700

Folks,

G'day.

While all of the very specific lines of discussion look quite 
interesting, it is not possible to tell whether any of these are 
important to the task of perpass, which was established to discuss:

    http://www.ietf.org/mail-archive/web/perpass/current/msg00000.html

    "privacy properties of IETF protocols and concrete ways in which
     those could be improved."

So my simple question is:

    What are the problems that need to be solved here?

Otherwise, all this activity is a random walk down some entertaining and 
possibly useful paths, but without any guidance, prioritization or 
systems view.

d/
-- 
Dave Crocker
Brandenburg InternetWorking
bbiw.net

---

From: Randy Bush <[email-masked]>
Date: Thu, 12 Sep 2013 16:58:52 -1000

> those could be improved."

to repeat my statement from berlin, privacy by default.  and as many
places as we can enable it.

randy

---

From: Patrick Pelletier <[email-masked]>
Date: Thu, 12 Sep 2013 20:12:08 -0700

>      those could be improved."

One obvious thing is the amount of (usually unnecessary) information 
leaked by the User-Agent field in HTTP.

Should we downgrade the User-Agent field (section 14.43 of RFC 2616) 
from a SHOULD to a MAY?

Or, if that's too radical, should we standardize a small number of fixed 
strings to use in the User-Agent field?  (For example, "Desktop/1.0" for 
desktop browsers, "Mobile/1.0" for mobile browsers, "Text/1.0" for text 
browsers like Lynx, "Batch/1.0" for non-interactive clients like curl 
which are performing a task more specific than crawling the web, and 
"Robot/1.0" for clients which are crawling the web?)

--Patrick

---

From: Dave Crocker <[email-masked]>
Date: Thu, 12 Sep 2013 20:44:44 -0700

> places as we can enable it.

If we -- the IETF community -- had a definition of privacy, that would 
help, but we don't.

If we knew what it meant to 'enable privacy' (nevermind whether by 
default), that would also help, but we don't.

So the idea of 'privacy by default' well could serve as a useful 
catch-phrase, but at the moment, it lacks technical substance.  We have 
no shared, technical understanding of its meaning.

No doubt you have particulars in mind.  No doubt lots of folks have 
their own set of particulars.  Unfortunately each person's particulars 
tends to be different.

So what we lack are a) anything like a specific technical meaning for 
privacy, b) any system-level sense of what it means to provide privacy, 
or c) any prioritization of privacy issues so that we can get the most 
benefit from initial efforts.

My question was meant to prompt discussion at a technical level.  There 
have been the tiniest wisps of comment on the list that might provide 
touchstones for this, but they haven't been pursued.  We need to pursue 
them.

Catch-phrases will be useful to summarize the results.

d/

---

From: Tim Bray <[email-masked]>
Date: Thu, 12 Sep 2013 20:55:35 -0700

Dave, there’s been some useful and grown-up discussion going on around
here.  I don’t think you’re trying to derail it, but you risk having that
effect. I think it would be better to ease off and see where it leads us,
because I’ve read some smart and enlightening things.

On Thu, Sep 12, 2013 at 8:44 PM, Dave Crocker <[email-masked]> wrote:

> catch-phrase, but at the moment, it lacks technical substance.  We have no
> shared, technical understanding of its meaning.
>
> No doubt you have particulars in mind.  No doubt lots of folks have their
> own set of particulars.  Unfortunately each person's particulars tends to
> be different.
>
> privacy, b) any system-level sense of what it means to provide privacy, or
> c) any prioritization of privacy issues so that we can get the most benefit
> from initial efforts.
>
> ______________________________**_________________
> perpass mailing list
> [email-masked]
> https://www.ietf.org/mailman/**listinfo/perpass<https://www.ietf.org/mailman/listinfo/perpass>
>

---

From: "Eduardo A. Suárez" <[email-masked]>
Date: Fri, 13 Sep 2013 01:24:17 -0300

> no shared, technical understanding of its meaning.

Here are something to understand what privacy means.

"Information privacy, or data privacy, is the relationship between  
collection and dissemination of data, technology, the public  
expectation of privacy, and the legal and political issues surrounding  
them.

Privacy concerns exist wherever personally identifiable information is  
collected and stored.

The challenge in data privacy is to share data while protecting  
personally identifiable information." [1]

"Internet privacy involves the right or mandate of personal privacy  
concerning the storing, repurposing, provision to third-parties, and  
displaying of information pertaining to oneself via the Internet." [2]

"Personally identifiable information (PII), as used in US privacy law  
and information security, is information that can be used on its own  
or with other information to identify, contact, or locate a single  
person, or to identify an individual in context." [3]

[1] http://en.wikipedia.org/wiki/Data_privacy
[2] http://en.wikipedia.org/wiki/Internet_privacy
[3] http://en.wikipedia.org/wiki/Personally_identifiable_information

ES.-

----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.

---

From: Leif Johansson <[email-masked]>
Date: Fri, 13 Sep 2013 08:55:00 +0200

> If we -- the IETF community -- had a definition of privacy, that would
Its always entertaining to go trolling back to first principles on one of
these IETF lists but we've done that before and its typically about as
useful as random walks through technology.

I hope and suspect some folks are busy working on problem statement
drafts already...

Personally I'd like to talk about specific proposals to do what Randy
just said: more P and by default, whatever your definition of P is.

        Cheers Leif

---

From: Alissa Cooper <[email-masked]>
Date: Fri, 13 Sep 2013 04:06:40 -0400

Hi Dave, all,

On Sep 12, 2013, at 11:44 PM, Dave Crocker <[email-masked]> wrote:

> If we -- the IETF community -- had a definition of privacy, that would help, but we don't.

RFC 6973 contains a rather comprehensive threat model for privacy on the Internet. It demonstrates that privacy, like "security" and "efficiency" and most any other design goal, is a complex, multi-faceted concept. I realize that in the past you have questioned the usefulness of defining "privacy" via the sum of its properties, but wanted to point it out to others on this list who are looking for common terminology and threat descriptions (and mitigations) to use. We need not resort to wikipedia for definitions.

Alissa

> 
> If we knew what it meant to 'enable privacy' (nevermind whether by default), that would also help, but we don't.
> 
> So the idea of 'privacy by default' well could serve as a useful catch-phrase, but at the moment, it lacks technical substance.  We have no shared, technical understanding of its meaning.
> 
> No doubt you have particulars in mind.  No doubt lots of folks have their own set of particulars.  Unfortunately each person's particulars tends to be different.
> 
> So what we lack are a) anything like a specific technical meaning for privacy, b) any system-level sense of what it means to provide privacy, or c) any prioritization of privacy issues so that we can get the most benefit from initial efforts.
> 
> My question was meant to prompt discussion at a technical level.  There have been the tiniest wisps of comment on the list that might provide touchstones for this, but they haven't been pursued.  We need to pursue them.
> 
> _______________________________________________
> perpass mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/perpass
>

---

From: Scott Brim <[email-masked]>
Date: Fri, 13 Sep 2013 05:58:28 -0400

For me, one focus is on breach of privacy through use of identifiers in
scopes with different confidentiality.  For example, if you take a VIN (a
public identifier mapping from vehicle to owner information, but with no
location information) and use it in a global IPv6 address (a public locator
that by itself would not be personally identifying) you have crossed scope
boundaries and breached privacy.  This sort of thing is low hanging fruit
in the IETF. It's an issue that can easily be checked, in every WG. This
group can provide guidance.

Scott

---

From: Stephen Farrell <[email-masked]>
Date: Fri, 13 Sep 2013 12:43:57 +0100

Hiya,

> useful as random walks through technology.

Well, to be fair to Dave, I don't think he's trolling.

We're seeing a bunch of proposals for things to do on this
list and that's great - keep 'em coming; write I-Ds, we can
figure out how to handle 'em etc. That's all good.

But, where I think Dave is right is that we don't have an
overview, so its possible that some effort we devote here could
be wasted if we miss something important.

So, while I definitely do not want to see us go into an
analytic paralysis, I do think some higher level consideration
would be good in addition to the specific proposals we need
to get.

> I hope and suspect some folks are busy working on problem statement

Great! If folks are doing that, I'd appreciate a heads-up
offlist, just so's we have a chance to avoid some duplication.
That's not required though - work away quietly if you prefer
until you're ready.

S.

---

From: Stephen Farrell <[email-masked]>
Date: Fri, 13 Sep 2013 13:17:41 +0100

> Should we downgrade the User-Agent field (section 14.43 of RFC 2616)

I think everyone finds those values problematic, and not only for
privacy reasons. But yes, if you believe [1] then its probably the
biggest contributor to browser fingerprinting that's in an IETF
spec. (No idea if that site's evaluation is sound myself though.)

   [1] https://panopticlick.eff.org/

> "Robot/1.0" for clients which are crawling the web?)

Interesting. An IANA registry of those kinds of value might just end
up like the UA string though, which also started out nice and simple.

Maybe ask this on httpbis if you don't get more feedback here? That's
where you'd find folks who know if it could be done and who could do
it.

S.

---

From: Leif Johansson <[email-masked]>
Date: Fri, 13 Sep 2013 15:08:00 +0200

> Well, to be fair to Dave, I don't think he's trolling.
Fair enough - bad choice or words. I blame my non-native English.
> be wasted if we miss something important.
I'd rather miss something than spend days talking about things like
"what is identity" or "what is privacy".
> would be good in addition to the specific proposals we need
Agree.

---

From: Patrick Pelletier <[email-masked]>
Date: Fri, 13 Sep 2013 12:18:32 -0700

Forwarding this idea along to httpbis as you (Stephen) suggested.   
Although this could be retrofitted onto existing HTTP, not just  
httpbis, since it's merely recommending practices which are already  
legal in HTTP.

> up like the UA string though, which also started out nice and simple.

I agree that things always start out simple and get messy.  However, I  
think there are some differences:

* The original User-Agent field was not designed with privacy in  
mind.  In fact, it was designed specifically to identify the product  
and version the user is using.  So, with a different goal (privacy  
first), we will hopefully get different results.

* By specifying only a single product token, omitting comments, and  
fixing the version number at 1.0, we've already eliminated a fair  
amount of information.  And then we further limit the information by  
making the product name not the actual name of the software, but  
merely a generic indication of the type of User-Agent; whatever is the  
minimal amount of information necessary for any legitimate browser  
sniffing that needs to occur.  (Such as differentiating desktop and  
mobile clients.)

And, of course, using the simplified User-Agent strings was just one  
of my two proposals.  My other proposal, which was even simpler,  
though perhaps more radical, was to downgrade the requirement on User- 
Agent from SHOULD to MAY, and encourage browsers not to send User- 
Agent at all.  (We could even change it to a SHOULD NOT if we feel  
really heavy-handed.)  One could argue that by using other techniques  
such as responsive layout, no browser sniffing should be necessary at  
all.

--Patrick

---

From: "Roy T. Fielding" <[email-masked]>
Date: Fri, 13 Sep 2013 13:26:33 -0700

One could argue a lot of things, but disabling the often-used
and very useful User-Agent string (used by almost every site
to avoid known bugs and incompatible implementation by both
past and current browsers) is not going to happen.  Among other
things, it plays a role in CSRF security filters.

Hiding the User-Agent field-value would not have any impact
on browser fingerprinting.  Common browsers do not send unique
User-Agent strings (the TCP packeting is far more unique per
user than the UA string); non-browser tools do, but they aren't
a concern.  Using an extremely uncommon browser will, of course,
lead to more identifiable traces, but the same is true because of
how it uniquely composes requests, how it buffers sends, and
how it reacts to embedded subrequests.

In any case, the primary source for fingerprinting information
in browsers is the DOM interfaces, and I've seen very little to
suggest that browser developers are willing to remove them.

....Roy

On Sep 13, 2013, at 12:18 PM, Patrick Pelletier wrote:

> Forwarding this idea along to httpbis as you (Stephen) suggested.  Although this could be retrofitted onto existing HTTP, not just httpbis, since it's merely recommending practices which are already legal in HTTP.
> 
> I agree that things always start out simple and get messy.  However, I think there are some differences:
> 
> * The original User-Agent field was not designed with privacy in mind.  In fact, it was designed specifically to identify the product and version the user is using.  So, with a different goal (privacy first), we will hopefully get different results.
> 
> * By specifying only a single product token, omitting comments, and fixing the version number at 1.0, we've already eliminated a fair amount of information.  And then we further limit the information by making the product name not the actual name of the software, but merely a generic indication of the type of User-Agent; whatever is the minimal amount of information necessary for any legitimate browser sniffing that needs to occur.  (Such as differentiating desktop and mobile clients.)
> 
> And, of course, using the simplified User-Agent strings was just one of my two proposals.  My other proposal, which was even simpler, though perhaps more radical, was to downgrade the requirement on User-Agent from SHOULD to MAY, and encourage browsers not to send User-Agent at all.  (We could even change it to a SHOULD NOT if we feel really heavy-handed.)  One could argue that by using other techniques such as responsive layout, no browser sniffing should be necessary at all.
>

---

From: David Morris <[email-masked]>
Date: Fri, 13 Sep 2013 13:28:57 -0700 (PDT)

I think you'll have a very hard time convincing web developers that they
can solve the browser bug issues with this approach. The User-Agent string
is an awful hack, but it has clasically allowed creativity on web sites
which wouldn't be possible if the web sites were forced to conform to
the LCD. In addition, I've noticed that some plug-ins add their version
information to the UA String so that down stream processes will know
about the specific features.

Web developers need pretty specific knowledge about browsers and their
enviornment. I would be hard to convince that any alternative mechanism
which provided the platform target profile that web apps need wouldn't 
expose the user to the same or worse issues re. fingerprinting.

On Fri, 13 Sep 2013, Patrick Pelletier wrote:

> Forwarding this idea along to httpbis as you (Stephen) suggested.  Although
> this could be retrofitted onto existing HTTP, not just httpbis, since it's
> merely recommending practices which are already legal in HTTP.
> 
> * The original User-Agent field was not designed with privacy in mind.  In
> fact, it was designed specifically to identify the product and version the
> user is using.  So, with a different goal (privacy first), we will hopefully
> get different results.
> 
> * By specifying only a single product token, omitting comments, and fixing the
> version number at 1.0, we've already eliminated a fair amount of information.
> And then we further limit the information by making the product name not the
> actual name of the software, but merely a generic indication of the type of
> User-Agent; whatever is the minimal amount of information necessary for any
> legitimate browser sniffing that needs to occur.  (Such as differentiating
> desktop and mobile clients.)
> 
> And, of course, using the simplified User-Agent strings was just one of my two
> proposals.  My other proposal, which was even simpler, though perhaps more
> radical, was to downgrade the requirement on User-Agent from SHOULD to MAY,
> and encourage browsers not to send User-Agent at all.  (We could even change
> it to a SHOULD NOT if we feel really heavy-handed.)  One could argue that by
> using other techniques such as responsive layout, no browser sniffing should
> be necessary at all.
>

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Fri, 13 Sep 2013 20:51:30 +0000

In message <[email-masked]>, "Roy T. Fielding" w
rites:

>and very useful User-Agent string [...]

How about making it intelligently usable instead ?

Right now everybody wastes bandwidth claiming to be "Mozilla/5.0"
with "Mozilla/4.0" being a distant second:

	root@phk:/usr/local/www/logs # grep -c Mozilla/4.0 thttpd.log
	44445
	root@phk:/usr/local/www/logs # grep -c Mozilla/5.0 thttpd.log
	369977
	root@phk:/usr/local/www/logs # wc -l thttpd.log
	  520850 thttpd.log

with the result that those 12 bytes (incl the next SP) is just
a total waste of bandwidth.

HTTP/2.0 would be a great chance to stop this race to the bottom
where everybody sticks everything they can think of into User-Agent
in the hope that the dudes in the other end are incompetent enough
to actually cater for broken browsers.

If nothing else, putting a hard 32 byte limit on the string would
be a BIG improvement, since that would force people to transmit
only the necessary and useful information.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
[email-masked]         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.

---

From: Martin Thomson <[email-masked]>
Date: Fri, 13 Sep 2013 13:51:44 -0700

> suggest that browser developers are willing to remove them.

Speaking as someone actively involved in expanding the browser
fingerprinting surface, the DOM is definitely where the worst
fingerprinting problems exist.

That's not to say that the right response is to throw our hands up and
say "the Chinese are doing nothing about climate change, why should we
suffer hardship?"  Putting the fact that the premise is completely
wrong aside for a moment, there are things that can be done.  On the
down side, we're not really the right group to do it.  On the up side,
the browsers - who are the right people - already are doing something.

There aren't that many browsers in wide use.  Most of those
automatically update.  The number of fingerprinting bits available
from User-Agent if you use one of these browsers is actually very low.
 The value derived from those bits is simultaneously diminishing as
more capability detection moves to the DOM.

It may be that at some future point, the value of User-Agent
diminishes to the point that browsers will cease sending it (or they
will all send the same thing).  At which point it contains zero bits
of fingerprint entropy.  For the moment, it's still very useful in
some contexts, particularly mobile, so I suspect that it would very
hard to go cold turkey.

---

From: Karl Dubost <[email-masked]>
Date: Fri, 13 Sep 2013 16:49:49 -0400

Patrick Pelletier [2013-09-13T15:18]:
> encourage browsers not to send User-Agent at all.

It will not happen. This simple decision would break most of the Web. (Unfortunately). My daily work is to fight against bad user agent detection. It's so deep rooted in the Web infrastructure that even simplifying it takes a lot of energy.

Today I was going through the stats of a very high traffic web sites that shared with me the list of all unique UA strings they collected on 1 week. 

cat access-ua-log-7days.txt | wc -l
386 844

Yes… each of these are different. Some of them are just the usual one, such as 
"Mozilla/5.0 (Windows NT 6.1; WOW64; rv:23.0) Gecko/20100101 Firefox/23.0"
with million of occurences

From looking at the file I have, there are things which can be improved though, some nasty things are done with some of the UA strings. With two areas worse than others:

* IE products such as 3rd party toolbars modifying the UA with a unique ID per user.
* some Mobile products with unique ID (most of the time put by Operators)

The spec could forbid it, but I guess it would be more a question of legal matter. Operators are sometimes using these unique ID for services.

btw it's not only User-Agent, there are secondary things such as 

X-Original-User-Agent
X-Device-User-Agent
Device-Stock-UA
X-OperaMini-Phone-UA

-- 
Karl Dubost
http://www.la-grange.net/karl/

---

From: William Chan (陈智昌) <[email-masked]>
Date: Fri, 13 Sep 2013 14:22:22 -0700

Speaking as a browser vendor, we do try to limit the info we send in the UA
string. It's sadly difficult. Please refer to
http://webaim.org/blog/user-agent-string-history/. Changing the UA string
breaks web compat, and when the end user's favorite site foo.com fails to
render in browser X, the end user switches to browser Y.

For a fairly recent discussion of this on the Chromium developer's mailing
list, feel free to refer to
https://groups.google.com/a/chromium.org/forum/#!msg/chromium-dev/N78vBMOZlF8/tQ-xnJV1lVsJ
.

On Fri, Sep 13, 2013 at 12:18 PM, Patrick Pelletier <
[email-masked]> wrote:

>  Although this could be retrofitted onto existing HTTP, not just httpbis,
> since it's merely recommending practices which are already legal in HTTP.
>
> user is using.  So, with a different goal (privacy first), we will
> hopefully get different results.
>
> * By specifying only a single product token, omitting comments, and fixing
> the version number at 1.0, we've already eliminated a fair amount of
> information.  And then we further limit the information by making the
> product name not the actual name of the software, but merely a generic
> indication of the type of User-Agent; whatever is the minimal amount of
> information necessary for any legitimate browser sniffing that needs to
> occur.  (Such as differentiating desktop and mobile clients.)
>
> And, of course, using the simplified User-Agent strings was just one of my
> two proposals.  My other proposal, which was even simpler, though perhaps
> more radical, was to downgrade the requirement on User-Agent from SHOULD to
> MAY, and encourage browsers not to send User-Agent at all.  (We could even
> change it to a SHOULD NOT if we feel really heavy-handed.)  One could argue
> that by using other techniques such as responsive layout, no browser
> sniffing should be necessary at all.
>

---

From: Randy Bush <[email-masked]>
Date: Sat, 14 Sep 2013 15:59:44 -1000

< a fair bit off topic >

forgetting ww2, isn't this the wrong way around?  the browser should
speak of which of a well-known feature set it supports, not what it's
'name' happens to be.

randy

---

From: Yoav Nir <[email-masked]>
Date: Sun, 15 Sep 2013 05:17:57 +0000

It's not just a set of features. Sure, it all started with Netscape supporting frames whereas Mosaic did not. So it would be nice if it just advertised "Supports: frames". But then IE also supported frames, but if the graphics designer made a web page so that it was nice and pretty and aligned perfectly in Netscape, it looked all screwy in IE, because of differences in spacing within and between frames. So it helped to be able to serve different pages to different browsers. And then came active content, with Java applets and ActiveX and embedded video and javascript/vbscript, and things got worse. It's been getting better in the last few years, but not that much better than you can make a single version of your website.

---

From: George Michaelson <[email-masked]>
Date: Sun, 15 Sep 2013 16:56:51 +1000

please don't laugh if I mention TELNET options negotiation or SMTP
capabilites exchange..

(I find browser strings fascinating. the sheer volume of 'I may behave like
<x>' statements adds up to a mighty mishmash of things none of which
directly relate to what I think web code developers wanted, although 'works
in IE5' may indeed be "it")

-G

On Sun, Sep 15, 2013 at 3:17 PM, Yoav Nir <[email-masked]> wrote:

> It's not just a set of features. Sure, it all started with Netscape
> supporting frames whereas Mosaic did not. So it would be nice if it just
> advertised "Supports: frames". But then IE also supported frames, but if
> the graphics designer made a web page so that it was nice and pretty and
> aligned perfectly in Netscape, it looked all screwy in IE, because of
> differences in spacing within and between frames. So it helped to be able
> to serve different pages to different browsers. And then came active
> content, with Java applets and ActiveX and embedded video and
> javascript/vbscript, and things got worse. It's been getting better in the
> last few years, but not that much better than you can make a single version
> of your website.
>

---

From: Yoav Nir <[email-masked]>
Date: Sun, 15 Sep 2013 07:16:41 +0000

If the foo browser says "works like IE5" but then your page looks bad there, users of the foo browser are going to complain both to you. So "works like IE5" is an OK first step, but as a web developer you really want to know what browser this really is, because you can't trust browser vendors to really exactly be like IE5.

I guess that's why Adobe Flash had so much success. It really provided a very consistent experience across platforms provided everyone had the plug-in.

From: George Michaelson [mailto:[email-masked]]
Sent: Sunday, September 15, 2013 9:57 AM
To: Yoav Nir
Cc: Randy Bush; perpass; William Chan
Subject: Re: [perpass] HTTP user-agent fingerprinting

please don't laugh if I mention TELNET options negotiation or SMTP capabilites exchange..

(I find browser strings fascinating. the sheer volume of 'I may behave like <x>' statements adds up to a mighty mishmash of things none of which directly relate to what I think web code developers wanted, although 'works in IE5' may indeed be "it")

-G

On Sun, Sep 15, 2013 at 3:17 PM, Yoav Nir <[email-masked]<mailto:[email-masked]>> wrote:
It's not just a set of features. Sure, it all started with Netscape supporting frames whereas Mosaic did not. So it would be nice if it just advertised "Supports: frames". But then IE also supported frames, but if the graphics designer made a web page so that it was nice and pretty and aligned perfectly in Netscape, it looked all screwy in IE, because of differences in spacing within and between frames. So it helped to be able to serve different pages to different browsers. And then came active content, with Java applets and ActiveX and embedded video and javascript/vbscript, and things got worse. It's been getting better in the last few years, but not that much better than you can make a single version of your website.

_______________________________________________
perpass mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/perpass

Email secured by Check Point

---

From: Randy Bush <[email-masked]>
Date: Sat, 14 Sep 2013 23:39:51 -1000

>> forgetting ww2, isn't this the wrong way around?

again, forget the past, even though you are deeply entangled in the
results of a design error, or lack of design.  my point was for next
time we design something, which is why i warned this is somewhat off
topic.

we have analogous lessons in many places.  when receiving an ebgp
announcement, color it indicating from what kind of peer it was
received, not what exit it should take, let the exit decide.  there
are also analogs in programming.  i could go on, but i'll be kind.

randy

---

From: Karl Dubost <[email-masked]>
Date: Mon, 16 Sep 2013 14:42:07 -0400

Nicolas Mailhot [2013-09-16T12:34]:
> User-Agent is invaluable for filtering out pathologic web clients in a network without bothering legitimate users.

* At which level do you need to filter out? (proxy, server, …)
* Which type of clients? (bot, browser, etc?)
* Why do you need to filter out?
* What are you looking for in the UA string?

/me is interested to know. I'm kind of collecting how the UA string is used out in the wild in good and bad ways.

---

From: "Nicolas Mailhot" <[email-masked]>
Date: Mon, 16 Sep 2013 18:34:57 +0200

Le Ven 13 septembre 2013 21:18, Patrick Pelletier a écrit :

> really heavy-handed.)

Please don't. If any change it from SHOULD to MUST

User-Agent is invaluable for filtering out pathologic web clients in a
network without bothering legitimate users. And in fact 9 web clients out
of ten that do not declare user-agent are broken one way or another
protocol-wise (and they're a PITA because you can't filter them out
without breaking the few correctly implemented clients that take advantage
of the SHOULD)

In fact no-user-agent should be "I swear on the penalty of [insert cruel
and deserved punishment] to never take liberties with the HTTP protocol
and never add a bug to my software".

Right now no-user-agent-is "I don't want to understand HTTP but it gets me
through the firewall, simulate a real web client and only implement the
parts I need most of the time with no error handling except for retrying
in a loop with no sleeps"

-- 
Nicolas Mailhot

---

From: "Nicolas Mailhot" <[email-masked]>
Date: Mon, 16 Sep 2013 21:59:59 +0200

Le Lun 16 septembre 2013 20:42, Karl Dubost a écrit :
>> network without bothering legitimate users.
>

proxy

> * Which type of clients? (bot, browser, etc?)

pretty much everything that tries to access the web in http(s)

> * Why do you need to filter out?

because the internet got so fast web client authors do pretty idiotic
things at the http level and it's lost in the youtube noise. OTOH those
idiotic things tend to clutter my logs with errors that make investigating
actual problems difficult, so it's easier to just blacklist miscreants

Major browsers almost never cause problems, but pretty much anything else
that does http is likely to make shortcuts

> * What are you looking for in the UA string?

Anything likely to identify the problem client and nothing else. For example:
1. corp policy is that windows stations must go through corp wsus →
blacklist windows update UA ;
2. something is hammering the proxy with corrupted connects (control chars
before CONNECT) → it says it is "Google Mail" → go away Google Mail,
3. someone deployed a widget that wants to go to the internet every 5
minutes to display a clock no one will look at → kill clock (not ntp, we
have perfectly fine ntp servers)

Actually, I don't care about the volume as long as it's not pathological
real-time updates, what I do care is the flood of errors that makes our
job more difficult than it should be.

>
> /me is interested to know. I'm kind of collecting how the UA string is
> used out in the wild in good and bad ways.
>

-- 
Nicolas Mailhot

---

From: Scott Brim <[email-masked]>
Date: Tue, 17 Sep 2013 08:07:57 -0400

> in a loop with no sleeps"

Granted user-agent is a pretty good ad hoc filter but I hope you're working
on dealing with this in the long term by some architecturally cleaner way.
:-)

Scott