Subject: Re: [lmap] LMAP: situation: Q5-Gaming the system

From: "STARK, BARBARA H" <[email-masked]>
Date: Thu, 11 Apr 2013 21:50:41 +0000

> Question 5: Gaming the system
> Do we agree that, from a technical point of view, there is nothing we can do to prevent gaming the system? So we're left with the assumption that people will behave.

I do not believe there is nothing we can do. I just think there are infinite ways to game the system and none of them are worth spending my time on. I believe that the costs of defining and deploying technical solutions to prevent gaming are greater than the risks/costs of doing nothing. In my priority list of things to do and ways to spend my time, I just don't see a lot of value in spending time on that topic.

I would characterize gaming as intentionally (and perhaps maliciously) inserting inaccuracy into the overall system or process. There are many, many ways to accomplish this in the context of lmap: an ISP could apply QoS to test traffic; an ISP could manipulate collected data prior to providing it to others; a 3rd party could manipulate collected data prior to providing it to others (to try to make an ISP look bad/good); a 3rd party could program their MA to manipulate data before sending it to the data collector; an ISP could provide inaccurate service attributes to use for comparison; a 3rd party could program all MAs to run measurements at the same time against an ISP measurement endpoint in an attempt to degrade performance of all tests against that endpoint; the end user hacks the MA to make it report bad performance data (because they hate their  ISP); etc. 

But to decide to spend effort guarding against any particular way to game the system, I believe that the incentive for doing that particular method of gaming must outweigh the probability of getting caught together with the costs of getting caught. For me, I don't perceive any of the ways of gaming as providing a sufficiently high incentive. At least not at this time. A large part of why I believe this is that (1) I'm not aware of any evidence of gaming occurring, and (2) there doesn't seem to have been massive negative/positive repercussions (lots of ISP switching, massive consumer outcry upon viewing results, fines for perceived poor performance, etc.) from broadband measurements conducted to date. If there were greater incentive to some party (e.g., "we'll offer free service for 10 years to the person who has current service from our competitor and will apply this hack to the MA in their home, to make our competitor look bad"), then I think it would be time to consider addressing that way of gaming.

Of course, it's also possible to inadvertently (not intentionally and not maliciously) insert inaccuracy. There has been evidence of that. For example, some users took their 3rd party MA over to a friend or relative's house to let it run measurements, not realizing that the MA had been associated (service attributes) with the broadband service of the original person. If the original person had a 20Mbps service, and the friend had a 6Mbps service, then the result interpretation would show that the measurement service was performing very poorly against the 20Mpbs benchmark. Because there is evidence of this problem, there are people asking for solutions to this problem (how to better associate the service attributes with the MA and the measured data). Inaccuracy has also been introduced by misinterpreting measurement data. Some of this misinterpretation can be fixed by better characterization of service attributes, and some can be dealt with by doing different tests which can be interpreted in the desired way (which is why there's some focus on figuring out the "right" tests to do). In other words, methods for inserting inaccuracies that are known to occur are higher priority to solve technically than inaccuracy insertion methods for which there is no evidence (and low incentive).
Barbara

---

From: <[email-masked]>
Date: Fri, 12 Apr 2013 14:38:11 +0100

agree with Barbara's analysis.

any party in-line can manipulate traffic  = the ISP and end user, and sometimes other parties. it seems to me that it's impossible to prevent the determined gamer doing this. one could devise tests to try and spot that some party was trying to game the system - i don't have the expertise to do that - would  be interesting to see tests /results from others, but I don't think it's a job for the WG.

best wishes
phil

________________________________________
From: [email-masked] [[email-masked]] On Behalf Of STARK, BARBARA H [[email-masked]]
Sent: 11 April 2013 22:50
To: [email-masked]
Subject: Re: [lmap] LMAP: situation: Q5-Gaming the system

But to decide to spend effort guarding against any particular way to game the system, I believe that the incentive for doing that particular method of gaming must outweigh the probability of getting caught together with the costs of getting caught. For me, I don't perceive any of the ways of gaming as providing a sufficiently high incentive. At least not at this time. A large part of why I believe this is that (1) I'm not aware of any evidence of gaming occurring, and (2) there doesn't seem to have been massive negative/positive repercussions (lots of ISP switching, massive consumer outcry upon viewing results, fines for perceived poor performance, etc.) from broadband measurements conducted to date. If there were greater incentive to some party (e.g., "we'll offer free service for 10 years to the person who has current service from our competitor and will apply this hack to the MA in their home, to make our competitor look bad"), then I think it would be time to consider add
 ressing that way of gaming.

Of course, it's also possible to inadvertently (not intentionally and not maliciously) insert inaccuracy. There has been evidence of that. For example, some users took their 3rd party MA over to a friend or relative's house to let it run measurements, not realizing that the MA had been associated (service attributes) with the broadband service of the original person. If the original person had a 20Mbps service, and the friend had a 6Mbps service, then the result interpretation would show that the measurement service was performing very poorly against the 20Mpbs benchmark. Because there is evidence of this problem, there are people asking for solutions to this problem (how to better associate the service attributes with the MA and the measured data). Inaccuracy has also been introduced by misinterpreting measurement data. Some of this misinterpretation can be fixed by better characterization of service attributes, and some can be dealt with by doing different tests which can b
 e interpreted in the desired way (which is why there's some focus on figuring out the "right" tests to do). In other words, methods for inserting inaccuracies that are known to occur are higher priority to solve technically than inaccuracy insertion methods for which there is no evidence (and low incentive).
Barbara
_______________________________________________
lmap mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/lmap

---

From: "MORTON JR., ALFRED C (AL)" <[email-masked]>
Date: Fri, 12 Apr 2013 11:06:13 -0400

I want to make the cross-thread linkage between my Q3 comment and Q5 thread explicit:

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> MORTON JR., ALFRED C (AL)
> Sent: Thursday, April 11, 2013 5:47 PM
> To: STARK, BARBARA H; [email-masked]
> Subject: Re: [lmap] LMAP: situation: Q3-Passive/Active measurements
> 
> 
> Barbara's response 
+++ to Q3 +++
> leads to another point stemming from discussion
> earlier this week: "neutrality" of measurements. Subscribers already
> trust their ISPs with the unencrypted contents of their packets, and
> an ISP would have few subscribers if it gave private information away.
> A similar code of conduct extends to measurements and published results,
> and the potential to loose subscribers for ethical transgressions is
> equally great.
> 
> Al
>

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> [email-masked]
> Sent: Friday, April 12, 2013 9:38 AM
> To: STARK, BARBARA H; [email-masked]
> any party in-line can manipulate traffic  = the ISP and end user, and
> sometimes other parties. it seems to me that it's impossible to prevent
> the determined gamer doing this. one could devise tests to try and spot
> that some party was trying to game the system - i don't have the expertise
> to do that - would  be interesting to see tests /results from others, but
> I don't think it's a job for the WG.
> 
> best wishes
> phil
> 
> > Do we agree that, from a technical point of view, there is nothing we
> can do to prevent gaming the system? So we're left with the assumption
> that people will behave.
> 
> I do not believe there is nothing we can do. I just think there are
> infinite ways to game the system and none of them are worth spending my
> time on. I believe that the costs of defining and deploying technical
> solutions to prevent gaming are greater than the risks/costs of doing
> nothing. In my priority list of things to do and ways to spend my time, I
> just don't see a lot of value in spending time on that topic.
> 
> I would characterize gaming as intentionally (and perhaps maliciously)
> inserting inaccuracy into the overall system or process. There are many,
> many ways to accomplish this in the context of lmap: an ISP could apply
> QoS to test traffic; an ISP could manipulate collected data prior to
> providing it to others; a 3rd party could manipulate collected data prior
> to providing it to others (to try to make an ISP look bad/good); a 3rd
> party could program their MA to manipulate data before sending it to the
> data collector; an ISP could provide inaccurate service attributes to use
> for comparison; a 3rd party could program all MAs to run measurements at
> the same time against an ISP measurement endpoint in an attempt to degrade
> performance of all tests against that endpoint; the end user hacks the MA
> to make it report bad performance data (because they hate their  ISP);
> etc.
> 
> But to decide to spend effort guarding against any particular way to game
> the system, I believe that the incentive for doing that particular method
> of gaming must outweigh the probability of getting caught together with
> the costs of getting caught. For me, I don't perceive any of the ways of
> gaming as providing a sufficiently high incentive. At least not at this
> time. A large part of why I believe this is that (1) I'm not aware of any
> evidence of gaming occurring, and (2) there doesn't seem to have been
> massive negative/positive repercussions (lots of ISP switching, massive
> consumer outcry upon viewing results, fines for perceived poor
> performance, etc.) from broadband measurements conducted to date. If there
> were greater incentive to some party (e.g., "we'll offer free service for
> 10 years to the person who has current service from our competitor and
> will apply this hack to the MA in their home, to make our competitor look
> bad"), then I think it would be time to consider add
> Of course, it's also possible to inadvertently (not intentionally and not
> maliciously) insert inaccuracy. There has been evidence of that. For
> example, some users took their 3rd party MA over to a friend or relative's
> house to let it run measurements, not realizing that the MA had been
> associated (service attributes) with the broadband service of the original
> person. If the original person had a 20Mbps service, and the friend had a
> 6Mbps service, then the result interpretation would show that the
> measurement service was performing very poorly against the 20Mpbs
> benchmark. Because there is evidence of this problem, there are people
> asking for solutions to this problem (how to better associate the service
> attributes with the MA and the measured data). Inaccuracy has also been
> introduced by misinterpreting measurement data. Some of this
> misinterpretation can be fixed by better characterization of service
> attributes, and some can be dealt with by doing different tests which can
> b
>  e interpreted in the desired way (which is why there's some focus on
> figuring out the "right" tests to do). In other words, methods for
> inserting inaccuracies that are known to occur are higher priority to
> solve technically than inaccuracy insertion methods for which there is no
> evidence (and low incentive).
> Barbara

---

From: Benoit Claise <[email-masked]>
Date: Fri, 19 Apr 2013 16:56:28 +0200

Dear all,

 From the answers, I understand that there are so many ways to game the 
system, that this group should not focus on technical solutions to all 
these problems. However, obvious (easy) cases might be in scope. For 
example, quoting Barbara: "If the original person had a 20Mbps service, 
and the friend had a 6Mbps service, then the result interpretation would 
show that the measurement service was performing very poorly against the 
20Mpbs benchmark."

Regards, Benoit