
From nobody Fri Jul  1 11:29:52 2016
Return-Path: <jiangsheng@huawei.com>
X-Original-To: nmlrg@ietfa.amsl.com
Delivered-To: nmlrg@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 65CF512D5A5 for <nmlrg@ietfa.amsl.com>; Fri,  1 Jul 2016 11:29:50 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -5.646
X-Spam-Level: 
X-Spam-Status: No, score=-5.646 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H4=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RP_MATCHES_RCVD=-1.426, SPF_PASS=-0.001] autolearn=ham autolearn_force=no
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id fypdj1MVfYho for <nmlrg@ietfa.amsl.com>; Fri,  1 Jul 2016 11:29:47 -0700 (PDT)
Received: from szxga02-in.huawei.com (szxga02-in.huawei.com [119.145.14.65]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id B2C8312D5A2 for <nmlrg@irtf.org>; Fri,  1 Jul 2016 11:29:43 -0700 (PDT)
Received: from 172.24.1.36 (EHLO nkgeml414-hub.china.huawei.com) ([172.24.1.36]) by szxrg02-dlp.huawei.com (MOS 4.3.7-GA FastPath queued) with ESMTP id DJQ88835; Sat, 02 Jul 2016 02:26:19 +0800 (CST)
Received: from NKGEML515-MBX.china.huawei.com ([fe80::a54a:89d2:c471:ff]) by nkgeml414-hub.china.huawei.com ([10.98.56.75]) with mapi id 14.03.0235.001; Sat, 2 Jul 2016 02:26:14 +0800
From: Sheng Jiang <jiangsheng@huawei.com>
To: David Meyer <dmm@1-4-5.net>, "nmlrg@irtf.org" <nmlrg@irtf.org>
Thread-Topic: [Nmlrg] new ways of dealing with sparse categorical data like IP addresses (Google Wide & Deep)
Thread-Index: AQHR0jbbC8ZP2gDp8UaVg4AogmAngqACUbXn
Date: Fri, 1 Jul 2016 18:26:13 +0000
Message-ID: <5D36713D8A4E7348A7E10DF7437A4B927CA8A31A@NKGEML515-MBX.china.huawei.com>
References: <CAHiKxWiqk8XAp-XAhmnp3K9vmNOW+YHJ_pjCpFdE_86NdoepYQ@mail.gmail.com>
In-Reply-To: <CAHiKxWiqk8XAp-XAhmnp3K9vmNOW+YHJ_pjCpFdE_86NdoepYQ@mail.gmail.com>
Accept-Language: en-GB, zh-CN, en-US
Content-Language: en-GB
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
x-originating-ip: [10.47.71.50]
Content-Type: multipart/alternative; boundary="_000_5D36713D8A4E7348A7E10DF7437A4B927CA8A31ANKGEML515MBXchi_"
MIME-Version: 1.0
X-CFilter-Loop: Reflected
X-Mirapoint-Virus-RAPID-Raw: score=unknown(0), refid=str=0001.0A090202.5776B5CC.000C, ss=1, re=0.000, recu=0.000, reip=0.000,  cl=1, cld=1, fgs=0, ip=0.0.0.0, so=2013-06-18 04:22:30, dmn=2013-03-21 17:37:32
X-Mirapoint-Loop-Id: a9126dc00d156842f283a600fff3395a
Archived-At: <https://mailarchive.ietf.org/arch/msg/nmlrg/CK7_NoU9s5AXRB6_xoOckzgldi0>
Subject: Re: [Nmlrg] new ways of dealing with sparse categorical data like IP addresses (Google Wide & Deep)
X-BeenThere: nmlrg@irtf.org
X-Mailman-Version: 2.1.17
Precedence: list
List-Id: Network Machine Learning Research Group <nmlrg.irtf.org>
List-Unsubscribe: <https://www.irtf.org/mailman/options/nmlrg>, <mailto:nmlrg-request@irtf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/nmlrg/>
List-Post: <mailto:nmlrg@irtf.org>
List-Help: <mailto:nmlrg-request@irtf.org?subject=help>
List-Subscribe: <https://www.irtf.org/mailman/listinfo/nmlrg>, <mailto:nmlrg-request@irtf.org?subject=subscribe>
X-List-Received-Date: Fri, 01 Jul 2016 18:29:50 -0000

--_000_5D36713D8A4E7348A7E10DF7437A4B927CA8A31ANKGEML515MBXchi_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

Hi, David,



Thanks for this information. This looks interesting. It is something worth =
of trying. However, I cannot claim I understand it pruperly. Particularly, =
there are a few explanation actually confused me a lot more, like a wide li=
near model (for memorization), a deep neural network (for generalization), =
sparse inputs (categorical feature with a large number of possible feature =
values), also your mentioned IPv{4,6} addresses as a sparse categorical fea=
ture, etc. I guess there are a lot terminology and statement in this field =
have not been properly specificed or standardized. So there are lots chance=
s to misunderstand each other.



Meanwhile, one of my read from the blog is that it is not new to combine mo=
re than one learning/analysing approaches for a same task. Multiple approac=
hes could be run simultaneously. If they could reach the same result, they =
prove each other. Then, we could have more confidence for the predict resul=
t. I am sure there will be some case, the result are different. Then what? =
I am not sure. But it would be interested to continue working on it.



Regards,



Sheng

________________________________
From: nmlrg [nmlrg-bounces@irtf.org] on behalf of David Meyer [dmm@1-4-5.ne=
t]
Sent: 30 June 2016 2:48
To: nmlrg@irtf.org
Subject: [Nmlrg] new ways of dealing with sparse categorical data like IP a=
ddresses (Google Wide & Deep)

Check out Wide & Deep: https://research.googleblog.com/2016/06/wide-deep-le=
arning-better-together-with.html

Among other things, Wide & Deep is

"useful for generic large-scale regression and classification problems with=
 sparse inputs (categorical features with a large number of possible featur=
e values), such as recommender systems, search, and ranking problems."

In the network space we have many such sparse categorical features such as =
IPv{4,6} addresses. Wide & Deep might give us a few  hints about how we mig=
ht approach this problem.

--dmm



--_000_5D36713D8A4E7348A7E10DF7437A4B927CA8A31ANKGEML515MBXchi_
Content-Type: text/html; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

<html dir=3D"ltr">
<head>
<meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Diso-8859-=
1">
<style id=3D"owaParaStyle">=0A=
<!--=0A=
p=0A=
	{margin-top:0px;=0A=
	margin-bottom:0px}=0A=
-->=0A=
</style>
</head>
<body fpstyle=3D"1" ocsi=3D"0">
<div style=3D"direction: ltr;font-family: Tahoma;color: #000000;font-size: =
10pt;">
<div style=3D"direction:ltr; font-family:Tahoma; color:#000000; font-size:1=
0pt">
<p>Hi, David,</p>
<p>&nbsp;</p>
<p>Thanks for this information. This looks interesting. It is something wor=
th of trying. However, I cannot claim I understand it pruperly. Particularl=
y, there are a few explanation actually confused me a lot more, like a wide=
 linear model (for memorization),
 a deep neural network (for generalization), sparse inputs (categorical fea=
ture with a large number of possible feature values), also your mentioned I=
Pv{4,6} addresses as a sparse categorical feature, etc. I guess there are a=
 lot terminology and statement in
 this field have not been properly specificed or standardized. So there are=
 lots chances to misunderstand each other.</p>
<p>&nbsp;</p>
<p>Meanwhile, one of my read from the blog is that it&nbsp;is&nbsp;not new =
to combine more than one learning/analysing approaches for a same task. Mul=
tiple approaches could be run simultaneously. If they could reach the same =
result, they prove each other. Then, we could
 have more confidence for the predict result. I am sure there will be some =
case, the result are different. Then what? I am not sure. But it would be i=
nterested to continue working on it.</p>
<p>&nbsp;</p>
<p>Regards,</p>
<p>&nbsp;</p>
<p>Sheng</p>
<div style=3D"font-family:Times New Roman; color:#000000; font-size:16px">
<hr tabindex=3D"-1">
<div id=3D"divRpF885664" style=3D"direction:ltr"><font color=3D"#000000" si=
ze=3D"2" face=3D"Tahoma"><b>From:</b> nmlrg [nmlrg-bounces@irtf.org] on beh=
alf of David Meyer [dmm@1-4-5.net]<br>
<b>Sent:</b> 30 June 2016 2:48<br>
<b>To:</b> nmlrg@irtf.org<br>
<b>Subject:</b> [Nmlrg] new ways of dealing with sparse categorical data li=
ke IP addresses (Google Wide &amp; Deep)<br>
</font><br>
</div>
<div></div>
<div>
<div dir=3D"ltr">Check out Wide &amp; Deep:&nbsp;<a href=3D"https://researc=
h.googleblog.com/2016/06/wide-deep-learning-better-together-with.html" targ=
et=3D"_blank">https://research.googleblog.com/2016/06/wide-deep-learning-be=
tter-together-with.html</a>
<div><br>
</div>
<div>
<div>Among other things, Wide &amp; Deep is</div>
<div><br>
</div>
<div>&quot;useful for generic large-scale regression and classification pro=
blems with sparse inputs (categorical features with a large number of possi=
ble feature values), such as recommender systems, search, and ranking probl=
ems.&quot;</div>
</div>
<div><br>
</div>
<div>In the network space we have many such sparse categorical features suc=
h as IPv{4,6} addresses. Wide &amp; Deep might give us a few &nbsp;hints ab=
out how we might approach this problem.</div>
<div><br>
</div>
<div>--dmm</div>
<div><br>
</div>
<div><br>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>

--_000_5D36713D8A4E7348A7E10DF7437A4B927CA8A31ANKGEML515MBXchi_--


From nobody Fri Jul  1 13:36:21 2016
Return-Path: <jiangsheng@huawei.com>
X-Original-To: nmlrg@ietfa.amsl.com
Delivered-To: nmlrg@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 05F2712D7DD for <nmlrg@ietfa.amsl.com>; Fri,  1 Jul 2016 13:36:21 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -5.646
X-Spam-Level: 
X-Spam-Status: No, score=-5.646 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RP_MATCHES_RCVD=-1.426, SPF_PASS=-0.001] autolearn=ham autolearn_force=no
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ak7NI_vdsGYD for <nmlrg@ietfa.amsl.com>; Fri,  1 Jul 2016 13:36:18 -0700 (PDT)
Received: from szxga03-in.huawei.com (szxga03-in.huawei.com [119.145.14.66]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id EDF0812D7E5 for <nmlrg@irtf.org>; Fri,  1 Jul 2016 13:36:16 -0700 (PDT)
Received: from 172.24.1.36 (EHLO nkgeml411-hub.china.huawei.com) ([172.24.1.36]) by szxrg03-dlp.huawei.com (MOS 4.4.3-GA FastPath queued) with ESMTP id CEF06963; Sat, 02 Jul 2016 04:36:08 +0800 (CST)
Received: from NKGEML515-MBX.china.huawei.com ([fe80::a54a:89d2:c471:ff]) by nkgeml411-hub.china.huawei.com ([10.98.56.70]) with mapi id 14.03.0235.001; Sat, 2 Jul 2016 04:36:06 +0800
From: Sheng Jiang <jiangsheng@huawei.com>
To: David Meyer <dmm@1-4-5.net>
Thread-Topic: [Nmlrg] links to each slide of presentations//RE: slides of NMLRG #3, June 27th, Athens
Thread-Index: AQHR0UXZfny1k5v9oUOv37+HaylYt5/+e2IAgAGH4UyAAAvZgIACSbUQ//+hNICAAe6t3A==
Date: Fri, 1 Jul 2016 20:36:05 +0000
Message-ID: <5D36713D8A4E7348A7E10DF7437A4B927CA8A355@NKGEML515-MBX.china.huawei.com>
References: <5D36713D8A4E7348A7E10DF7437A4B927CA893E9@NKGEML515-MBX.china.huawei.com> <CAHiKxWhfdukjRVnSKnbLwMwamiqZYoCD350BoNqO4njqkQoU1g@mail.gmail.com> <5D36713D8A4E7348A7E10DF7437A4B927CA897D4@NKGEML515-MBX.china.huawei.com> <CAHiKxWhMPjKY24zucg-UND0ARWHbjnOyYv=Tia-yk-Vq6Ta_SA@mail.gmail.com> <5D36713D8A4E7348A7E10DF7437A4B927CA89FB4@NKGEML515-MBX.china.huawei.com>, <CAHiKxWh5D0JiDLTKV3ePT4XNhk9ZhmDu66Z=5hxOpoo5pxkorA@mail.gmail.com>
In-Reply-To: <CAHiKxWh5D0JiDLTKV3ePT4XNhk9ZhmDu66Z=5hxOpoo5pxkorA@mail.gmail.com>
Accept-Language: en-GB, zh-CN, en-US
Content-Language: en-GB
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
x-originating-ip: [10.47.71.136]
Content-Type: multipart/alternative; boundary="_000_5D36713D8A4E7348A7E10DF7437A4B927CA8A355NKGEML515MBXchi_"
MIME-Version: 1.0
X-CFilter-Loop: Reflected
X-Mirapoint-Virus-RAPID-Raw: score=unknown(0), refid=str=0001.0A020202.5776D439.0003, ss=1, re=0.000, recu=0.000, reip=0.000,  cl=1, cld=1, fgs=0, ip=0.0.0.0, so=2013-05-26 15:14:31, dmn=2013-03-21 17:37:32
X-Mirapoint-Loop-Id: 8ad9c0ea0191dfae4144450b4de86aa8
Archived-At: <https://mailarchive.ietf.org/arch/msg/nmlrg/WcLkd861BlMmjQ2tU3C_lWnN0UU>
Cc: "nmlrg@irtf.org" <nmlrg@irtf.org>
Subject: Re: [Nmlrg] links to each slide of presentations//RE: slides of NMLRG #3, June 27th, Athens
X-BeenThere: nmlrg@irtf.org
X-Mailman-Version: 2.1.17
Precedence: list
List-Id: Network Machine Learning Research Group <nmlrg.irtf.org>
List-Unsubscribe: <https://www.irtf.org/mailman/options/nmlrg>, <mailto:nmlrg-request@irtf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/nmlrg/>
List-Post: <mailto:nmlrg@irtf.org>
List-Help: <mailto:nmlrg-request@irtf.org?subject=help>
List-Subscribe: <https://www.irtf.org/mailman/listinfo/nmlrg>, <mailto:nmlrg-request@irtf.org?subject=subscribe>
X-List-Received-Date: Fri, 01 Jul 2016 20:36:21 -0000

--_000_5D36713D8A4E7348A7E10DF7437A4B927CA8A355NKGEML515MBXchi_
Content-Type: text/plain; charset="Windows-1252"
Content-Transfer-Encoding: quoted-printable

Hi, David,

//Cut to short since the original text has become unreadable giving too man=
y replies.

I guess there should be a clarification what one-off solution I meant. It a=
ctually meant specific ML solution for every specific network tasks, but th=
ey should be as generalized among networks. It should be able to work on mo=
st of networks, if not all of them. For example, we should have a DNS-orien=
ted ML solution for DNS malicious attacks. I am not sure how much it could =
be generalized with router-oriented ML solution for DDoS attacks on routers=
. I am also not sure whether some part of these ML solutions could be reuse=
d from more general ML-based analysis on network traffics. But, I am intere=
sting to work on general learning on the fundamental properties of network =
as you suggested.

With the above explanation, the "one-off" ML solutions for a specific netwo=
rk task that is general among networks, are legitimate topics for our RG, I=
 believe. I could emphasis the generalization among networks in our future =
version of RG charter, but I am not sure how much we could do on generalizi=
ng among various network tasks.


>>Although having the above concerns, I  think we should: a) research on ce=
rtain dataset.

I meant we should try to create a "standardized" data sets as you suggested=
.


>>Many of machine learning algorithms are based on statistics theory, which=
 may
>>not be that much accurate.
>><Sheng>The reason that statistics-based ML algorithms can work is that th=
ey
>>chose to focus on only high probability and igrone corner cases.

>I don't believe this is correct, but perhaps I don't understand your point=
. Can
>you give a concrete example of (a). what you mean by statistics-based ML
>algorithm and (b). one that ignores corner cases and how/why?

Maybe the most close word for my "statistics-based" is probabilistic. Then,=
 let's take an example of linear regression algorithm. Based on the trainin=
g data, the algorithm could work out a curve that could be used to predict.=
 However, not all of training data are on this curve. Until we expand this =
curve with a tolerance range, we may cover majority of training data. There=
 may still some dispersed data which is chosen to be ignored. And there is =
no guarantee that the future data would perfectly much the prediction on th=
is curve. Till we give a tolerance scope, we could be confidence. However, =
it is still not 100%. The ML chooses to ignore the dispersed data, otherwis=
e it may not be able to produce a smooth curve based on pure probabilistic =
rather than understanding the semantic of them. So, when the future data is=
 impacted by the same reasons that cause such dispersed data. The ML based =
prediction would certainly fail. It causes the inaccurate, even tolerance s=
cope may not be able to cover it.


>BTW, people are going to deploy ML based systems not because they are inte=
resting
>or cool (which they are), but rather because their competitors are/will.

Agree. But this won=92t happen until we have successfully persuade a first =
wave usage.


>>Also the process of machine learning may be hard to be controlled or inte=
rvened.
>><Sheng> Here, I actually refer to DNN too. We don't know how to intervene=
d
>>DNN in its running time.

>Do you mean that once you've trained a DNN you can only use it for classif=
ication
>(or whatever) and can't further train it, or do you mean that we can't lea=
rn from
>additional data, or something else?

No. Of cause, DNN can learn from additional data. What I meant is we could =
not be sure the DNN=92s results or behaviors, even we could choose the trai=
ning data or the incremental data. This links back to the fact that we do n=
ot understand in details how DNN works. Therefore, we could not intervene o=
r control DNN to behavior the way we want 100%. This is also the trust issu=
es for DNN, and even for AI.


>><Sheng>Inaccurate or stochastic, they are not what many network people ca=
n accept.
>>In their eyes, network is definite, every parameter is definite. However,=
 actually, if
>>you could analyze every network parameter, the value may be definite, but
>>actually they all have toleration for some level of inaccurate as long as=
 their
>>design purposes could meet. It is the fundamental that we could use ML to
>>solve network task.

>I didn't understand what you mean here.

I have been questioned many times by traditional network people who think t=
heir network and parameters on their network devices are definite. Giving t=
he inaccurate of ML predict result, they are very resisting. However, our s=
tudies showed only the design purposes need to be definitely meet, not the =
value of parameters. For example, in one of our studies, we got various val=
ues for COST in ISIS protocol, like 5, 50, 100, 200, 500, 1000, etc. Howeve=
r, further studies on the semantic and design purposes showed the exact val=
ues are not meaningful. In any giving network, the COST is used to distingu=
ish interface into two or three classes. If ML could learn this design purp=
ose, then it could assign arbitrary two of three numbers in a giving networ=
k and the network will work fine.


Thanks and regards,

Sheng

--_000_5D36713D8A4E7348A7E10DF7437A4B927CA8A355NKGEML515MBXchi_
Content-Type: text/html; charset="Windows-1252"
Content-Transfer-Encoding: quoted-printable

<html dir=3D"ltr">
<head>
<meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3DWindows-1=
252">
<style type=3D"text/css" id=3D"owaParaStyle"></style>
</head>
<body fpstyle=3D"1" ocsi=3D"0">
<div style=3D"direction: ltr;font-family: Tahoma;color: #000000;font-size: =
10pt;">
<div style=3D"direction:ltr; font-family:Tahoma; color:#000000; font-size:1=
0pt">
<div style=3D"direction:ltr; font-family:Tahoma; color:#000000; font-size:1=
0pt">
<p class=3D"MsoNormal"><span lang=3D"EN-US">Hi, David,<o:p></o:p></span></p=
>
<p class=3D"MsoNormal"><span lang=3D"EN-US">&nbsp;</span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">//Cut to short since the origin=
al text has become unreadable giving too many replies.&nbsp;
<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">&nbsp;</span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">I guess there should be a clari=
fication what one-off solution I meant. It actually meant specific ML solut=
ion for every specific network tasks, but they should be as generalized amo=
ng networks. It should be able to work
 on most of networks, if not all of them. For example, we should have a DNS=
-oriented ML solution for DNS malicious attacks. I am not sure how much it =
could be generalized with router-oriented ML solution for DDoS attacks on r=
outers. I am also not sure whether
 some part of these ML solutions could be reused from more general ML-based=
 analysis on network traffics. But, I am interesting to work on general lea=
rning on the fundamental properties of network as you suggested.<o:p></o:p>=
</span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">&nbsp;</span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">With the above explanation, the=
 &quot;one-off&quot; ML solutions for a specific network task that is gener=
al among networks, are legitimate topics for our RG, I believe. I could emp=
hasis the generalization among networks in our
 future version of RG charter, but I am not sure how much we could do on ge=
neralizing among various network tasks.<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">&nbsp;</span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">&nbsp;</span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">&gt;&gt;Although having the abo=
ve concerns, I&nbsp; think we should: a) research on certain dataset.
<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">&nbsp;</span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">I meant we should try to create=
 a &quot;standardized&quot; data sets as you suggested.<o:p></o:p></span></=
p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">&nbsp;</span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">&nbsp;</span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">&gt;&gt;Many of machine learnin=
g algorithms are based on statistics theory, which may&nbsp;</span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">&gt;&gt;not be that much accura=
te.<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">&gt;&gt;&lt;Sheng&gt;The reason=
 that statistics-based ML algorithms can work is that they&nbsp;</span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">&gt;&gt;chose to focus on only =
high probability and igrone corner cases.<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">&nbsp;</span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">&gt;I don't believe this is cor=
rect, but perhaps I don't understand your point. Can
<br>
&gt;you give a concrete example of (a). what you mean by statistics-based M=
L&nbsp;</span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">&gt;algorithm and (b). one that=
 ignores corner cases and how/why?<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">&nbsp;</span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">Maybe the most close word for m=
y &quot;statistics-based&quot; is probabilistic. Then, let's take an exampl=
e of linear regression algorithm. Based on the training data, the algorithm=
 could work out a curve that could be used to
 predict. However, not all of training data are on this curve. Until we exp=
and this curve with a tolerance range, we may cover majority of training da=
ta. There may still some dispersed data which is chosen to be ignored. And =
there is no guarantee that the future
 data would perfectly much the prediction on this curve. Till we give a tol=
erance scope, we could be confidence. However, it is still not 100%. The ML=
 chooses to ignore the dispersed data, otherwise it may not be able to prod=
uce a smooth curve based on pure
 probabilistic rather than understanding the semantic of them. So, when the=
 future data is impacted by the same reasons that cause such dispersed data=
. The ML based prediction would certainly fail. It causes the inaccurate, e=
ven tolerance scope may not be able
 to cover it.<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">&nbsp;</span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">&nbsp;</span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">&gt;BTW, people are going to de=
ploy ML based systems not because they are interesting</span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">&gt;or cool (which they are), b=
ut rather because their competitors are/will.<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">&nbsp;</span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">Agree. But this won=92t happen =
until we have successfully persuade a first wave usage.</span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">&nbsp;</span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">&nbsp;</span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">&gt;&gt;Also the process of mac=
hine learning may be hard to be controlled or intervened.<o:p></o:p></span>=
</p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">&gt;&gt;&lt;Sheng&gt; Here, I a=
ctually refer to DNN too. We don't know how to intervened&nbsp;</span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">&gt;&gt;DNN in its running time=
.<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">&nbsp;</span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">&gt;Do you mean that once you'v=
e trained a DNN you can only use it for classification&nbsp;</span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">&gt;(or whatever) and can't fur=
ther train it, or do you mean that we can't learn from&nbsp;</span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">&gt;additional data, or somethi=
ng else?</span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">&nbsp;</span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">No. Of cause, DNN can learn fro=
m additional data. What I meant is we could not be sure the DNN=92s results=
 or behaviors, even we could choose the training data or the incremental da=
ta. This links back to the fact that we
 do not understand in details how DNN works. Therefore, we could not interv=
ene or control DNN to behavior the way we want 100%. This is also the trust=
 issues for DNN, and even for AI.<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">&nbsp;</span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">&nbsp;</span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">&gt;&gt;&lt;Sheng&gt;Inaccurate=
 or stochastic, they are not what many network people can accept.</span></p=
>
<p class=3D"MsoNormal"><span lang=3D"EN-US">&gt;&gt;In their eyes, network =
is definite, every parameter is definite. However, actually, if
</span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">&gt;&gt;you could analyze every=
 network parameter, the value may be definite, but
</span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">&gt;&gt;actually they all have =
toleration for some level of inaccurate as long as their
</span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">&gt;&gt;design purposes could m=
eet. It is the fundamental that we could use ML to
</span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">&gt;&gt;solve network task.<o:p=
></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">&nbsp;</span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">&gt;I didn't understand what yo=
u mean here.</span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">&nbsp;</span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">I have been questioned many tim=
es by traditional network people who think their network and parameters on =
their network devices are definite. Giving the inaccurate of ML predict res=
ult, they are very resisting. However,
 our studies showed only the design purposes need to be definitely meet, no=
t the value of parameters. For example, in one of our studies, we got vario=
us values for COST in ISIS protocol, like 5, 50, 100, 200, 500, 1000, etc. =
However, further studies on the
 semantic and design purposes showed the exact values are not meaningful. I=
n any giving network, the COST is used to distinguish interface into two or=
 three classes. If ML could learn this design purpose, then it could assign=
 arbitrary two of three numbers
 in a giving network and the network will work fine.</span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">&nbsp;</span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">&nbsp;</span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">Thanks and regards,</span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">&nbsp;</span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">Sheng<o:p></o:p></span></p>
</div>
</div>
</div>
</body>
</html>

--_000_5D36713D8A4E7348A7E10DF7437A4B927CA8A355NKGEML515MBXchi_--


From nobody Mon Jul  4 20:59:25 2016
Return-Path: <jiangsheng@huawei.com>
X-Original-To: nmlrg@ietfa.amsl.com
Delivered-To: nmlrg@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id EA7EE12D0B5 for <nmlrg@ietfa.amsl.com>; Mon,  4 Jul 2016 20:59:24 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -5.646
X-Spam-Level: 
X-Spam-Status: No, score=-5.646 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RP_MATCHES_RCVD=-1.426, SPF_PASS=-0.001] autolearn=ham autolearn_force=no
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id rYKXv-9cdoYM for <nmlrg@ietfa.amsl.com>; Mon,  4 Jul 2016 20:59:22 -0700 (PDT)
Received: from szxga03-in.huawei.com (szxga03-in.huawei.com [119.145.14.66]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id C81E312B02B for <nmlrg@irtf.org>; Mon,  4 Jul 2016 20:59:16 -0700 (PDT)
Received: from 172.24.1.137 (EHLO nkgeml411-hub.china.huawei.com) ([172.24.1.137]) by szxrg03-dlp.huawei.com (MOS 4.4.3-GA FastPath queued) with ESMTP id CEI10077; Tue, 05 Jul 2016 11:59:01 +0800 (CST)
Received: from NKGEML515-MBX.china.huawei.com ([fe80::a54a:89d2:c471:ff]) by nkgeml411-hub.china.huawei.com ([10.98.56.70]) with mapi id 14.03.0235.001; Tue, 5 Jul 2016 11:58:55 +0800
From: Sheng Jiang <jiangsheng@huawei.com>
To: "nmlrg@irtf.org" <nmlrg@irtf.org>
Thread-Topic: Call for agenda NMLRG #4 @ IETF 96, Berlin, Germany
Thread-Index: AdHWcYFWz+goLKpFS6KCCdhZcpV+TQ==
Date: Tue, 5 Jul 2016 03:58:55 +0000
Message-ID: <5D36713D8A4E7348A7E10DF7437A4B927CA8C67A@NKGEML515-MBX.china.huawei.com>
Accept-Language: en-GB, zh-CN, en-US
Content-Language: zh-CN
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
x-originating-ip: [10.111.99.197]
Content-Type: multipart/alternative; boundary="_000_5D36713D8A4E7348A7E10DF7437A4B927CA8C67ANKGEML515MBXchi_"
MIME-Version: 1.0
X-CFilter-Loop: Reflected
X-Mirapoint-Virus-RAPID-Raw: score=unknown(0), refid=str=0001.0A020202.577B3085.0081, ss=1, re=0.000, recu=0.000, reip=0.000,  cl=1, cld=1, fgs=0, ip=0.0.0.0, so=2013-05-26 15:14:31, dmn=2013-03-21 17:37:32
X-Mirapoint-Loop-Id: d6e7ab4972680aa35de077475ae7b508
Archived-At: <https://mailarchive.ietf.org/arch/msg/nmlrg/_HcpS5_X2XyYBo9Nfcw025DfIaQ>
Cc: Albert Cabellos <albert.cabellos@gmail.com>
Subject: [Nmlrg] Call for agenda NMLRG #4 @ IETF 96, Berlin, Germany
X-BeenThere: nmlrg@irtf.org
X-Mailman-Version: 2.1.17
Precedence: list
List-Id: Network Machine Learning Research Group <nmlrg.irtf.org>
List-Unsubscribe: <https://www.irtf.org/mailman/options/nmlrg>, <mailto:nmlrg-request@irtf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/nmlrg/>
List-Post: <mailto:nmlrg@irtf.org>
List-Help: <mailto:nmlrg-request@irtf.org?subject=help>
List-Subscribe: <https://www.irtf.org/mailman/listinfo/nmlrg>, <mailto:nmlrg-request@irtf.org?subject=subscribe>
X-List-Received-Date: Tue, 05 Jul 2016 03:59:25 -0000

--_000_5D36713D8A4E7348A7E10DF7437A4B927CA8C67ANKGEML515MBXchi_
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: base64

SGksIGFsbCBOTUxSRywNCg0KV2UgaGF2ZSByZXF1ZXN0ZWQgYSBzZXNzaW9uIG9mIDIgaG91ciBm
b3IgdGhlIG91ciBwcm9wb3NlZCBOTUxSRyBtZWV0aW5nIGZvciBJRVRGLTk2IChCZXJsaW4sIEdl
cm1hbnkpIGFuZCBhcmUgc3RhcnRpbmcgdG8gY29sbGVjdCBhZ2VuZGEgaXRlbXMgZm9yIHRoaXMg
c2Vzc2lvbnMuDQoNClBsZWFzZSBzZW5kIHVzIChhbmltYS1jaGFpcnMgYXQgdG9vbHMuaWV0Zi5v
cmcpIHJlcXVlc3RzIGZvciBzZXNzaW9ucyBieSBKdWx5ICAxMHRoLCBTdW5kYXksIGFuZCBpbmNs
dWRlOg0KDQpOYW1lIG9mIHRpbWUgc2xvdDoNCk5hbWUgb2YgZHJhZnQocyk6DQpUaW1lIHJlcXVl
c3RlZDoNClByZXNlbnRlciBuYW1lKHMpOg0KQnJpZWYgZGVzY3JpcHRpb24gb2Ygd2hhdCBpc3N1
ZXMgbmVlZCBkaXNjdXNzaW5nIGFuZCB3aGF0IHlvdSBob3BlIHRvDQphY2NvbXBsaXNoIGJ5IHBy
ZXNlbnRpbmcgKHBsZWFzZSBmb2N1cyBvbiBvcGVuIGlzc3VlcywgcmF0aGVyIHRoYW4gYQ0Kc3Rh
dHVzIHVwZGF0ZSk6DQoNCk1vcmUgZGV0YWlscyBhYm91dCB0aGUgSUVURiA5NiwgQmVybGluLCBH
ZXJtYW55IGNhbiBiZSBmb3VuZCBhdDoNCmh0dHA6Ly93d3cuaWV0Zi5vcmcvbWVldGluZy85Ni9p
bmRleC5odG1sDQoNCkFnYWluLCBwcmVzZW50ZXJzIGFuZCBkcmFmdCBhdXRob3JzIHBsZWFzZSBp
bnZva2UgYWN0aXZlIGRpc2N1c3Npb25zIGluDQp0aGUgTk1MUkcgbGlzdC4gV2UgaGF2ZSB2ZXJ5
IGxpbWl0ZWQgdGltZSBmb3IgZWFjaCB0b3BpYyBpbiB0aGUNCmZhY2UtdG8tZmFjZSBtZWV0aW5n
LiBNYWlsIGxpc3QgaXMgYSB2ZXJ5IGdvb2QgcGxhY2UgdG8gZGlzY3VzcyBhbmQgcmVhY2gNCmNv
bnNlbnN1cyBvbiB0ZWNobmljYWwgaXNzdWVzLg0KDQpCZXN0IHJlZ2FyZHMsDQoNCkFsYmVydCAm
IFNoZW5nDQo=

--_000_5D36713D8A4E7348A7E10DF7437A4B927CA8C67ANKGEML515MBXchi_
Content-Type: text/html; charset="utf-8"
Content-Transfer-Encoding: base64

PGh0bWwgeG1sbnM6dj0idXJuOnNjaGVtYXMtbWljcm9zb2Z0LWNvbTp2bWwiIHhtbG5zOm89InVy
bjpzY2hlbWFzLW1pY3Jvc29mdC1jb206b2ZmaWNlOm9mZmljZSIgeG1sbnM6dz0idXJuOnNjaGVt
YXMtbWljcm9zb2Z0LWNvbTpvZmZpY2U6d29yZCIgeG1sbnM6bT0iaHR0cDovL3NjaGVtYXMubWlj
cm9zb2Z0LmNvbS9vZmZpY2UvMjAwNC8xMi9vbW1sIiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcv
VFIvUkVDLWh0bWw0MCI+DQo8aGVhZD4NCjxtZXRhIGh0dHAtZXF1aXY9IkNvbnRlbnQtVHlwZSIg
Y29udGVudD0idGV4dC9odG1sOyBjaGFyc2V0PXV0Zi04Ij4NCjxtZXRhIG5hbWU9IkdlbmVyYXRv
ciIgY29udGVudD0iTWljcm9zb2Z0IFdvcmQgMTIgKGZpbHRlcmVkIG1lZGl1bSkiPg0KPHN0eWxl
PjwhLS0NCi8qIEZvbnQgRGVmaW5pdGlvbnMgKi8NCkBmb250LWZhY2UNCgl7Zm9udC1mYW1pbHk6
Q291cmllcjsNCglwYW5vc2UtMToyIDcgNCA5IDIgMiA1IDIgNCA0O30NCkBmb250LWZhY2UNCgl7
Zm9udC1mYW1pbHk65a6L5L2TOw0KCXBhbm9zZS0xOjIgMSA2IDAgMyAxIDEgMSAxIDE7fQ0KQGZv
bnQtZmFjZQ0KCXtmb250LWZhbWlseToiQ2FtYnJpYSBNYXRoIjsNCglwYW5vc2UtMToyIDQgNSAz
IDUgNCA2IDMgMiA0O30NCkBmb250LWZhY2UNCgl7Zm9udC1mYW1pbHk6Q2FsaWJyaTsNCglwYW5v
c2UtMToyIDE1IDUgMiAyIDIgNCAzIDIgNDt9DQpAZm9udC1mYWNlDQoJe2ZvbnQtZmFtaWx5OiJc
QOWui+S9kyI7DQoJcGFub3NlLTE6MiAxIDYgMCAzIDEgMSAxIDEgMTt9DQovKiBTdHlsZSBEZWZp
bml0aW9ucyAqLw0KcC5Nc29Ob3JtYWwsIGxpLk1zb05vcm1hbCwgZGl2Lk1zb05vcm1hbA0KCXtt
YXJnaW46MGNtOw0KCW1hcmdpbi1ib3R0b206LjAwMDFwdDsNCgl0ZXh0LWFsaWduOmp1c3RpZnk7
DQoJdGV4dC1qdXN0aWZ5OmludGVyLWlkZW9ncmFwaDsNCglmb250LXNpemU6MTAuNXB0Ow0KCWZv
bnQtZmFtaWx5OiJDYWxpYnJpIiwic2Fucy1zZXJpZiI7fQ0KYTpsaW5rLCBzcGFuLk1zb0h5cGVy
bGluaw0KCXttc28tc3R5bGUtcHJpb3JpdHk6OTk7DQoJY29sb3I6Ymx1ZTsNCgl0ZXh0LWRlY29y
YXRpb246dW5kZXJsaW5lO30NCmE6dmlzaXRlZCwgc3Bhbi5Nc29IeXBlcmxpbmtGb2xsb3dlZA0K
CXttc28tc3R5bGUtcHJpb3JpdHk6OTk7DQoJY29sb3I6cHVycGxlOw0KCXRleHQtZGVjb3JhdGlv
bjp1bmRlcmxpbmU7fQ0KcHJlDQoJe21zby1zdHlsZS1wcmlvcml0eTo5OTsNCgltc28tc3R5bGUt
bGluazoiSFRNTCDpooTorr7moLzlvI8gQ2hhciI7DQoJbWFyZ2luOjBjbTsNCgltYXJnaW4tYm90
dG9tOi4wMDAxcHQ7DQoJZm9udC1zaXplOjEyLjBwdDsNCglmb250LWZhbWlseTrlrovkvZM7fQ0K
c3Bhbi5FbWFpbFN0eWxlMTcNCgl7bXNvLXN0eWxlLXR5cGU6cGVyc29uYWwtY29tcG9zZTsNCglm
b250LWZhbWlseToiQ2FsaWJyaSIsInNhbnMtc2VyaWYiOw0KCWNvbG9yOndpbmRvd3RleHQ7fQ0K
c3Bhbi5IVE1MQ2hhcg0KCXttc28tc3R5bGUtbmFtZToiSFRNTCDpooTorr7moLzlvI8gQ2hhciI7
DQoJbXNvLXN0eWxlLXByaW9yaXR5Ojk5Ow0KCW1zby1zdHlsZS1saW5rOiJIVE1MIOmihOiuvuag
vOW8jyI7DQoJZm9udC1mYW1pbHk65a6L5L2TO30NCi5Nc29DaHBEZWZhdWx0DQoJe21zby1zdHls
ZS10eXBlOmV4cG9ydC1vbmx5O30NCi8qIFBhZ2UgRGVmaW5pdGlvbnMgKi8NCkBwYWdlIFdvcmRT
ZWN0aW9uMQ0KCXtzaXplOjYxMi4wcHQgNzkyLjBwdDsNCgltYXJnaW46NzIuMHB0IDkwLjBwdCA3
Mi4wcHQgOTAuMHB0O30NCmRpdi5Xb3JkU2VjdGlvbjENCgl7cGFnZTpXb3JkU2VjdGlvbjE7fQ0K
LS0+PC9zdHlsZT48IS0tW2lmIGd0ZSBtc28gOV0+PHhtbD4NCjxvOnNoYXBlZGVmYXVsdHMgdjpl
eHQ9ImVkaXQiIHNwaWRtYXg9IjEwMjYiIC8+DQo8L3htbD48IVtlbmRpZl0tLT48IS0tW2lmIGd0
ZSBtc28gOV0+PHhtbD4NCjxvOnNoYXBlbGF5b3V0IHY6ZXh0PSJlZGl0Ij4NCjxvOmlkbWFwIHY6
ZXh0PSJlZGl0IiBkYXRhPSIxIiAvPg0KPC9vOnNoYXBlbGF5b3V0PjwveG1sPjwhW2VuZGlmXS0t
Pg0KPC9oZWFkPg0KPGJvZHkgbGFuZz0iWkgtQ04iIGxpbms9ImJsdWUiIHZsaW5rPSJwdXJwbGUi
IHN0eWxlPSJ0ZXh0LWp1c3RpZnktdHJpbTpwdW5jdHVhdGlvbiI+DQo8ZGl2IGNsYXNzPSJXb3Jk
U2VjdGlvbjEiPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCIgYWxpZ249ImxlZnQiIHN0eWxlPSJ0ZXh0
LWFsaWduOmxlZnQ7bGluZS1oZWlnaHQ6MTAuOHB0O2JhY2tncm91bmQ6d2hpdGU7dmVydGljYWwt
YWxpZ246YmFzZWxpbmUiPg0KPHNwYW4gbGFuZz0iRU4tVVMiIHN0eWxlPSJmb250LXNpemU6Ny4w
cHQ7Zm9udC1mYW1pbHk6Q291cmllcjtjb2xvcjpibGFjayI+SGksIGFsbCBOTUxSRyw8bzpwPjwv
bzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIiBhbGlnbj0ibGVmdCIgc3R5bGU9
InRleHQtYWxpZ246bGVmdDtsaW5lLWhlaWdodDoxMC44cHQ7YmFja2dyb3VuZDp3aGl0ZTt2ZXJ0
aWNhbC1hbGlnbjpiYXNlbGluZSI+DQo8c3BhbiBsYW5nPSJFTi1VUyIgc3R5bGU9ImZvbnQtc2l6
ZTo3LjBwdDtmb250LWZhbWlseTpDb3VyaWVyO2NvbG9yOmJsYWNrIj48bzpwPiZuYnNwOzwvbzpw
Pjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIiBhbGlnbj0ibGVmdCIgc3R5bGU9InRl
eHQtYWxpZ246bGVmdDtsaW5lLWhlaWdodDoxMC44cHQ7YmFja2dyb3VuZDp3aGl0ZTt2ZXJ0aWNh
bC1hbGlnbjpiYXNlbGluZSI+DQo8c3BhbiBsYW5nPSJFTi1VUyIgc3R5bGU9ImZvbnQtc2l6ZTo3
LjBwdDtmb250LWZhbWlseTpDb3VyaWVyO2NvbG9yOmJsYWNrIj5XZSBoYXZlIHJlcXVlc3RlZCBh
IHNlc3Npb24gb2YgMiBob3VyIGZvciB0aGUgb3VyIHByb3Bvc2VkIE5NTFJHIG1lZXRpbmcgZm9y
IElFVEYtOTYgKEJlcmxpbiwgR2VybWFueSkgYW5kIGFyZSBzdGFydGluZyB0byBjb2xsZWN0IGFn
ZW5kYSBpdGVtcyBmb3IgdGhpcyBzZXNzaW9ucy48bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8cCBj
bGFzcz0iTXNvTm9ybWFsIiBhbGlnbj0ibGVmdCIgc3R5bGU9InRleHQtYWxpZ246bGVmdDtsaW5l
LWhlaWdodDoxMC44cHQ7YmFja2dyb3VuZDp3aGl0ZTt2ZXJ0aWNhbC1hbGlnbjpiYXNlbGluZSI+
DQo8c3BhbiBsYW5nPSJFTi1VUyIgc3R5bGU9ImZvbnQtc2l6ZTo3LjBwdDtmb250LWZhbWlseTpD
b3VyaWVyO2NvbG9yOmJsYWNrIj48bzpwPiZuYnNwOzwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFz
cz0iTXNvTm9ybWFsIiBhbGlnbj0ibGVmdCIgc3R5bGU9InRleHQtYWxpZ246bGVmdDtsaW5lLWhl
aWdodDoxMC44cHQ7YmFja2dyb3VuZDp3aGl0ZTt2ZXJ0aWNhbC1hbGlnbjpiYXNlbGluZSI+DQo8
c3BhbiBsYW5nPSJFTi1VUyIgc3R5bGU9ImZvbnQtc2l6ZTo3LjBwdDtmb250LWZhbWlseTpDb3Vy
aWVyO2NvbG9yOmJsYWNrIj5QbGVhc2Ugc2VuZCB1cyAoYW5pbWEtY2hhaXJzIGF0IHRvb2xzLmll
dGYub3JnKSByZXF1ZXN0cyBmb3Igc2Vzc2lvbnMgYnkgSnVseSAmbmJzcDsxMDxzdXA+dGg8L3N1
cD4sIFN1bmRheSwgYW5kIGluY2x1ZGU6PG86cD48L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9
Ik1zb05vcm1hbCIgYWxpZ249ImxlZnQiIHN0eWxlPSJ0ZXh0LWFsaWduOmxlZnQ7bGluZS1oZWln
aHQ6MTAuOHB0O2JhY2tncm91bmQ6d2hpdGU7dmVydGljYWwtYWxpZ246YmFzZWxpbmUiPg0KPHNw
YW4gbGFuZz0iRU4tVVMiIHN0eWxlPSJmb250LXNpemU6Ny4wcHQ7Zm9udC1mYW1pbHk6Q291cmll
cjtjb2xvcjpibGFjayI+PG86cD4mbmJzcDs8L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1z
b05vcm1hbCIgYWxpZ249ImxlZnQiIHN0eWxlPSJ0ZXh0LWFsaWduOmxlZnQ7bGluZS1oZWlnaHQ6
MTAuOHB0O2JhY2tncm91bmQ6d2hpdGU7dmVydGljYWwtYWxpZ246YmFzZWxpbmUiPg0KPHNwYW4g
bGFuZz0iRU4tVVMiIHN0eWxlPSJmb250LXNpemU6Ny4wcHQ7Zm9udC1mYW1pbHk6Q291cmllcjtj
b2xvcjpibGFjayI+TmFtZSBvZiB0aW1lIHNsb3Q6PG86cD48L286cD48L3NwYW4+PC9wPg0KPHAg
Y2xhc3M9Ik1zb05vcm1hbCIgYWxpZ249ImxlZnQiIHN0eWxlPSJ0ZXh0LWFsaWduOmxlZnQ7bGlu
ZS1oZWlnaHQ6MTAuOHB0O2JhY2tncm91bmQ6d2hpdGU7dmVydGljYWwtYWxpZ246YmFzZWxpbmUi
Pg0KPHNwYW4gbGFuZz0iRU4tVVMiIHN0eWxlPSJmb250LXNpemU6Ny4wcHQ7Zm9udC1mYW1pbHk6
Q291cmllcjtjb2xvcjpibGFjayI+TmFtZSBvZiBkcmFmdChzKTo8bzpwPjwvbzpwPjwvc3Bhbj48
L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIiBhbGlnbj0ibGVmdCIgc3R5bGU9InRleHQtYWxpZ246
bGVmdDtsaW5lLWhlaWdodDoxMC44cHQ7YmFja2dyb3VuZDp3aGl0ZTt2ZXJ0aWNhbC1hbGlnbjpi
YXNlbGluZSI+DQo8c3BhbiBsYW5nPSJFTi1VUyIgc3R5bGU9ImZvbnQtc2l6ZTo3LjBwdDtmb250
LWZhbWlseTpDb3VyaWVyO2NvbG9yOmJsYWNrIj5UaW1lIHJlcXVlc3RlZDo8bzpwPjwvbzpwPjwv
c3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIiBhbGlnbj0ibGVmdCIgc3R5bGU9InRleHQt
YWxpZ246bGVmdDtsaW5lLWhlaWdodDoxMC44cHQ7YmFja2dyb3VuZDp3aGl0ZTt2ZXJ0aWNhbC1h
bGlnbjpiYXNlbGluZSI+DQo8c3BhbiBsYW5nPSJFTi1VUyIgc3R5bGU9ImZvbnQtc2l6ZTo3LjBw
dDtmb250LWZhbWlseTpDb3VyaWVyO2NvbG9yOmJsYWNrIj5QcmVzZW50ZXIgbmFtZShzKTo8bzpw
PjwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIiBhbGlnbj0ibGVmdCIgc3R5
bGU9InRleHQtYWxpZ246bGVmdDtsaW5lLWhlaWdodDoxMC44cHQ7YmFja2dyb3VuZDp3aGl0ZTt2
ZXJ0aWNhbC1hbGlnbjpiYXNlbGluZSI+DQo8c3BhbiBsYW5nPSJFTi1VUyIgc3R5bGU9ImZvbnQt
c2l6ZTo3LjBwdDtmb250LWZhbWlseTpDb3VyaWVyO2NvbG9yOmJsYWNrIj5CcmllZiBkZXNjcmlw
dGlvbiBvZiB3aGF0IGlzc3VlcyBuZWVkIGRpc2N1c3NpbmcgYW5kIHdoYXQgeW91IGhvcGUgdG88
bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIiBhbGlnbj0ibGVmdCIg
c3R5bGU9InRleHQtYWxpZ246bGVmdDtsaW5lLWhlaWdodDoxMC44cHQ7YmFja2dyb3VuZDp3aGl0
ZTt2ZXJ0aWNhbC1hbGlnbjpiYXNlbGluZSI+DQo8c3BhbiBsYW5nPSJFTi1VUyIgc3R5bGU9ImZv
bnQtc2l6ZTo3LjBwdDtmb250LWZhbWlseTpDb3VyaWVyO2NvbG9yOmJsYWNrIj5hY2NvbXBsaXNo
IGJ5IHByZXNlbnRpbmcgKHBsZWFzZSBmb2N1cyBvbiBvcGVuIGlzc3VlcywgcmF0aGVyIHRoYW4g
YTxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiIGFsaWduPSJsZWZ0
IiBzdHlsZT0idGV4dC1hbGlnbjpsZWZ0O2xpbmUtaGVpZ2h0OjEwLjhwdDtiYWNrZ3JvdW5kOndo
aXRlO3ZlcnRpY2FsLWFsaWduOmJhc2VsaW5lIj4NCjxzcGFuIGxhbmc9IkVOLVVTIiBzdHlsZT0i
Zm9udC1zaXplOjcuMHB0O2ZvbnQtZmFtaWx5OkNvdXJpZXI7Y29sb3I6YmxhY2siPnN0YXR1cyB1
cGRhdGUpOjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiIGFsaWdu
PSJsZWZ0IiBzdHlsZT0idGV4dC1hbGlnbjpsZWZ0O2xpbmUtaGVpZ2h0OjEwLjhwdDtiYWNrZ3Jv
dW5kOndoaXRlO3ZlcnRpY2FsLWFsaWduOmJhc2VsaW5lIj4NCjxzcGFuIGxhbmc9IkVOLVVTIiBz
dHlsZT0iZm9udC1zaXplOjcuMHB0O2ZvbnQtZmFtaWx5OkNvdXJpZXI7Y29sb3I6YmxhY2siPjxv
OnA+Jm5ic3A7PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiIGFsaWduPSJs
ZWZ0IiBzdHlsZT0idGV4dC1hbGlnbjpsZWZ0O2xpbmUtaGVpZ2h0OjEwLjhwdDtiYWNrZ3JvdW5k
OndoaXRlO3ZlcnRpY2FsLWFsaWduOmJhc2VsaW5lIj4NCjxzcGFuIGxhbmc9IkVOLVVTIiBzdHls
ZT0iZm9udC1zaXplOjcuMHB0O2ZvbnQtZmFtaWx5OkNvdXJpZXI7Y29sb3I6YmxhY2siPk1vcmUg
ZGV0YWlscyBhYm91dCB0aGUgSUVURiA5NiwgQmVybGluLCBHZXJtYW55IGNhbiBiZSBmb3VuZCBh
dDo8bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIiBhbGlnbj0ibGVm
dCIgc3R5bGU9InRleHQtYWxpZ246bGVmdDtsaW5lLWhlaWdodDoxMC44cHQ7YmFja2dyb3VuZDp3
aGl0ZTt2ZXJ0aWNhbC1hbGlnbjpiYXNlbGluZSI+DQo8c3BhbiBsYW5nPSJFTi1VUyIgc3R5bGU9
ImZvbnQtc2l6ZTo3LjBwdDtmb250LWZhbWlseTpDb3VyaWVyO2NvbG9yOmJsYWNrIj5odHRwOi8v
d3d3LmlldGYub3JnL21lZXRpbmcvOTYvaW5kZXguaHRtbDxvOnA+PC9vOnA+PC9zcGFuPjwvcD4N
CjxwIGNsYXNzPSJNc29Ob3JtYWwiIGFsaWduPSJsZWZ0IiBzdHlsZT0idGV4dC1hbGlnbjpsZWZ0
O2xpbmUtaGVpZ2h0OjEwLjhwdDtiYWNrZ3JvdW5kOndoaXRlO3ZlcnRpY2FsLWFsaWduOmJhc2Vs
aW5lIj4NCjxzcGFuIGxhbmc9IkVOLVVTIiBzdHlsZT0iZm9udC1zaXplOjcuMHB0O2ZvbnQtZmFt
aWx5OkNvdXJpZXI7Y29sb3I6YmxhY2siPjxvOnA+Jm5ic3A7PC9vOnA+PC9zcGFuPjwvcD4NCjxw
IGNsYXNzPSJNc29Ob3JtYWwiIGFsaWduPSJsZWZ0IiBzdHlsZT0idGV4dC1hbGlnbjpsZWZ0O2xp
bmUtaGVpZ2h0OjEwLjhwdDtiYWNrZ3JvdW5kOndoaXRlO3ZlcnRpY2FsLWFsaWduOmJhc2VsaW5l
Ij4NCjxzcGFuIGxhbmc9IkVOLVVTIiBzdHlsZT0iZm9udC1zaXplOjcuMHB0O2ZvbnQtZmFtaWx5
OkNvdXJpZXI7Y29sb3I6YmxhY2siPkFnYWluLCBwcmVzZW50ZXJzIGFuZCBkcmFmdCBhdXRob3Jz
IHBsZWFzZSBpbnZva2UgYWN0aXZlIGRpc2N1c3Npb25zIGluPG86cD48L286cD48L3NwYW4+PC9w
Pg0KPHAgY2xhc3M9Ik1zb05vcm1hbCIgYWxpZ249ImxlZnQiIHN0eWxlPSJ0ZXh0LWFsaWduOmxl
ZnQ7bGluZS1oZWlnaHQ6MTAuOHB0O2JhY2tncm91bmQ6d2hpdGU7dmVydGljYWwtYWxpZ246YmFz
ZWxpbmUiPg0KPHNwYW4gbGFuZz0iRU4tVVMiIHN0eWxlPSJmb250LXNpemU6Ny4wcHQ7Zm9udC1m
YW1pbHk6Q291cmllcjtjb2xvcjpibGFjayI+dGhlIE5NTFJHIGxpc3QuIFdlIGhhdmUgdmVyeSBs
aW1pdGVkIHRpbWUgZm9yIGVhY2ggdG9waWMgaW4gdGhlPG86cD48L286cD48L3NwYW4+PC9wPg0K
PHAgY2xhc3M9Ik1zb05vcm1hbCIgYWxpZ249ImxlZnQiIHN0eWxlPSJ0ZXh0LWFsaWduOmxlZnQ7
bGluZS1oZWlnaHQ6MTAuOHB0O2JhY2tncm91bmQ6d2hpdGU7dmVydGljYWwtYWxpZ246YmFzZWxp
bmUiPg0KPHNwYW4gbGFuZz0iRU4tVVMiIHN0eWxlPSJmb250LXNpemU6Ny4wcHQ7Zm9udC1mYW1p
bHk6Q291cmllcjtjb2xvcjpibGFjayI+ZmFjZS10by1mYWNlIG1lZXRpbmcuIE1haWwgbGlzdCBp
cyBhIHZlcnkgZ29vZCBwbGFjZSB0byBkaXNjdXNzIGFuZCByZWFjaDxvOnA+PC9vOnA+PC9zcGFu
PjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiIGFsaWduPSJsZWZ0IiBzdHlsZT0idGV4dC1hbGln
bjpsZWZ0O2xpbmUtaGVpZ2h0OjEwLjhwdDtiYWNrZ3JvdW5kOndoaXRlO3ZlcnRpY2FsLWFsaWdu
OmJhc2VsaW5lIj4NCjxzcGFuIGxhbmc9IkVOLVVTIiBzdHlsZT0iZm9udC1zaXplOjcuMHB0O2Zv
bnQtZmFtaWx5OkNvdXJpZXI7Y29sb3I6YmxhY2siPmNvbnNlbnN1cyBvbiB0ZWNobmljYWwgaXNz
dWVzLjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiIGFsaWduPSJs
ZWZ0IiBzdHlsZT0idGV4dC1hbGlnbjpsZWZ0O2xpbmUtaGVpZ2h0OjEwLjhwdDtiYWNrZ3JvdW5k
OndoaXRlO3ZlcnRpY2FsLWFsaWduOmJhc2VsaW5lIj4NCjxzcGFuIGxhbmc9IkVOLVVTIiBzdHls
ZT0iZm9udC1zaXplOjcuMHB0O2ZvbnQtZmFtaWx5OkNvdXJpZXI7Y29sb3I6YmxhY2siPjxvOnA+
Jm5ic3A7PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiIGFsaWduPSJsZWZ0
IiBzdHlsZT0idGV4dC1hbGlnbjpsZWZ0O2xpbmUtaGVpZ2h0OjEwLjhwdDtiYWNrZ3JvdW5kOndo
aXRlO3ZlcnRpY2FsLWFsaWduOmJhc2VsaW5lIj4NCjxzcGFuIGxhbmc9IkVOLVVTIiBzdHlsZT0i
Zm9udC1zaXplOjcuMHB0O2ZvbnQtZmFtaWx5OkNvdXJpZXI7Y29sb3I6YmxhY2siPkJlc3QgcmVn
YXJkcyw8bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIiBhbGlnbj0i
bGVmdCIgc3R5bGU9InRleHQtYWxpZ246bGVmdDtsaW5lLWhlaWdodDoxMC44cHQ7YmFja2dyb3Vu
ZDp3aGl0ZTt2ZXJ0aWNhbC1hbGlnbjpiYXNlbGluZSI+DQo8c3BhbiBsYW5nPSJFTi1VUyIgc3R5
bGU9ImZvbnQtc2l6ZTo3LjBwdDtmb250LWZhbWlseTpDb3VyaWVyO2NvbG9yOmJsYWNrIj48bzpw
PiZuYnNwOzwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIiBhbGlnbj0ibGVm
dCIgc3R5bGU9InRleHQtYWxpZ246bGVmdDtsaW5lLWhlaWdodDoxMC44cHQ7YmFja2dyb3VuZDp3
aGl0ZTt2ZXJ0aWNhbC1hbGlnbjpiYXNlbGluZSI+DQo8c3BhbiBsYW5nPSJFTi1VUyIgc3R5bGU9
ImZvbnQtc2l6ZTo3LjBwdDtmb250LWZhbWlseTpDb3VyaWVyO2NvbG9yOmJsYWNrIj5BbGJlcnQg
JmFtcDsgU2hlbmc8L3NwYW4+PHNwYW4gbGFuZz0iRU4tVVMiPjxvOnA+PC9vOnA+PC9zcGFuPjwv
cD4NCjwvZGl2Pg0KPC9ib2R5Pg0KPC9odG1sPg0K

--_000_5D36713D8A4E7348A7E10DF7437A4B927CA8C67ANKGEML515MBXchi_--


From nobody Wed Jul 13 20:11:22 2016
Return-Path: <jiangsheng@huawei.com>
X-Original-To: nmlrg@ietfa.amsl.com
Delivered-To: nmlrg@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 5654712D0F7 for <nmlrg@ietfa.amsl.com>; Wed, 13 Jul 2016 20:11:21 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -5.507
X-Spam-Level: 
X-Spam-Status: No, score=-5.507 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RP_MATCHES_RCVD=-1.287, SPF_PASS=-0.001] autolearn=ham autolearn_force=no
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id XxhgULun2z0c for <nmlrg@ietfa.amsl.com>; Wed, 13 Jul 2016 20:11:16 -0700 (PDT)
Received: from szxga01-in.huawei.com (szxga01-in.huawei.com [58.251.152.64]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 9FFA612B009 for <nmlrg@irtf.org>; Wed, 13 Jul 2016 20:11:12 -0700 (PDT)
Received: from 172.24.1.136 (EHLO nkgeml414-hub.china.huawei.com) ([172.24.1.136]) by szxrg01-dlp.huawei.com (MOS 4.3.7-GA FastPath queued) with ESMTP id DNP82054; Thu, 14 Jul 2016 11:11:02 +0800 (CST)
Received: from NKGEML515-MBX.china.huawei.com ([fe80::a54a:89d2:c471:ff]) by nkgeml414-hub.china.huawei.com ([10.98.56.75]) with mapi id 14.03.0235.001; Thu, 14 Jul 2016 11:10:58 +0800
From: Sheng Jiang <jiangsheng@huawei.com>
To: "nmlrg@irtf.org" <nmlrg@irtf.org>
Thread-Topic: Agenda for NMLRG #4, July 18th, IETF 96, Berlin, Germany
Thread-Index: AdHdfUHloG9PfHxeTfapE/21TrLyhg==
Date: Thu, 14 Jul 2016 03:10:57 +0000
Message-ID: <5D36713D8A4E7348A7E10DF7437A4B927CAC2CE6@NKGEML515-MBX.china.huawei.com>
Accept-Language: en-GB, zh-CN, en-US
Content-Language: zh-CN
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
x-originating-ip: [10.111.99.197]
Content-Type: multipart/alternative; boundary="_000_5D36713D8A4E7348A7E10DF7437A4B927CAC2CE6NKGEML515MBXchi_"
MIME-Version: 1.0
X-CFilter-Loop: Reflected
X-Mirapoint-Virus-RAPID-Raw: score=unknown(0), refid=str=0001.0A020205.578702C7.0085, ss=1, re=0.000, recu=0.000, reip=0.000,  cl=1, cld=1, fgs=0, ip=0.0.0.0, so=2013-06-18 04:22:30, dmn=2013-03-21 17:37:32
X-Mirapoint-Loop-Id: 54dbace6833ab79fec24885813859d2c
Archived-At: <https://mailarchive.ietf.org/arch/msg/nmlrg/yrUye0dFTeyiprSb2XRRmV-GsWg>
Subject: [Nmlrg] Agenda for NMLRG #4, July 18th, IETF 96, Berlin, Germany
X-BeenThere: nmlrg@irtf.org
X-Mailman-Version: 2.1.17
Precedence: list
List-Id: Network Machine Learning Research Group <nmlrg.irtf.org>
List-Unsubscribe: <https://www.irtf.org/mailman/options/nmlrg>, <mailto:nmlrg-request@irtf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/nmlrg/>
List-Post: <mailto:nmlrg@irtf.org>
List-Help: <mailto:nmlrg-request@irtf.org?subject=help>
List-Subscribe: <https://www.irtf.org/mailman/listinfo/nmlrg>, <mailto:nmlrg-request@irtf.org?subject=subscribe>
X-List-Received-Date: Thu, 14 Jul 2016 03:11:21 -0000

--_000_5D36713D8A4E7348A7E10DF7437A4B927CAC2CE6NKGEML515MBXchi_
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: base64

SGksIGFsbCwNCg0KVGhlIGNoYWlycyBoYXZlIHVwbG9hZGVkIGEgcHJpbWFyeSBtZWV0aW5nIGFn
ZW5kYSBmb3Igb3VyIHByb3Bvc2VkIE5NTCBSRyBpbiBCZXJsaW4sIEdlcm1hbnkuDQoNCjE1OjQw
LTE3OjQwICBKdWx5IDE4dGgsIE1vbmRheSBBZnRlcm5vb24gU2Vzc2lvbiBJSSwgUG90c2RhbSBJ
SQ0KDQpodHRwczovL3d3dy5pZXRmLm9yZy9wcm9jZWVkaW5ncy85Ni9hZ2VuZGEvYWdlbmRhLTk2
LW5tbHJnDQoNCklmIHRoZXJlIGlzIGFueSBxdWVzdGlvbiBvciByZXF1ZXN0IG9uIHRoZSBhZ2Vu
ZGEsIHBsZWFzZSBsZXQgdGhlIGNoYWlycyBrbm93Lg0KDQpUaGUgc3BlYWtlcnMgcGxlYXNlIHNl
bmQgeW91ciBzbGlkZXMgdG8gdGhlIGNoYWlyIGluIGFkdmFuY2UsIHNvIHRoYXQgd2UgY291bGQg
dXBsb2FkIHRoZW0gYmVmb3JlIHRoZSBtZWV0aW5nLiBUaGUgc2xpZGVzIGFyZSBkdWUgYmVmb3Jl
IDRwbSBvbiBTdW5kYXksIEp1bHkgMTd0aC4NCg0KQmVzdCByZWdhcmRzLA0KDQpTaGVuZyArIEFs
YmVydA0K

--_000_5D36713D8A4E7348A7E10DF7437A4B927CAC2CE6NKGEML515MBXchi_
Content-Type: text/html; charset="utf-8"
Content-Transfer-Encoding: base64

PGh0bWwgeG1sbnM6dj0idXJuOnNjaGVtYXMtbWljcm9zb2Z0LWNvbTp2bWwiIHhtbG5zOm89InVy
bjpzY2hlbWFzLW1pY3Jvc29mdC1jb206b2ZmaWNlOm9mZmljZSIgeG1sbnM6dz0idXJuOnNjaGVt
YXMtbWljcm9zb2Z0LWNvbTpvZmZpY2U6d29yZCIgeG1sbnM6bT0iaHR0cDovL3NjaGVtYXMubWlj
cm9zb2Z0LmNvbS9vZmZpY2UvMjAwNC8xMi9vbW1sIiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcv
VFIvUkVDLWh0bWw0MCI+DQo8aGVhZD4NCjxtZXRhIGh0dHAtZXF1aXY9IkNvbnRlbnQtVHlwZSIg
Y29udGVudD0idGV4dC9odG1sOyBjaGFyc2V0PXV0Zi04Ij4NCjxtZXRhIG5hbWU9IkdlbmVyYXRv
ciIgY29udGVudD0iTWljcm9zb2Z0IFdvcmQgMTIgKGZpbHRlcmVkIG1lZGl1bSkiPg0KPHN0eWxl
PjwhLS0NCi8qIEZvbnQgRGVmaW5pdGlvbnMgKi8NCkBmb250LWZhY2UNCgl7Zm9udC1mYW1pbHk6
5a6L5L2TOw0KCXBhbm9zZS0xOjIgMSA2IDAgMyAxIDEgMSAxIDE7fQ0KQGZvbnQtZmFjZQ0KCXtm
b250LWZhbWlseToiQ2FtYnJpYSBNYXRoIjsNCglwYW5vc2UtMToyIDQgNSAzIDUgNCA2IDMgMiA0
O30NCkBmb250LWZhY2UNCgl7Zm9udC1mYW1pbHk6Q2FsaWJyaTsNCglwYW5vc2UtMToyIDE1IDUg
MiAyIDIgNCAzIDIgNDt9DQpAZm9udC1mYWNlDQoJe2ZvbnQtZmFtaWx5OiJcQOWui+S9kyI7DQoJ
cGFub3NlLTE6MiAxIDYgMCAzIDEgMSAxIDEgMTt9DQovKiBTdHlsZSBEZWZpbml0aW9ucyAqLw0K
cC5Nc29Ob3JtYWwsIGxpLk1zb05vcm1hbCwgZGl2Lk1zb05vcm1hbA0KCXttYXJnaW46MGNtOw0K
CW1hcmdpbi1ib3R0b206LjAwMDFwdDsNCgl0ZXh0LWFsaWduOmp1c3RpZnk7DQoJdGV4dC1qdXN0
aWZ5OmludGVyLWlkZW9ncmFwaDsNCglmb250LXNpemU6MTAuNXB0Ow0KCWZvbnQtZmFtaWx5OiJD
YWxpYnJpIiwic2Fucy1zZXJpZiI7fQ0KYTpsaW5rLCBzcGFuLk1zb0h5cGVybGluaw0KCXttc28t
c3R5bGUtcHJpb3JpdHk6OTk7DQoJY29sb3I6Ymx1ZTsNCgl0ZXh0LWRlY29yYXRpb246dW5kZXJs
aW5lO30NCmE6dmlzaXRlZCwgc3Bhbi5Nc29IeXBlcmxpbmtGb2xsb3dlZA0KCXttc28tc3R5bGUt
cHJpb3JpdHk6OTk7DQoJY29sb3I6cHVycGxlOw0KCXRleHQtZGVjb3JhdGlvbjp1bmRlcmxpbmU7
fQ0KcHJlDQoJe21zby1zdHlsZS1wcmlvcml0eTo5OTsNCgltc28tc3R5bGUtbGluazoiSFRNTCDp
ooTorr7moLzlvI8gQ2hhciI7DQoJbWFyZ2luOjBjbTsNCgltYXJnaW4tYm90dG9tOi4wMDAxcHQ7
DQoJZm9udC1zaXplOjEyLjBwdDsNCglmb250LWZhbWlseTrlrovkvZM7fQ0Kc3Bhbi5FbWFpbFN0
eWxlMTcNCgl7bXNvLXN0eWxlLXR5cGU6cGVyc29uYWwtY29tcG9zZTsNCglmb250LWZhbWlseToi
Q2FsaWJyaSIsInNhbnMtc2VyaWYiOw0KCWNvbG9yOndpbmRvd3RleHQ7fQ0Kc3Bhbi5IVE1MQ2hh
cg0KCXttc28tc3R5bGUtbmFtZToiSFRNTCDpooTorr7moLzlvI8gQ2hhciI7DQoJbXNvLXN0eWxl
LXByaW9yaXR5Ojk5Ow0KCW1zby1zdHlsZS1saW5rOiJIVE1MIOmihOiuvuagvOW8jyI7DQoJZm9u
dC1mYW1pbHk65a6L5L2TO30NCi5Nc29DaHBEZWZhdWx0DQoJe21zby1zdHlsZS10eXBlOmV4cG9y
dC1vbmx5O30NCi8qIFBhZ2UgRGVmaW5pdGlvbnMgKi8NCkBwYWdlIFdvcmRTZWN0aW9uMQ0KCXtz
aXplOjYxMi4wcHQgNzkyLjBwdDsNCgltYXJnaW46NzIuMHB0IDkwLjBwdCA3Mi4wcHQgOTAuMHB0
O30NCmRpdi5Xb3JkU2VjdGlvbjENCgl7cGFnZTpXb3JkU2VjdGlvbjE7fQ0KLS0+PC9zdHlsZT48
IS0tW2lmIGd0ZSBtc28gOV0+PHhtbD4NCjxvOnNoYXBlZGVmYXVsdHMgdjpleHQ9ImVkaXQiIHNw
aWRtYXg9IjEwMjYiIC8+DQo8L3htbD48IVtlbmRpZl0tLT48IS0tW2lmIGd0ZSBtc28gOV0+PHht
bD4NCjxvOnNoYXBlbGF5b3V0IHY6ZXh0PSJlZGl0Ij4NCjxvOmlkbWFwIHY6ZXh0PSJlZGl0IiBk
YXRhPSIxIiAvPg0KPC9vOnNoYXBlbGF5b3V0PjwveG1sPjwhW2VuZGlmXS0tPg0KPC9oZWFkPg0K
PGJvZHkgbGFuZz0iWkgtQ04iIGxpbms9ImJsdWUiIHZsaW5rPSJwdXJwbGUiIHN0eWxlPSJ0ZXh0
LWp1c3RpZnktdHJpbTpwdW5jdHVhdGlvbiI+DQo8ZGl2IGNsYXNzPSJXb3JkU2VjdGlvbjEiPg0K
PHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gbGFuZz0iRU4tVVMiPkhpLCBhbGwsPG86cD48L286
cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gbGFuZz0iRU4tVVMiPjxv
OnA+Jm5ic3A7PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIGxh
bmc9IkVOLVVTIj5UaGUgY2hhaXJzIGhhdmUgdXBsb2FkZWQgYSBwcmltYXJ5IG1lZXRpbmcgYWdl
bmRhIGZvciBvdXIgcHJvcG9zZWQgTk1MIFJHIGluIEJlcmxpbiwgR2VybWFueS48bzpwPjwvbzpw
Pjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBsYW5nPSJFTi1VUyI+PG86
cD4mbmJzcDs8L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gbGFu
Zz0iRU4tVVMiPjE1OjQwLTE3OjQwJm5ic3A7IEp1bHkgMTg8c3VwPnRoPC9zdXA+LCBNb25kYXkg
QWZ0ZXJub29uIFNlc3Npb24gSUksIFBvdHNkYW0gSUk8bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8
cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBsYW5nPSJFTi1VUyI+PG86cD4mbmJzcDs8L286cD48
L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gbGFuZz0iRU4tVVMiPmh0dHBz
Oi8vd3d3LmlldGYub3JnL3Byb2NlZWRpbmdzLzk2L2FnZW5kYS9hZ2VuZGEtOTYtbm1scmc8bzpw
PjwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBsYW5nPSJFTi1V
UyI+PG86cD4mbmJzcDs8L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNw
YW4gbGFuZz0iRU4tVVMiPklmIHRoZXJlIGlzIGFueSBxdWVzdGlvbiBvciByZXF1ZXN0IG9uIHRo
ZSBhZ2VuZGEsIHBsZWFzZSBsZXQgdGhlIGNoYWlycyBrbm93LjxvOnA+PC9vOnA+PC9zcGFuPjwv
cD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIGxhbmc9IkVOLVVTIj48bzpwPiZuYnNwOzwv
bzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBsYW5nPSJFTi1VUyI+
VGhlIHNwZWFrZXJzIHBsZWFzZSBzZW5kIHlvdXIgc2xpZGVzIHRvIHRoZSBjaGFpciBpbiBhZHZh
bmNlLCBzbyB0aGF0IHdlIGNvdWxkIHVwbG9hZCB0aGVtIGJlZm9yZSB0aGUgbWVldGluZy4gVGhl
IHNsaWRlcyBhcmUgZHVlIGJlZm9yZSA0cG0gb24gU3VuZGF5LCBKdWx5IDE3dGguPG86cD48L286
cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gbGFuZz0iRU4tVVMiPjxv
OnA+Jm5ic3A7PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIGxh
bmc9IkVOLVVTIj5CZXN0IHJlZ2FyZHMsPG86cD48L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9
Ik1zb05vcm1hbCI+PHNwYW4gbGFuZz0iRU4tVVMiPjxvOnA+Jm5ic3A7PC9vOnA+PC9zcGFuPjwv
cD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIGxhbmc9IkVOLVVTIj5TaGVuZyAmIzQzOyBB
bGJlcnQ8bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8L2Rpdj4NCjwvYm9keT4NCjwvaHRtbD4NCg==

--_000_5D36713D8A4E7348A7E10DF7437A4B927CAC2CE6NKGEML515MBXchi_--


From nobody Sat Jul 16 07:11:31 2016
Return-Path: <albert.cabellos@gmail.com>
X-Original-To: nmlrg@ietfa.amsl.com
Delivered-To: nmlrg@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 31C6B12D572 for <nmlrg@ietfa.amsl.com>; Sat, 16 Jul 2016 07:11:29 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.699
X-Spam-Level: 
X-Spam-Status: No, score=-2.699 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-0.7, SPF_PASS=-0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Apv27tR4jrhX for <nmlrg@ietfa.amsl.com>; Sat, 16 Jul 2016 07:11:26 -0700 (PDT)
Received: from mail-wm0-x232.google.com (mail-wm0-x232.google.com [IPv6:2a00:1450:400c:c09::232]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 5571812D56F for <nmlrg@irtf.org>; Sat, 16 Jul 2016 07:11:25 -0700 (PDT)
Received: by mail-wm0-x232.google.com with SMTP id f65so55823254wmi.0 for <nmlrg@irtf.org>; Sat, 16 Jul 2016 07:11:25 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113;  h=mime-version:from:date:message-id:subject:to; bh=PUR7DGcfdF66wRm38lHNwJmEaMz/tPgGUwkgyx1JZWM=; b=LNrUXEK3w09ibi3v5+mw4eT39cmjKDh3iwxz/ZOXjhBeYpIhDz9094s5EJQYFAZEdS T5vLfsI4Nb2slP2+G1yjsoWqtogWRz7RvYIYjAdb56Y/F2h447YcizT0NVK3dQ6uyeuY oQWR8CB+FKyYXydEO9GjMAY6kWuUrwydg0LpMwE3UE89Ls+KYUvQVLVI2BbOF6HU0aj+ mkeAMBByfC9itOSawKizwm92pYzcnvJPX7SNwPwzBGEMgpp94/7bmlKF9t5QEXEiLljl susJMSEmEA+bV336u5dCJQqmyTw9LC43gC2khRAFpRDj9sEtqKwUe96KN4e6BX8GM0Jo umjQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=PUR7DGcfdF66wRm38lHNwJmEaMz/tPgGUwkgyx1JZWM=; b=P8ScJq5jFlu4iJ477Wi5ct7YZOoQzTfZzBT+8uqTFkwQOgrk3O5C5gRIuYgzD9lQlM 79s0BWyl8dpr4iPWbfH7CfTnRUyp9DnJcQds/M6tpVJOQRK++ICeLJR2q3/JDRLxjc0l WhBnth3LIi8z8oXq9YPsQVZtdUYGUNDx7b0AMIiJXpRBGVlotmk6tv26+Yq8H78As6W8 re9+vmqYInnqKDz3XUnogks8K3OB89K71d4PDDO4jfbX27zTiZPLCJiM5HprrwfAJqye www/Ps4F6ZFiTy4xFjV+M9RnUjdepP1twoPDX097/Fc9/ItBdJ8mNEccyQH99Gzg28uL Itkg==
X-Gm-Message-State: ALyK8tKHrEfLxRWTK4CymglchIzwgv/NGelnt8v2lOmxR5x/WU2bc1rbKoVu0AT//wlKMBGS4v8FZD6DMkdThg==
X-Received: by 10.28.9.213 with SMTP id 204mr46781534wmj.88.1468678283688; Sat, 16 Jul 2016 07:11:23 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.28.56.131 with HTTP; Sat, 16 Jul 2016 07:11:23 -0700 (PDT)
From: Albert Cabellos <albert.cabellos@gmail.com>
Date: Sat, 16 Jul 2016 16:11:23 +0200
Message-ID: <CAGE_QewtGRL58K-XLrFOE9a-vMjJEV8v5sthMQ3OeHdzAOKK8A@mail.gmail.com>
To: draft-jiang-nmlrg-traffic-machine-learning@ietf.org, nmlrg@irtf.org
Content-Type: multipart/alternative; boundary=001a11443e3063bad00537c14f75
Archived-At: <https://mailarchive.ietf.org/arch/msg/nmlrg/bTITpyP38HiD4xsnaKXvIrqLTaI>
Subject: [Nmlrg] Review for draft-jiang-nmlrg-traffic-machine-learning-00.txt
X-BeenThere: nmlrg@irtf.org
X-Mailman-Version: 2.1.17
Precedence: list
List-Id: Network Machine Learning Research Group <nmlrg.irtf.org>
List-Unsubscribe: <https://www.irtf.org/mailman/options/nmlrg>, <mailto:nmlrg-request@irtf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/nmlrg/>
List-Post: <mailto:nmlrg@irtf.org>
List-Help: <mailto:nmlrg-request@irtf.org?subject=help>
List-Subscribe: <https://www.irtf.org/mailman/listinfo/nmlrg>, <mailto:nmlrg-request@irtf.org?subject=subscribe>
X-List-Received-Date: Sat, 16 Jul 2016 14:11:29 -0000

--001a11443e3063bad00537c14f75
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Hi all

Find below a review for draft-jiang-nmlrg-traffic-machine-learning-00.txt,
thanks for putting up this very relevant draft.

Albert


1.  Introduction


[snip]

   It is natural to utilize powerful machine learning technology to
>    analyze the large mount of data regarding network traffic, to
>    understand the network's status, such as performance, failures,
>    security, etc.  It is a big advantage that machines can measure and
>    analyse the network traffic, then report the results and predictions
>    to humans for further decision.  The machines could handle vast
>    amounts of data which is almost impossible for humans to deal with,
>    in close to real time.  Even more, if the speed and accuracy of the
>    prediction is high enough, it is possible that the subsequent action
>    based on the prediction result could form a closed control loop to
>    achieve autonomic management.  However, the maturity of latter might
>    be far in the future.  Today, the traditional control programs still
>    look more reliable than machine learning based control mechanisms.


I don=C2=B4t personally see the closed control-loop that far in the future,=
 a
ML-based (flow-analysis) application detection can be used to automatically
choose appropriate routes for the flows.

>
>    This document firstly analyzes the data of the network traffic from
>    various perspectives; and also discusses several important practical
>    considerations, including the training data source, data storage and
>    the learning system architecture.  It then introduce a set of use
>    cases, which have been shown to work well although there is large
>    scope for improvements, including ML-based traffic classification,
>    traffic management, interface failure prediction, etc.


This is a very relevant objective, I=C2=B4d also like to see general
conclusions, lessons learnt, maybe a general architecture that accommodates
all the use-cases, etc? What do these use-cases have in common? What are
the common challenges? Are there traffic features common to all the
use-cases? Can we list such traffic features?

2.  Terminology


[snip]


   Traffic Flow  A sequence of packets from a source computer to a
>       destination [RFC6437].  It is the unit of network traffic.



To me the unit of network traffic is a packet, I haven=C2=B4t read 6437, ma=
ybe I
am missing something.

[snip]

3.1.  Data of the Network Traffic


[snip]

   User content  User contents are the payload of packets, which might
>       be obtained by DPI (Deep Packet Inspection) within the transit
>       network if the packets are unencrypted, or they could be analyzed
>       by the source or destination nodes.


Why =E2=80=98user content=E2=80=99? Why not =E2=80=98content=E2=80=99? Ther=
e are initiatives for DPI over
encrypted traffic:

http://iot.stanford.edu/pubs/sherry-blindbox-sigcomm15.pdf

3.3.  Architecture Considerations


>    Offline & online learning
>
>       *  Co-located mode: training (offline, based on historic data) and
>          prediction (online, based on real-time data) are both done
>          within the same entity.  The entity could be a central
>          repository or a specific node.
>
>       *  De-coupled mode: training is done in the central repository,
>          and prediction is made by the routers/switches/firewalls or
>          other devices that directly process the network traffic.
>
>
>    Central learning & distributed learning  Central learning means the
>       learning process is done at a single entity, which is either a
>       central repository or a node.  Distributed learning refer to
>       ensemble learning that multiple entities do the learning
>       simultaneously and ensemble the results together to sort out a
>       final results.  Since network devices are naturally distributed,
>       it could be foreseen that ensemble learning is a good approach for
>       a certain of use cases.


This last paragraph is obscure and unclear to me, what is the relation
between this an offline/online learning? I think that this could be
simplified by stating than learning can be offline and applied online, or
directly online, either on a network device or in a centralized entity.

3.4.  Closed Control Loop
>
>    The prediction made by machine learning mechanism could be directly
>    used on manipulating the network traffic, or other relevant actions,
>    such as changing the device configuration, etc.
>
>    However, as the introduction section said, this kind of utilization
>    might be suitable only for a small set of the use cases, due to the
>    limited accuracy of machine learning technologies.  Besides, some
>    critical usages simply cannot tolerate any false decision.


See my comment above, this paragraph suggests that closed control-loop is a
very long-term application, why? ML-based closed control-loops are
currently applied in many fields (e.g, computer vision, self-driving cars,
etc). Some people may argue that such applications are more complex and/or
critical than network traffic analysis.

[snip]

4.1.  HTTPS Traffic Classification


[snip]

   As a concrete example, Google, Facebook or Amazon are service
>    providers while maps, drive, gmail are services of Google.  To
>    identify them when they are accessed by a user, IP addresses and DNS
>    (Domain Name System) names based identification is not reliable as
>    the users can relies on intermediates to respectively serve as proxy
>    or resolve DNS requests.  The SNI (Server Name Indication) [RFC5246]
>    is an extension of HTTPS which is indicated by the user when
>    initiating the TLS handshake (Client Hello).  SNI actually contains
>    the hostname to which the request is addressed.  Such an hostname is
>    significative of the service and service provider name.  However, SNI
>    is an optional field and can be easily forged to circumvent HTTPS
>    filtering without impacting service use [bypasssni].  More advanced
>    mechanisms are hence necessary to improve the robustness of
>    identification even in the case of non collaborative users.


I suggest being vendor-agnostic in the examples, the specific examples do
not improve the draft by any means.

[snip]

>
>
>      HTTPS Connection
>            +
>            |(1)
>    +-------v------+
>    |TLS Connection|
>    |Reconstruction|
>    +-------+------+
>            |(2)
>    +-------v------+    (3')                    (4')
>    |  Features    +-------------+----------------------------+
>    |  Extraction  |             |                            |
>    +-------+------+     +-------v---------+             +----v----+
>            |            |Service Provider +------------->Services |
>            |(3)         |L1 model         |   Load      |L2 model |
>            |            +-------^---------+   services  +----^----+
>    +-------v------+             |             model X        |
>    |SNI Labelling |             +----------------------------+
>    +-------+------+                         |(5)
>            |            +-----------------------------------------+
>            +------------>              Training and               |
>                    (4)  |              Models building            |
>                         +-----------------------------------------+
>
>    Two-levels HTTPS traffic classification
>
>    In figure above, step(1) consists in reconstructing the HTTPS
>    connection and retrieving packets on top of which the following
>    metrics are observed (2):
>
>    o  Inter Arrival Time
>
>    o  Packet size
>
>    o  Encrypted data size: this feature has the advantage to be strongly
>       related to the service accessed instead of the packet size which
>       is biased by other lower layer headers
>
>    Based on these values, aggregated features are computed: average,
>    minimum, maximum, 25th percentile, median, 75th percentile.
>
>
Does the authors see value on listing all the traffic features in an ANNEX?


[snip]


4.2.  Malicious Domains: Automatic Detection with DNS Traffic Analysis


[snip]


>    As a result, in an automated fashion, a large variety of suspicious
>    domains can be detected, including phishing, malware, but also other
>    types, such as fake pharmaceutical shops as well as counterfeit
>    sneakers.  In this particular case, the responsible registrars are
>    notified in this pilot study about these websites.  Ultimately, it
>    allows these websites to be taken down, minimizing the potential
>    number of victims.



Can this use-cases be elaborated a little bit further? Which are features
used? And how ML is applied? Which algorithm? Where does the training set
come from?


4.3.  Machine-learning based Policy Derivation and Evaluation in
>       Broadband Networks



[snip]

   It is evident that machine learning can have significant importance
>    in policy derivation and evaluation in broadband networks, especially
>    towards in 5G infrastructures which will be complex, heterogeneous
>    and need to accommodate multi-services ranging from mobile broadband
>    to massive machine type, mission critical and vehicular
>    communications.


This use-cases relates to a very relevant area in ML (5G), but I don=C2=B4t=
 see
how ML is being applied? Is ML generating knowledge (unsupervised) to offer
recommendations to the netadmins? Is ML used to estimate the performance of
a given policy?

4.4.  Traffic Anomaly Detection in the Router


[snip]

   Besides wavelet analysis, there might be more techniques to explore,
>    such as correlation analysis of traffic anomaly events among multiple
>    devices.


In some cases and beyond correlation with time, ML can be applied to
correlate traffic with external information (weather, calendar, etc) that
may play an important role in the traffic profile.

4.5.  Applications of Machine Learning to Flow Monitoring


>    A commercial cloud-based flow monitoring service from Network
>    Polygraph [polygraph] has used Machine Learning analysis as a cost-
>    effective alternative to DPI for traffic classification, which
>    identifies the application responsible for each network traffic flow.


I suggest to make the use-case neutral.

   The target objective is to progressively reduce the dependence on DPI
>    technologies, which are expensive, difficult to deploy, not scalable,
>    and not robust against encryption, in favor of flow-based machine
>    learning approaches that are more cost-effective and can be easily
>    offered as a cloud service.  In this direction, some research
>    challenges include the classification of web services and CDN traffic
>    from flow-based measurements, and the combination of multiple ground
>    truths obtained from vantage points in different networks.
>

Does the authors see value on listing all traffic features used in an ANNEX=
?

5.  Security Considerations
>
>    This document is focused on applying machine learning in network,
>    including of course applying machine learning in network security, on
>    higher-layer concepts.  Therefore, it does not itself create any new
>    security issues.


I second Brian=E2=80=99s comment, a brief discussion on privacy concerns fo=
r these
very specific use-cases would be very valuable.

--001a11443e3063bad00537c14f75
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hi all<br><br>Find below a review for draft-jiang-nmlrg-tr=
affic-machine-learning-00.txt, thanks for putting up this very relevant dra=
ft.<div><br></div><div>Albert</div><div><br></div><div><br></div><blockquot=
e class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left-width=
:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-lef=
t:1ex">1.=C2=A0 Introduction</blockquote><div>=C2=A0</div><div>[snip]<br><b=
r><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;borde=
r-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204)=
;padding-left:1ex">=C2=A0 =C2=A0It is natural to utilize powerful machine l=
earning technology to<br>=C2=A0 =C2=A0analyze the large mount of data regar=
ding network traffic, to<br>=C2=A0 =C2=A0understand the network&#39;s statu=
s, such as performance, failures,<br>=C2=A0 =C2=A0security, etc.=C2=A0 It i=
s a big advantage that machines can measure and<br>=C2=A0 =C2=A0analyse the=
 network traffic, then report the results and predictions<br>=C2=A0 =C2=A0t=
o humans for further decision.=C2=A0 The machines could handle vast<br>=C2=
=A0 =C2=A0amounts of data which is almost impossible for humans to deal wit=
h,<br>=C2=A0 =C2=A0in close to real time.=C2=A0 Even more, if the speed and=
 accuracy of the<br>=C2=A0 =C2=A0prediction is high enough, it is possible =
that the subsequent action<br>=C2=A0 =C2=A0based on the prediction result c=
ould form a closed control loop to<br>=C2=A0 =C2=A0achieve autonomic manage=
ment.=C2=A0 However, the maturity of latter might<br>=C2=A0 =C2=A0be far in=
 the future.=C2=A0 Today, the traditional control programs still<br>=C2=A0 =
=C2=A0look more reliable than machine learning based control mechanisms.</b=
lockquote><br>I don=C2=B4t personally see the closed control-loop that far =
in the future, a ML-based (flow-analysis) application detection can be used=
 to automatically choose appropriate routes for the flows.<br><blockquote c=
lass=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left-width:1p=
x;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1=
ex"><br>=C2=A0 =C2=A0This document firstly analyzes the data of the network=
 traffic from<br>=C2=A0 =C2=A0various perspectives; and also discusses seve=
ral important practical<br>=C2=A0 =C2=A0considerations, including the train=
ing data source, data storage and<br>=C2=A0 =C2=A0the learning system archi=
tecture.=C2=A0 It then introduce a set of use<br>=C2=A0 =C2=A0cases, which =
have been shown to work well although there is large<br>=C2=A0 =C2=A0scope =
for improvements, including ML-based traffic classification,<br>=C2=A0 =C2=
=A0traffic management, interface failure prediction, etc.</blockquote><br>T=
his is a very relevant objective, I=C2=B4d also like to see general conclus=
ions, lessons learnt, maybe a general architecture that accommodates all th=
e use-cases, etc? What do these use-cases have in common? What are the comm=
on challenges? Are there traffic features common to all the use-cases? Can =
we list such traffic features?<br><br><blockquote class=3D"gmail_quote" sty=
le=3D"margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:soli=
d;border-left-color:rgb(204,204,204);padding-left:1ex">2.=C2=A0 Terminology=
</blockquote><br>[snip]<br><br><br><blockquote class=3D"gmail_quote" style=
=3D"margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;=
border-left-color:rgb(204,204,204);padding-left:1ex">=C2=A0 =C2=A0Traffic F=
low =C2=A0A sequence of packets from a source computer to a<br>=C2=A0 =C2=
=A0 =C2=A0 destination [RFC6437].=C2=A0 It is the unit of network traffic.<=
/blockquote><br><br>To me the unit of network traffic is a packet, I haven=
=C2=B4t read 6437, maybe I am missing something.<br><br>[snip]<br><br><bloc=
kquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left-=
width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);paddin=
g-left:1ex">3.1.=C2=A0 Data of the Network Traffic</blockquote><br>[snip]<b=
r><br><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;b=
order-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,=
204);padding-left:1ex">=C2=A0 =C2=A0User content =C2=A0User contents are th=
e payload of packets, which might<br>=C2=A0 =C2=A0 =C2=A0 be obtained by DP=
I (Deep Packet Inspection) within the transit<br>=C2=A0 =C2=A0 =C2=A0 netwo=
rk if the packets are unencrypted, or they could be analyzed<br>=C2=A0 =C2=
=A0 =C2=A0 by the source or destination nodes.</blockquote><br>Why =E2=80=
=98user content=E2=80=99? Why not =E2=80=98content=E2=80=99? There are init=
iatives for DPI over encrypted traffic:<br><br><a href=3D"http://iot.stanfo=
rd.edu/pubs/sherry-blindbox-sigcomm15.pdf">http://iot.stanford.edu/pubs/she=
rry-blindbox-sigcomm15.pdf</a><br><br><blockquote class=3D"gmail_quote" sty=
le=3D"margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:soli=
d;border-left-color:rgb(204,204,204);padding-left:1ex">3.3.=C2=A0 Architect=
ure Considerations</blockquote><blockquote class=3D"gmail_quote" style=3D"m=
argin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;borde=
r-left-color:rgb(204,204,204);padding-left:1ex"><br>=C2=A0 =C2=A0Offline &a=
mp; online learning<br><br>=C2=A0 =C2=A0 =C2=A0 * =C2=A0Co-located mode: tr=
aining (offline, based on historic data) and<br>=C2=A0 =C2=A0 =C2=A0 =C2=A0=
 =C2=A0prediction (online, based on real-time data) are both done<br>=C2=A0=
 =C2=A0 =C2=A0 =C2=A0 =C2=A0within the same entity.=C2=A0 The entity could =
be a central<br>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0repository or a specific =
node.<br><br>=C2=A0 =C2=A0 =C2=A0 * =C2=A0De-coupled mode: training is done=
 in the central repository,<br>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0and predic=
tion is made by the routers/switches/firewalls or<br>=C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0other devices that directly process the network traffic.<br><b=
r><br>=C2=A0 =C2=A0Central learning &amp; distributed learning =C2=A0Centra=
l learning means the<br>=C2=A0 =C2=A0 =C2=A0 learning process is done at a =
single entity, which is either a<br>=C2=A0 =C2=A0 =C2=A0 central repository=
 or a node.=C2=A0 Distributed learning refer to<br>=C2=A0 =C2=A0 =C2=A0 ens=
emble learning that multiple entities do the learning<br>=C2=A0 =C2=A0 =C2=
=A0 simultaneously and ensemble the results together to sort out a<br>=C2=
=A0 =C2=A0 =C2=A0 final results.=C2=A0 Since network devices are naturally =
distributed,<br>=C2=A0 =C2=A0 =C2=A0 it could be foreseen that ensemble lea=
rning is a good approach for<br>=C2=A0 =C2=A0 =C2=A0 a certain of use cases=
.</blockquote><br>This last paragraph is obscure and unclear to me, what is=
 the relation between this an offline/online learning? I think that this co=
uld be simplified by stating than learning can be offline and applied onlin=
e, or directly online, either on a network device or in a centralized entit=
y.<br><br><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8=
ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,=
204,204);padding-left:1ex">3.4.=C2=A0 Closed Control Loop<br><br>=C2=A0 =C2=
=A0The prediction made by machine learning mechanism could be directly<br>=
=C2=A0 =C2=A0used on manipulating the network traffic, or other relevant ac=
tions,<br>=C2=A0 =C2=A0such as changing the device configuration, etc.<br><=
br>=C2=A0 =C2=A0However, as the introduction section said, this kind of uti=
lization<br>=C2=A0 =C2=A0might be suitable only for a small set of the use =
cases, due to the<br>=C2=A0 =C2=A0limited accuracy of machine learning tech=
nologies.=C2=A0 Besides, some<br>=C2=A0 =C2=A0critical usages simply cannot=
 tolerate any false decision.</blockquote><br>See my comment above, this pa=
ragraph suggests that closed control-loop is a very long-term application, =
why? ML-based closed control-loops are currently applied in many fields (e.=
g, computer vision, self-driving cars, etc). Some people may argue that suc=
h applications are more complex and/or critical than network traffic analys=
is.<br><br>[snip]<br><br><blockquote class=3D"gmail_quote" style=3D"margin:=
0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left=
-color:rgb(204,204,204);padding-left:1ex">4.1.=C2=A0 HTTPS Traffic Classifi=
cation</blockquote><br>[snip]<br><br><blockquote class=3D"gmail_quote" styl=
e=3D"margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid=
;border-left-color:rgb(204,204,204);padding-left:1ex">=C2=A0 =C2=A0As a con=
crete example, Google, Facebook or Amazon are service<br>=C2=A0 =C2=A0provi=
ders while maps, drive, gmail are services of Google.=C2=A0 To<br>=C2=A0 =
=C2=A0identify them when they are accessed by a user, IP addresses and DNS<=
br>=C2=A0 =C2=A0(Domain Name System) names based identification is not reli=
able as<br>=C2=A0 =C2=A0the users can relies on intermediates to respective=
ly serve as proxy<br>=C2=A0 =C2=A0or resolve DNS requests.=C2=A0 The SNI (S=
erver Name Indication) [RFC5246]<br>=C2=A0 =C2=A0is an extension of HTTPS w=
hich is indicated by the user when<br>=C2=A0 =C2=A0initiating the TLS hands=
hake (Client Hello).=C2=A0 SNI actually contains<br>=C2=A0 =C2=A0the hostna=
me to which the request is addressed.=C2=A0 Such an hostname is<br>=C2=A0 =
=C2=A0significative of the service and service provider name.=C2=A0 However=
, SNI<br>=C2=A0 =C2=A0is an optional field and can be easily forged to circ=
umvent HTTPS<br>=C2=A0 =C2=A0filtering without impacting service use [bypas=
ssni].=C2=A0 More advanced<br>=C2=A0 =C2=A0mechanisms are hence necessary t=
o improve the robustness of<br>=C2=A0 =C2=A0identification even in the case=
 of non collaborative users.</blockquote><br>I suggest being vendor-agnosti=
c in the examples, the specific examples do not improve the draft by any me=
ans.<br><br>[snip]<br><blockquote class=3D"gmail_quote" style=3D"margin:0px=
 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-co=
lor:rgb(204,204,204);padding-left:1ex"><br><br>=C2=A0 =C2=A0 =C2=A0HTTPS Co=
nnection<br>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0+<br>=C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0|(1)<br>=C2=A0 =C2=A0+-------v------+<br>=C2=A0 =C2=
=A0|TLS Connection|<br>=C2=A0 =C2=A0|Reconstruction|<br>=C2=A0 =C2=A0+-----=
--+------+<br>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0|(2)<br>=C2=A0 =C2=
=A0+-------v------+ =C2=A0 =C2=A0(3&#39;) =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(4&#39;)<br>=C2=A0 =C2=A0| =C2=A0Feat=
ures =C2=A0 =C2=A0+-------------+----------------------------+<br>=C2=A0 =
=C2=A0| =C2=A0Extraction =C2=A0| =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
| =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0|<br>=C2=A0 =C2=A0+-------+------+ =C2=A0 =C2=A0 +-=
------v---------+ =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 +----v----+<br>=
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0| =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0|Service Provider +-------------&gt;Services |<br>=C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0|(3) =C2=A0 =C2=A0 =C2=A0 =C2=A0 |L1 model =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 | =C2=A0 Load =C2=A0 =C2=A0 =C2=A0|L2 model |<br>=
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0| =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0+-------^---------+ =C2=A0 services =C2=A0+----^----+<br>=C2=A0 =
=C2=A0+-------v------+ =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 | =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 model X =C2=A0 =C2=A0 =C2=A0 =C2=A0|<br>=
=C2=A0 =C2=A0|SNI Labelling | =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 +--=
--------------------------+<br>=C2=A0 =C2=A0+-------+------+ =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 |(5)<=
br>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0| =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0+-----------------------------------------+<br>=C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0+------------&gt; =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0Training and =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 |<br>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0(4) =C2=A0| =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Models bu=
ilding =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0|<br>=C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 +-----------=
------------------------------+<br><br>=C2=A0 =C2=A0Two-levels HTTPS traffi=
c classification<br><br>=C2=A0 =C2=A0In figure above, step(1) consists in r=
econstructing the HTTPS<br>=C2=A0 =C2=A0connection and retrieving packets o=
n top of which the following<br>=C2=A0 =C2=A0metrics are observed (2):<br><=
br>=C2=A0 =C2=A0o =C2=A0Inter Arrival Time<br><br>=C2=A0 =C2=A0o =C2=A0Pack=
et size<br><br>=C2=A0 =C2=A0o =C2=A0Encrypted data size: this feature has t=
he advantage to be strongly<br>=C2=A0 =C2=A0 =C2=A0 related to the service =
accessed instead of the packet size which<br>=C2=A0 =C2=A0 =C2=A0 is biased=
 by other lower layer headers<br><br>=C2=A0 =C2=A0Based on these values, ag=
gregated features are computed: average,<br>=C2=A0 =C2=A0minimum, maximum, =
25th percentile, median, 75th percentile.<br><br></blockquote><div>=C2=A0</=
div>Does the authors see value on listing all the traffic features in an AN=
NEX?<br><br><br>[snip]<br><br><br><blockquote class=3D"gmail_quote" style=
=3D"margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;=
border-left-color:rgb(204,204,204);padding-left:1ex">4.2.=C2=A0 Malicious D=
omains: Automatic Detection with DNS Traffic Analysis</blockquote><br>[snip=
]<br><br><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8e=
x;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,2=
04,204);padding-left:1ex"><br>=C2=A0 =C2=A0As a result, in an automated fas=
hion, a large variety of suspicious<br>=C2=A0 =C2=A0domains can be detected=
, including phishing, malware, but also other<br>=C2=A0 =C2=A0types, such a=
s fake pharmaceutical shops as well as counterfeit<br>=C2=A0 =C2=A0sneakers=
.=C2=A0 In this particular case, the responsible registrars are<br>=C2=A0 =
=C2=A0notified in this pilot study about these websites.=C2=A0 Ultimately, =
it<br>=C2=A0 =C2=A0allows these websites to be taken down, minimizing the p=
otential<br>=C2=A0 =C2=A0number of victims.</blockquote><br><br>Can this us=
e-cases be elaborated a little bit further? Which are features used? And ho=
w ML is applied? Which algorithm? Where does the training set come from?<br=
><br><br><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8e=
x;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,2=
04,204);padding-left:1ex">4.3.=C2=A0 Machine-learning based Policy Derivati=
on and Evaluation in<br>=C2=A0 =C2=A0 =C2=A0 Broadband Networks</blockquote=
><br><br>[snip]<br><br><blockquote class=3D"gmail_quote" style=3D"margin:0p=
x 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-c=
olor:rgb(204,204,204);padding-left:1ex">=C2=A0 =C2=A0It is evident that mac=
hine learning can have significant importance<br>=C2=A0 =C2=A0in policy der=
ivation and evaluation in broadband networks, especially<br>=C2=A0 =C2=A0to=
wards in 5G infrastructures which will be complex, heterogeneous<br>=C2=A0 =
=C2=A0and need to accommodate multi-services ranging from mobile broadband<=
br>=C2=A0 =C2=A0to massive machine type, mission critical and vehicular<br>=
=C2=A0 =C2=A0communications.</blockquote><br>This use-cases relates to a ve=
ry relevant area in ML (5G), but I don=C2=B4t see how ML is being applied? =
Is ML generating knowledge (unsupervised) to offer recommendations to the n=
etadmins? Is ML used to estimate the performance of a given policy?<br><br>=
<blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-=
left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);p=
adding-left:1ex">4.4.=C2=A0 Traffic Anomaly Detection in the Router</blockq=
uote><br>[snip]</div><div><br><blockquote class=3D"gmail_quote" style=3D"ma=
rgin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border=
-left-color:rgb(204,204,204);padding-left:1ex">=C2=A0 =C2=A0Besides wavelet=
 analysis, there might be more techniques to explore,<br>=C2=A0 =C2=A0such =
as correlation analysis of traffic anomaly events among multiple<br>=C2=A0 =
=C2=A0devices.</blockquote><br>In some cases and beyond correlation with ti=
me, ML can be applied to correlate traffic with external information (weath=
er, calendar, etc) that may play an important role in the traffic profile.<=
br><br><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;=
border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204=
,204);padding-left:1ex">4.5.=C2=A0 Applications of Machine Learning to Flow=
 Monitoring</blockquote><blockquote class=3D"gmail_quote" style=3D"margin:0=
px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-=
color:rgb(204,204,204);padding-left:1ex"><br>=C2=A0 =C2=A0A commercial clou=
d-based flow monitoring service from Network<br>=C2=A0 =C2=A0Polygraph [pol=
ygraph] has used Machine Learning analysis as a cost-<br>=C2=A0 =C2=A0effec=
tive alternative to DPI for traffic classification, which<br>=C2=A0 =C2=A0i=
dentifies the application responsible for each network traffic flow.</block=
quote><br>I suggest to make the use-case neutral.</div><div><br><blockquote=
 class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left-width:=
1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left=
:1ex">=C2=A0 =C2=A0The target objective is to progressively reduce the depe=
ndence on DPI<br>=C2=A0 =C2=A0technologies, which are expensive, difficult =
to deploy, not scalable,<br>=C2=A0 =C2=A0and not robust against encryption,=
 in favor of flow-based machine<br>=C2=A0 =C2=A0learning approaches that ar=
e more cost-effective and can be easily<br>=C2=A0 =C2=A0offered as a cloud =
service.=C2=A0 In this direction, some research<br>=C2=A0 =C2=A0challenges =
include the classification of web services and CDN traffic<br>=C2=A0 =C2=A0=
from flow-based measurements, and the combination of multiple ground<br>=C2=
=A0 =C2=A0truths obtained from vantage points in different networks.<br></b=
lockquote><br>Does the authors see value on listing all traffic features us=
ed in an ANNEX?<br><br><blockquote class=3D"gmail_quote" style=3D"margin:0p=
x 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-c=
olor:rgb(204,204,204);padding-left:1ex">5.=C2=A0 Security Considerations<br=
><br>=C2=A0 =C2=A0This document is focused on applying machine learning in =
network,<br>=C2=A0 =C2=A0including of course applying machine learning in n=
etwork security, on<br>=C2=A0 =C2=A0higher-layer concepts.=C2=A0 Therefore,=
 it does not itself create any new<br>=C2=A0 =C2=A0security issues.</blockq=
uote><br>I second Brian=E2=80=99s comment, a brief discussion on privacy co=
ncerns for these very specific use-cases would be very valuable.<br><br><br=
><br></div></div>

--001a11443e3063bad00537c14f75--


From nobody Sun Jul 17 14:04:10 2016
Return-Path: <jerome.francois@inria.fr>
X-Original-To: nmlrg@ietfa.amsl.com
Delivered-To: nmlrg@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 92A3B12D0B6 for <nmlrg@ietfa.amsl.com>; Sun, 17 Jul 2016 14:04:08 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -8.186
X-Spam-Level: 
X-Spam-Status: No, score=-8.186 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_HI=-5, RP_MATCHES_RCVD=-1.287] autolearn=ham autolearn_force=no
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id BuX5sLRhSYIh for <nmlrg@ietfa.amsl.com>; Sun, 17 Jul 2016 14:04:06 -0700 (PDT)
Received: from mail2-relais-roc.national.inria.fr (mail2-relais-roc.national.inria.fr [192.134.164.83]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 39D0612B04D for <nmlrg@irtf.org>; Sun, 17 Jul 2016 14:04:06 -0700 (PDT)
X-IronPort-AV: E=Sophos;i="5.28,380,1464645600";  d="scan'208,217";a="226939911"
Received: from unknown (HELO [10.141.69.144]) ([46.189.28.189]) by mail2-relais-roc.national.inria.fr with ESMTP/TLS/DHE-RSA-AES128-SHA; 17 Jul 2016 23:04:03 +0200
Message-ID: <578BF2BD.1060900@inria.fr>
Date: Sun, 17 Jul 2016 23:03:57 +0200
From: =?UTF-8?B?SsOpcsO0bWUgRnJhbsOnb2lz?= <jerome.francois@inria.fr>
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.5.0
MIME-Version: 1.0
To: Albert Cabellos <albert.cabellos@gmail.com>,  draft-jiang-nmlrg-traffic-machine-learning@ietf.org, nmlrg@irtf.org
References: <CAGE_QewtGRL58K-XLrFOE9a-vMjJEV8v5sthMQ3OeHdzAOKK8A@mail.gmail.com>
In-Reply-To: <CAGE_QewtGRL58K-XLrFOE9a-vMjJEV8v5sthMQ3OeHdzAOKK8A@mail.gmail.com>
Content-Type: multipart/alternative; boundary="------------050509060708010607030708"
Archived-At: <https://mailarchive.ietf.org/arch/msg/nmlrg/IiRqdG7pfmB4qeMs32xWyo8fGzs>
Subject: Re: [Nmlrg] Review for draft-jiang-nmlrg-traffic-machine-learning-00.txt
X-BeenThere: nmlrg@irtf.org
X-Mailman-Version: 2.1.17
Precedence: list
List-Id: Network Machine Learning Research Group <nmlrg.irtf.org>
List-Unsubscribe: <https://www.irtf.org/mailman/options/nmlrg>, <mailto:nmlrg-request@irtf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/nmlrg/>
List-Post: <mailto:nmlrg@irtf.org>
List-Help: <mailto:nmlrg-request@irtf.org?subject=help>
List-Subscribe: <https://www.irtf.org/mailman/listinfo/nmlrg>, <mailto:nmlrg-request@irtf.org?subject=subscribe>
X-List-Received-Date: Sun, 17 Jul 2016 21:04:08 -0000

This is a multi-part message in MIME format.
--------------050509060708010607030708
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

Hi Albert,

>     4.1.  HTTPS Traffic Classification
>
>
> [snip]
>
>        As a concrete example, Google, Facebook or Amazon are service
>        providers while maps, drive, gmail are services of Google.  To
>        identify them when they are accessed by a user, IP addresses
>     and DNS
>        (Domain Name System) names based identification is not reliable =
as
>        the users can relies on intermediates to respectively serve as
>     proxy
>        or resolve DNS requests.  The SNI (Server Name Indication)
>     [RFC5246]
>        is an extension of HTTPS which is indicated by the user when
>        initiating the TLS handshake (Client Hello).  SNI actually conta=
ins
>        the hostname to which the request is addressed.  Such an
>     hostname is
>        significative of the service and service provider name.=20
>     However, SNI
>        is an optional field and can be easily forged to circumvent HTTP=
S
>        filtering without impacting service use [bypasssni].  More advan=
ced
>        mechanisms are hence necessary to improve the robustness of
>        identification even in the case of non collaborative users.
>
>
> I suggest being vendor-agnostic in the examples, the specific examples
> do not improve the draft by any means.
I guess that the examples helps to understand what we mean by service
provider and service, i.e. to illustrate that having two levels is
something common nowadays.
>
> [snip]
>
>
>
>          HTTPS Connection
>                +
>                |(1)
>        +-------v------+
>        |TLS Connection|
>        |Reconstruction|
>        +-------+------+
>                |(2)
>        +-------v------+    (3')                    (4')
>        |  Features    +-------------+----------------------------+
>        |  Extraction  |             |                            |
>        +-------+------+     +-------v---------+             +----v----+=

>                |            |Service Provider +------------->Services |=

>                |(3)         |L1 model         |   Load      |L2 model |=

>                |            +-------^---------+   services  +----^----+=

>        +-------v------+             |             model X        |
>        |SNI Labelling |             +----------------------------+
>        +-------+------+                         |(5)
>                |            +-----------------------------------------+=

>                +------------>              Training and               |=

>                        (4)  |              Models building            |=

>                             +-----------------------------------------+=

>
>        Two-levels HTTPS traffic classification
>
>        In figure above, step(1) consists in reconstructing the HTTPS
>        connection and retrieving packets on top of which the following
>        metrics are observed (2):
>
>        o  Inter Arrival Time
>
>        o  Packet size
>
>        o  Encrypted data size: this feature has the advantage to be
>     strongly
>           related to the service accessed instead of the packet size wh=
ich
>           is biased by other lower layer headers
>
>        Based on these values, aggregated features are computed: average=
,
>        minimum, maximum, 25th percentile, median, 75th percentile.
>
> =20
> Does the authors see value on listing all the traffic features in an
> ANNEX?
>
All can be found in the referenced paper and those that contribute to a
good classification
are the ones given in the draft. So, as the author, I would say "no".

Best regards,
J=C3=A9r=C3=B4me

--------------050509060708010607030708
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 8bit

<html>
  <head>
    <meta content="text/html; charset=utf-8" http-equiv="Content-Type">
  </head>
  <body text="#000000" bgcolor="#FFFFFF">
    Hi Albert,<br>
    <br>
    <blockquote
cite="mid:CAGE_QewtGRL58K-XLrFOE9a-vMjJEV8v5sthMQ3OeHdzAOKK8A@mail.gmail.com"
      type="cite">
      <div dir="ltr">
        <div>
          <blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">4.1. 
            HTTPS Traffic Classification</blockquote>
          <br>
          [snip]<br>
          <br>
          <blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"> 
             As a concrete example, Google, Facebook or Amazon are
            service<br>
               providers while maps, drive, gmail are services of
            Google.  To<br>
               identify them when they are accessed by a user, IP
            addresses and DNS<br>
               (Domain Name System) names based identification is not
            reliable as<br>
               the users can relies on intermediates to respectively
            serve as proxy<br>
               or resolve DNS requests.  The SNI (Server Name
            Indication) [RFC5246]<br>
               is an extension of HTTPS which is indicated by the user
            when<br>
               initiating the TLS handshake (Client Hello).  SNI
            actually contains<br>
               the hostname to which the request is addressed.  Such an
            hostname is<br>
               significative of the service and service provider name. 
            However, SNI<br>
               is an optional field and can be easily forged to
            circumvent HTTPS<br>
               filtering without impacting service use [bypasssni]. 
            More advanced<br>
               mechanisms are hence necessary to improve the robustness
            of<br>
               identification even in the case of non collaborative
            users.</blockquote>
          <br>
          I suggest being vendor-agnostic in the examples, the specific
          examples do not improve the draft by any means.<br>
        </div>
      </div>
    </blockquote>
    I guess that the examples helps to understand what we mean by
    service provider and service, i.e. to illustrate that having two
    levels is something common nowadays.<br>
    <blockquote
cite="mid:CAGE_QewtGRL58K-XLrFOE9a-vMjJEV8v5sthMQ3OeHdzAOKK8A@mail.gmail.com"
      type="cite">
      <div dir="ltr">
        <div><br>
          [snip]<br>
          <blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><br>
            <br>
                 HTTPS Connection<br>
                       +<br>
                       |(1)<br>
               +-------v------+<br>
               |TLS Connection|<br>
               |Reconstruction|<br>
               +-------+------+<br>
                       |(2)<br>
               +-------v------+    (3')                    (4')<br>
               |  Features  
             +-------------+----------------------------+<br>
               |  Extraction  |             |                          
             |<br>
               +-------+------+     +-------v---------+            
            +----v----+<br>
                       |            |Service Provider
            +-------------&gt;Services |<br>
                       |(3)         |L1 model         |   Load      |L2
            model |<br>
                       |            +-------^---------+   services
             +----^----+<br>
               +-------v------+             |             model X      
             |<br>
               |SNI Labelling |            
            +----------------------------+<br>
               +-------+------+                         |(5)<br>
                       |          
             +-----------------------------------------+<br>
                       +------------&gt;              Training and      
                    |<br>
                               (4)  |              Models building      
                 |<br>
                                   
            +-----------------------------------------+<br>
            <br>
               Two-levels HTTPS traffic classification<br>
            <br>
               In figure above, step(1) consists in reconstructing the
            HTTPS<br>
               connection and retrieving packets on top of which the
            following<br>
               metrics are observed (2):<br>
            <br>
               o  Inter Arrival Time<br>
            <br>
               o  Packet size<br>
            <br>
               o  Encrypted data size: this feature has the advantage to
            be strongly<br>
                  related to the service accessed instead of the packet
            size which<br>
                  is biased by other lower layer headers<br>
            <br>
               Based on these values, aggregated features are computed:
            average,<br>
               minimum, maximum, 25th percentile, median, 75th
            percentile.<br>
            <br>
          </blockquote>
          <div> </div>
          Does the authors see value on listing all the traffic features
          in an ANNEX?<br>
          <br>
        </div>
      </div>
    </blockquote>
    All can be found in the referenced paper and those that contribute
    to a good classification <br>
    are the ones given in the draft. So, as the author, I would say
    "no".<br>
    <br>
    Best regards,<br>
    Jérôme<br>
  </body>
</html>

--------------050509060708010607030708--


From nobody Mon Jul 18 04:33:56 2016
Return-Path: <albert.cabellos@gmail.com>
X-Original-To: nmlrg@ietfa.amsl.com
Delivered-To: nmlrg@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 136E112D8E1 for <nmlrg@ietfa.amsl.com>; Mon, 18 Jul 2016 04:33:55 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.699
X-Spam-Level: 
X-Spam-Status: No, score=-2.699 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-0.7, SPF_PASS=-0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Jd9onJPWVtMz for <nmlrg@ietfa.amsl.com>; Mon, 18 Jul 2016 04:33:53 -0700 (PDT)
Received: from mail-wm0-x22b.google.com (mail-wm0-x22b.google.com [IPv6:2a00:1450:400c:c09::22b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 415AC12D8F7 for <nmlrg@irtf.org>; Mon, 18 Jul 2016 04:33:51 -0700 (PDT)
Received: by mail-wm0-x22b.google.com with SMTP id i5so112188770wmg.0 for <nmlrg@irtf.org>; Mon, 18 Jul 2016 04:33:51 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113;  h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=UxuHaaZ28vseobFT7M/MOsKeVrs+xpA8qESuQ6UAzzM=; b=C4C9UyIQqUDJO5TnsDc9Bt6z87dxY10kx6sC/bQKpWHq73cQaV7aO8VZaKjmG1wCai dKH/KhV4JogLba2WcbomlvmKbLwNXgEjk8vR7PXq3vSq8+pqbBiwB8/yk2kgemr6BEB4 t02/inIjWNNJ6nOvkUOWHakNZ7rlWvKU4TyZGa+g7irRshTWV+aH/DC3emXn1JaALwS3 aYxCPaCf+VkRIdfHN3B+KqNhQEtIWqEw6hA5KAFEMGeIDZ7P3Z/EAcCqE37jrI6ByOKq dJpkPowqcyiA6qAva/56KmoOAysVbvO15auo2dOHtnzfWllT4CQWedvWIZbCUx2i3OfB a9Vg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=UxuHaaZ28vseobFT7M/MOsKeVrs+xpA8qESuQ6UAzzM=; b=URxnyY1uNfM527NQbj8WCg/JOPElrPPrhi/mszR1rFHyWoIy3zsm0XvzUBVdwSq4LQ aeH8FRXeun7g32Vqj0ia/HVHbBCuNXfy8KnsM1HVgKmoaghNZlRApydXZCCaDWM1bKZV IV4ksR6BHkUg7HaHazU05uydCjTAcDOcTizvOHa+EVdYDJfVqdZ5ya5JRbyRm3TXJzB8 uOOTh+nYGgktgLgdm8Sgm81uPHEPShqzc5ag8Dtxgo43QbwWaCfPFAzPPYlhmL/E4U9+ GReM3DU18qAz2yTqwjglF+Q8bqLI/fO7GIy2DaSvy7eDXXRwvWqIHUFeD1F6jGir6AG5 g/LQ==
X-Gm-Message-State: ALyK8tKbWmoUAglMIcHpb1nmIYx4mfegsNtOSoysZOzBMtVr/OFqu+LzGJ4W45y1Cenz6OUbRKGwnv2BEacYBQ==
X-Received: by 10.28.154.21 with SMTP id c21mr38512451wme.63.1468841629745; Mon, 18 Jul 2016 04:33:49 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.28.56.131 with HTTP; Mon, 18 Jul 2016 04:33:49 -0700 (PDT)
In-Reply-To: <578BF2BD.1060900@inria.fr>
References: <CAGE_QewtGRL58K-XLrFOE9a-vMjJEV8v5sthMQ3OeHdzAOKK8A@mail.gmail.com> <578BF2BD.1060900@inria.fr>
From: Albert Cabellos <albert.cabellos@gmail.com>
Date: Mon, 18 Jul 2016 13:33:49 +0200
Message-ID: <CAGE_QewKEGcLqb1XD-h98_sqHxxFAFzt22A-jDG-bWNN0ATQXA@mail.gmail.com>
To: =?UTF-8?B?SsOpcsO0bWUgRnJhbsOnb2lz?= <jerome.francois@inria.fr>
Content-Type: multipart/alternative; boundary=001a114bde2c9289810537e757d9
Archived-At: <https://mailarchive.ietf.org/arch/msg/nmlrg/Yzcm2krjr8Zamv0PQdtVBmJfhWE>
Cc: nmlrg@irtf.org, draft-jiang-nmlrg-traffic-machine-learning@ietf.org
Subject: Re: [Nmlrg] Review for draft-jiang-nmlrg-traffic-machine-learning-00.txt
X-BeenThere: nmlrg@irtf.org
X-Mailman-Version: 2.1.17
Precedence: list
List-Id: Network Machine Learning Research Group <nmlrg.irtf.org>
List-Unsubscribe: <https://www.irtf.org/mailman/options/nmlrg>, <mailto:nmlrg-request@irtf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/nmlrg/>
List-Post: <mailto:nmlrg@irtf.org>
List-Help: <mailto:nmlrg-request@irtf.org?subject=help>
List-Subscribe: <https://www.irtf.org/mailman/listinfo/nmlrg>, <mailto:nmlrg-request@irtf.org?subject=subscribe>
X-List-Received-Date: Mon, 18 Jul 2016 11:33:55 -0000

--001a114bde2c9289810537e757d9
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Hi J=C3=A9r=C3=B4me

Please see inline:

On Sun, Jul 17, 2016 at 11:03 PM, J=C3=A9r=C3=B4me Fran=C3=A7ois <jerome.fr=
ancois@inria.fr>
wrote:

> Hi Albert,
>
> 4.1.  HTTPS Traffic Classification
>
>
> [snip]
>
>    As a concrete example, Google, Facebook or Amazon are service
>>    providers while maps, drive, gmail are services of Google.  To
>>    identify them when they are accessed by a user, IP addresses and DNS
>>    (Domain Name System) names based identification is not reliable as
>>    the users can relies on intermediates to respectively serve as proxy
>>    or resolve DNS requests.  The SNI (Server Name Indication) [RFC5246]
>>    is an extension of HTTPS which is indicated by the user when
>>    initiating the TLS handshake (Client Hello).  SNI actually contains
>>    the hostname to which the request is addressed.  Such an hostname is
>>    significative of the service and service provider name.  However, SNI
>>    is an optional field and can be easily forged to circumvent HTTPS
>>    filtering without impacting service use [bypasssni].  More advanced
>>    mechanisms are hence necessary to improve the robustness of
>>    identification even in the case of non collaborative users.
>
>
> I suggest being vendor-agnostic in the examples, the specific examples do
> not improve the draft by any means.
>
> I guess that the examples helps to understand what we mean by service
> provider and service, i.e. to illustrate that having two levels is
> something common nowadays.
>
>
I think that everyone understands this, I suggest to have a vendor-neutral
document.

>
> [snip]
>
>>
>>
>>      HTTPS Connection
>>            +
>>            |(1)
>>    +-------v------+
>>    |TLS Connection|
>>    |Reconstruction|
>>    +-------+------+
>>            |(2)
>>    +-------v------+    (3')                    (4')
>>    |  Features    +-------------+----------------------------+
>>    |  Extraction  |             |                            |
>>    +-------+------+     +-------v---------+             +----v----+
>>            |            |Service Provider +------------->Services |
>>            |(3)         |L1 model         |   Load      |L2 model |
>>            |            +-------^---------+   services  +----^----+
>>    +-------v------+             |             model X        |
>>    |SNI Labelling |             +----------------------------+
>>    +-------+------+                         |(5)
>>            |            +-----------------------------------------+
>>            +------------>              Training and               |
>>                    (4)  |              Models building            |
>>                         +-----------------------------------------+
>>
>>    Two-levels HTTPS traffic classification
>>
>>    In figure above, step(1) consists in reconstructing the HTTPS
>>    connection and retrieving packets on top of which the following
>>    metrics are observed (2):
>>
>>    o  Inter Arrival Time
>>
>>    o  Packet size
>>
>>    o  Encrypted data size: this feature has the advantage to be strongly
>>       related to the service accessed instead of the packet size which
>>       is biased by other lower layer headers
>>
>>    Based on these values, aggregated features are computed: average,
>>    minimum, maximum, 25th percentile, median, 75th percentile.
>>
>>
> Does the authors see value on listing all the traffic features in an ANNE=
X?
>
> All can be found in the referenced paper and those that contribute to a
> good classification
> are the ones given in the draft. So, as the author, I would say "no".
>
>
Thanks! In any case, can the authors try to put together the features that
they are using and see if there are any common ones? The document should be
-to a reasonable extent- self-contained.

cheers

Albert


> Best regards,
> J=C3=A9r=C3=B4me
>

--001a114bde2c9289810537e757d9
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hi J=C3=A9r=C3=B4me<div><br></div><div>Please see inline:<=
/div><div class=3D"gmail_extra"><br><div class=3D"gmail_quote">On Sun, Jul =
17, 2016 at 11:03 PM, J=C3=A9r=C3=B4me Fran=C3=A7ois <span dir=3D"ltr">&lt;=
<a href=3D"mailto:jerome.francois@inria.fr" target=3D"_blank">jerome.franco=
is@inria.fr</a>&gt;</span> wrote:<br><blockquote class=3D"gmail_quote" styl=
e=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
 =20
   =20
 =20
  <div text=3D"#000000" bgcolor=3D"#FFFFFF">
    Hi Albert,<span class=3D""><br>
    <br>
    <blockquote type=3D"cite">
      <div dir=3D"ltr">
        <div>
          <blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8=
ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,=
204,204);padding-left:1ex">4.1.=C2=A0
            HTTPS Traffic Classification</blockquote>
          <br>
          [snip]<br>
          <br>
          <blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8=
ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,=
204,204);padding-left:1ex">=C2=A0
            =C2=A0As a concrete example, Google, Facebook or Amazon are
            service<br>
            =C2=A0 =C2=A0providers while maps, drive, gmail are services of
            Google.=C2=A0 To<br>
            =C2=A0 =C2=A0identify them when they are accessed by a user, IP
            addresses and DNS<br>
            =C2=A0 =C2=A0(Domain Name System) names based identification is=
 not
            reliable as<br>
            =C2=A0 =C2=A0the users can relies on intermediates to respectiv=
ely
            serve as proxy<br>
            =C2=A0 =C2=A0or resolve DNS requests.=C2=A0 The SNI (Server Nam=
e
            Indication) [RFC5246]<br>
            =C2=A0 =C2=A0is an extension of HTTPS which is indicated by the=
 user
            when<br>
            =C2=A0 =C2=A0initiating the TLS handshake (Client Hello).=C2=A0=
 SNI
            actually contains<br>
            =C2=A0 =C2=A0the hostname to which the request is addressed.=C2=
=A0 Such an
            hostname is<br>
            =C2=A0 =C2=A0significative of the service and service provider =
name.=C2=A0
            However, SNI<br>
            =C2=A0 =C2=A0is an optional field and can be easily forged to
            circumvent HTTPS<br>
            =C2=A0 =C2=A0filtering without impacting service use [bypasssni=
].=C2=A0
            More advanced<br>
            =C2=A0 =C2=A0mechanisms are hence necessary to improve the robu=
stness
            of<br>
            =C2=A0 =C2=A0identification even in the case of non collaborati=
ve
            users.</blockquote>
          <br>
          I suggest being vendor-agnostic in the examples, the specific
          examples do not improve the draft by any means.<br>
        </div>
      </div>
    </blockquote></span>
    I guess that the examples helps to understand what we mean by
    service provider and service, i.e. to illustrate that having two
    levels is something common nowadays.<div><div class=3D"h5"><br></div></=
div></div></blockquote><div><br></div><div>I think that everyone understand=
s this, I suggest to have a vendor-neutral document.=C2=A0</div><blockquote=
 class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc soli=
d;padding-left:1ex"><div text=3D"#000000" bgcolor=3D"#FFFFFF"><div><div cla=
ss=3D"h5">
    <blockquote type=3D"cite">
      <div dir=3D"ltr">
        <div><br>
          [snip]<br>
          <blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8=
ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,=
204,204);padding-left:1ex"><br>
            <br>
            =C2=A0 =C2=A0 =C2=A0HTTPS Connection<br>
            =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0+<br>
            =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0|(1)<br>
            =C2=A0 =C2=A0+-------v------+<br>
            =C2=A0 =C2=A0|TLS Connection|<br>
            =C2=A0 =C2=A0|Reconstruction|<br>
            =C2=A0 =C2=A0+-------+------+<br>
            =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0|(2)<br>
            =C2=A0 =C2=A0+-------v------+ =C2=A0 =C2=A0(3&#39;) =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(4&#39;)<br>
            =C2=A0 =C2=A0| =C2=A0Features =C2=A0
            =C2=A0+-------------+----------------------------+<br>
            =C2=A0 =C2=A0| =C2=A0Extraction =C2=A0| =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 | =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0
            =C2=A0|<br>
            =C2=A0 =C2=A0+-------+------+ =C2=A0 =C2=A0 +-------v---------+=
 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0
            +----v----+<br>
            =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0| =C2=A0 =C2=A0 =C2=A0=
 =C2=A0 =C2=A0 =C2=A0|Service Provider
            +-------------&gt;Services |<br>
            =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0|(3) =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 |L1 model =C2=A0 =C2=A0 =C2=A0 =C2=A0 | =C2=A0 Load =C2=A0 =C2=
=A0 =C2=A0|L2
            model |<br>
            =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0| =C2=A0 =C2=A0 =C2=A0=
 =C2=A0 =C2=A0 =C2=A0+-------^---------+ =C2=A0 services
            =C2=A0+----^----+<br>
            =C2=A0 =C2=A0+-------v------+ =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 | =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 model X =C2=A0 =C2=
=A0 =C2=A0
            =C2=A0|<br>
            =C2=A0 =C2=A0|SNI Labelling | =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0
            +----------------------------+<br>
            =C2=A0 =C2=A0+-------+------+ =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 |(5)<br>
            =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0| =C2=A0 =C2=A0 =C2=A0=
 =C2=A0 =C2=A0
            =C2=A0+-----------------------------------------+<br>
            =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0+------------&gt; =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Training and =C2=A0 =C2=A0 =C2=
=A0
            =C2=A0 =C2=A0 =C2=A0 =C2=A0 |<br>
            =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0(4) =C2=A0| =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Models bu=
ilding =C2=A0 =C2=A0 =C2=A0
            =C2=A0 =C2=A0 =C2=A0|<br>
            =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0
            +-----------------------------------------+<br>
            <br>
            =C2=A0 =C2=A0Two-levels HTTPS traffic classification<br>
            <br>
            =C2=A0 =C2=A0In figure above, step(1) consists in reconstructin=
g the
            HTTPS<br>
            =C2=A0 =C2=A0connection and retrieving packets on top of which =
the
            following<br>
            =C2=A0 =C2=A0metrics are observed (2):<br>
            <br>
            =C2=A0 =C2=A0o =C2=A0Inter Arrival Time<br>
            <br>
            =C2=A0 =C2=A0o =C2=A0Packet size<br>
            <br>
            =C2=A0 =C2=A0o =C2=A0Encrypted data size: this feature has the =
advantage to
            be strongly<br>
            =C2=A0 =C2=A0 =C2=A0 related to the service accessed instead of=
 the packet
            size which<br>
            =C2=A0 =C2=A0 =C2=A0 is biased by other lower layer headers<br>
            <br>
            =C2=A0 =C2=A0Based on these values, aggregated features are com=
puted:
            average,<br>
            =C2=A0 =C2=A0minimum, maximum, 25th percentile, median, 75th
            percentile.<br>
            <br>
          </blockquote>
          <div>=C2=A0</div>
          Does the authors see value on listing all the traffic features
          in an ANNEX?<br>
          <br>
        </div>
      </div>
    </blockquote></div></div>
    All can be found in the referenced paper and those that contribute
    to a good classification <br>
    are the ones given in the draft. So, as the author, I would say
    &quot;no&quot;.<br>
    <br></div></blockquote><div><br></div><div>Thanks! In any case, can the=
 authors try to put together the features that they are using and see if th=
ere are any common ones? The document should be -to a reasonable extent- se=
lf-contained.</div><div><br></div><div>cheers</div><div><br></div><div>Albe=
rt</div><div>=C2=A0</div><blockquote class=3D"gmail_quote" style=3D"margin:=
0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div text=3D"#00000=
0" bgcolor=3D"#FFFFFF">
    Best regards,<br>
    J=C3=A9r=C3=B4me<br>
  </div>

</blockquote></div><br></div></div>

--001a114bde2c9289810537e757d9--


From nobody Tue Jul 19 00:20:54 2016
Return-Path: <dmm@1-4-5.net>
X-Original-To: nmlrg@ietfa.amsl.com
Delivered-To: nmlrg@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 50B1A12D133 for <nmlrg@ietfa.amsl.com>; Tue, 19 Jul 2016 00:20:52 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.599
X-Spam-Level: 
X-Spam-Status: No, score=-2.599 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-0.7] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=1-4-5-net.20150623.gappssmtp.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Gr5gqs9pIPgY for <nmlrg@ietfa.amsl.com>; Tue, 19 Jul 2016 00:20:50 -0700 (PDT)
Received: from mail-it0-x235.google.com (mail-it0-x235.google.com [IPv6:2607:f8b0:4001:c0b::235]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 7D8E412D100 for <nmlrg@irtf.org>; Tue, 19 Jul 2016 00:20:50 -0700 (PDT)
Received: by mail-it0-x235.google.com with SMTP id u186so86134339ita.0 for <nmlrg@irtf.org>; Tue, 19 Jul 2016 00:20:50 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1-4-5-net.20150623.gappssmtp.com; s=20150623; h=mime-version:from:date:message-id:subject:to; bh=VQtBMkM5ImXWgRDmXIrU6uxuDzWH4X6DyZ7cD0fAino=; b=pgqh5rECgv2ZKxi6yisR1t3HoatFMCZ9IQOFR7KS92HIqguKmkwFudzfIWU/DGiMPB cI8IP3yPdzuepC78uX+Ug0WPgtwjt112VF59brQN1e2mvEslPhHsjRzHiK+KfC28mhvo U183WknoVvTf3MZIOl2VvtTyrvWmskhlcI1zjRowoXf9BpxjsEXzs+/MHQzHzaHNPHCC T4dgSvHbz0H2k3k3pw/a/+vCCO/AIbg7r2VAjzh/xFXgb3ahaJkTfOpud5R4fjA2kjVg x2bGMYvgEazNVokRz6LxplfXQC1ODTrb8krDQ2hr6BeNcGhmJYaTWeecx2ve9nfNtR7K soKg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=VQtBMkM5ImXWgRDmXIrU6uxuDzWH4X6DyZ7cD0fAino=; b=FFCFiF++Cq2+X6hea4qwRob0llAB7TfB655jNFjfYwcfmsec0DRaACIeF7UhjGJrWW XgZgAtyKZcS1hahuPKYqoSQscgwhxy7apODwCHCp/ZReYJ/D49VGIGofTWTKY+4y8zGU 6Fh+F0s4F0fCClbwmghwQ35JPXk1979+VweUcPaPdqDErz3WaEDPMF7ArMuanYjzTKLR CUcu8FB/pz65QmjAMuqURfTb87+z3vWhm+i5IfXtPsqa4+eEf8loZfEvLw63Te2P9u/B FaCsYBZKxwnBYq18DFENab8R1nBa6XXhMtVZ1pG4xsTHs0XadSlKKfDSxAOhw1hM9OcL w4ag==
X-Gm-Message-State: ALyK8tJWdG+FjzOn/vo9PAzu4N4uMIEtWMvILbY5RZpfhYkr9I4Yy/K1HkbxMVuj9U48jlagn4OcTqfDY8XlYw==
X-Received: by 10.36.227.13 with SMTP id d13mr2387962ith.18.1468912849507; Tue, 19 Jul 2016 00:20:49 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.64.227.203 with HTTP; Tue, 19 Jul 2016 00:20:48 -0700 (PDT)
X-Originating-IP: [31.133.143.218]
From: David Meyer <dmm@1-4-5.net>
Date: Tue, 19 Jul 2016 09:20:48 +0200
Message-ID: <CAHiKxWhXoa1WbqZFRqJ+fJiqQG64LtGT1g-EBBTpCy24OeTtOw@mail.gmail.com>
To: nmlrg@irtf.org
Content-Type: multipart/alternative; boundary=94eb2c111b8c9a0c220537f7eccc
Archived-At: <https://mailarchive.ietf.org/arch/msg/nmlrg/3gvyUUbWuB5IxHGTPnPVojVSNzE>
Subject: [Nmlrg] a few references I mentioned yesterday
X-BeenThere: nmlrg@irtf.org
X-Mailman-Version: 2.1.17
Precedence: list
List-Id: Network Machine Learning Research Group <nmlrg.irtf.org>
List-Unsubscribe: <https://www.irtf.org/mailman/options/nmlrg>, <mailto:nmlrg-request@irtf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/nmlrg/>
List-Post: <mailto:nmlrg@irtf.org>
List-Help: <mailto:nmlrg-request@irtf.org?subject=help>
List-Subscribe: <https://www.irtf.org/mailman/listinfo/nmlrg>, <mailto:nmlrg-request@irtf.org?subject=subscribe>
X-List-Received-Date: Tue, 19 Jul 2016 07:20:52 -0000

--94eb2c111b8c9a0c220537f7eccc
Content-Type: text/plain; charset=UTF-8

(i).   On adversarial images; The canonical reference is here:
http://arxiv.org/pdf/1312.6199v4.pdf. More recent work on Generative
Adversarial Nets (GANs) here:
https://papers.nips.cc/paper/5423-generative-adversarial-nets.pdf (GANs
have emerged as a new and promising technique for unsupervised training).

(ii).  On AlphaGo:
http://www.nature.com/nature/journal/v529/n7587/pdf/nature16961.pdf and
http://www.nature.com/news/google-ai-algorithm-masters-ancient-game-of-go-1.19234.
If you are interested in more on reinforcement learning see Sutton's book
(Reinforcement Learning: An Introduction; you can find it online from
several sources).

(iii).  On LDA and other statistical clustering techniques for
high-cardinality categorical data:
http://www.1-4-5.net/~dmm/ml/lda_intro.pdf (see also
http://www.1-4-5.net/~dmm/ml/)

Thx,

--dmm

--94eb2c111b8c9a0c220537f7eccc
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">(i). =C2=A0 On adversarial images; The canonical reference=
 is here:=C2=A0<a href=3D"http://arxiv.org/pdf/1312.6199v4.pdf">http://arxi=
v.org/pdf/1312.6199v4.pdf</a>. More recent work on Generative Adversarial N=
ets (GANs) here:=C2=A0<a href=3D"https://papers.nips.cc/paper/5423-generati=
ve-adversarial-nets.pdf">https://papers.nips.cc/paper/5423-generative-adver=
sarial-nets.pdf</a> (GANs have emerged as a new and promising technique for=
 unsupervised training).<div><br></div><div>(ii).=C2=A0 On AlphaGo: =C2=A0<=
a href=3D"http://www.nature.com/nature/journal/v529/n7587/pdf/nature16961.p=
df">http://www.nature.com/nature/journal/v529/n7587/pdf/nature16961.pdf</a>=
 and <a href=3D"http://www.nature.com/news/google-ai-algorithm-masters-anci=
ent-game-of-go-1.19234">http://www.nature.com/news/google-ai-algorithm-mast=
ers-ancient-game-of-go-1.19234</a>. If you are interested in more on reinfo=
rcement learning see Sutton&#39;s book (Reinforcement Learning:
An Introduction; you can find it online from several sources).</div><div><b=
r></div><div>(iii).=C2=A0 On LDA and other statistical clustering technique=
s for high-cardinality categorical data: <a href=3D"http://www.1-4-5.net/~d=
mm/ml/lda_intro.pdf">http://www.1-4-5.net/~dmm/ml/lda_intro.pdf</a> (see al=
so=C2=A0<a href=3D"http://www.1-4-5.net/~dmm/ml/">http://www.1-4-5.net/~dmm=
/ml/</a>)</div><div><br></div><div>Thx,</div><div><br></div><div>--dmm</div=
><div><br></div></div>

--94eb2c111b8c9a0c220537f7eccc--


From nobody Tue Jul 19 00:32:09 2016
Return-Path: <dmm@1-4-5.net>
X-Original-To: nmlrg@ietfa.amsl.com
Delivered-To: nmlrg@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id CFFD612B04F for <nmlrg@ietfa.amsl.com>; Tue, 19 Jul 2016 00:32:08 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.599
X-Spam-Level: 
X-Spam-Status: No, score=-2.599 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-0.7] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=1-4-5-net.20150623.gappssmtp.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id NfG0kCgxgqQr for <nmlrg@ietfa.amsl.com>; Tue, 19 Jul 2016 00:32:07 -0700 (PDT)
Received: from mail-it0-x22e.google.com (mail-it0-x22e.google.com [IPv6:2607:f8b0:4001:c0b::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 5C788127078 for <nmlrg@irtf.org>; Tue, 19 Jul 2016 00:32:07 -0700 (PDT)
Received: by mail-it0-x22e.google.com with SMTP id f6so12282295ith.0 for <nmlrg@irtf.org>; Tue, 19 Jul 2016 00:32:07 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1-4-5-net.20150623.gappssmtp.com; s=20150623; h=mime-version:from:date:message-id:subject:to; bh=AzDUzIBCnnsy3FPbTv8EPE8FTEwP2jYpxRtRmaVFZNw=; b=gWYsguYsKmA1rli2SfqhvZbUb5K7v+pyxWVznoXiZBDGjdhVlhyvP/Cu34GOGMt4ws /Eb9txXDtL0ahbwtDbI7KRCz4v23qtWJVhWG/CYxBq2LhChaWpW4z3Rusr30TvuXPZg8 90kzy3TBrASAUYfm5GrI+zHihjz6uNzdD0WzhbX4uof5nPcTF+cwC7JKOwHuwFIjwJBB SIu0nDa11SZZ6F5Y5gOlo4kfQEzohen8HJlsNlYwRP6ZvdKL0JGl75dir318n/aZB1mz y2B63KzmazROrn88zYmoxq4+ZXD8Co2SYhfZf07E2DwxFdNQKhvjR8IRZSx57Elcaxht X7nw==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=AzDUzIBCnnsy3FPbTv8EPE8FTEwP2jYpxRtRmaVFZNw=; b=jGAgZkLUiKLU/Mj1+45S/6LjQLU3G5AEC4MdWY4OrtI33Mb0h/PPO5ppyna5V7u7Ti e6d8SFKhmI5AcmizXSU4dlvnpHu7z783fhbJZ37S9Og6MLFM/TgWeW2oD33qsm+4oAmS 5hAsGgswYjUJ/mLNM0s0vzinlnlWt13bksN017fmtUZHPgswnCxP3oZjjfvai/iqTJp4 k7ZkrdUH/kgKrML+X2S5qfR8agi+XmG71i2pMFHawGxU3QtN+7AWJ26UUaAbu/0MN5WZ JdMN7A03JcH2x/IWUrhHmL8Pxfd3Wf0SQfLTFk8uu0q8St1aDa9kChJp2viBchDlEQ1N gH3w==
X-Gm-Message-State: ALyK8tJevIRZv4Xeih+C0hyea9X1/A51+2ublpHPh8hW801b3fELSpdrxQEEhGDwl1vsr9rQULtP8wxUwKF28A==
X-Received: by 10.36.227.13 with SMTP id d13mr2431056ith.18.1468913526619; Tue, 19 Jul 2016 00:32:06 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.64.227.203 with HTTP; Tue, 19 Jul 2016 00:32:05 -0700 (PDT)
X-Originating-IP: [31.133.143.218]
From: David Meyer <dmm@1-4-5.net>
Date: Tue, 19 Jul 2016 09:32:05 +0200
Message-ID: <CAHiKxWjWyTBMLRFHnYvMCv3O1dTvW=FxUUoQf=0VuCmurCjynA@mail.gmail.com>
To: nmlrg@irtf.org
Content-Type: multipart/alternative; boundary=94eb2c111b8cf5c8200537f814e3
Archived-At: <https://mailarchive.ietf.org/arch/msg/nmlrg/RBU72_xXxsJyrqkBzE9WPB-bbZM>
Subject: [Nmlrg] a public data set we've been using
X-BeenThere: nmlrg@irtf.org
X-Mailman-Version: 2.1.17
Precedence: list
List-Id: Network Machine Learning Research Group <nmlrg.irtf.org>
List-Unsubscribe: <https://www.irtf.org/mailman/options/nmlrg>, <mailto:nmlrg-request@irtf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/nmlrg/>
List-Post: <mailto:nmlrg@irtf.org>
List-Help: <mailto:nmlrg-request@irtf.org?subject=help>
List-Subscribe: <https://www.irtf.org/mailman/listinfo/nmlrg>, <mailto:nmlrg-request@irtf.org?subject=subscribe>
X-List-Received-Date: Tue, 19 Jul 2016 07:32:09 -0000

--94eb2c111b8cf5c8200537f814e3
Content-Type: text/plain; charset=UTF-8

http://mcfp.weebly.com/the-ctu-13-dataset-a-labeled-dataset-with-botnet-normal-and-background-traffic.html

--dmm

--94eb2c111b8cf5c8200537f814e3
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><a href=3D"http://mcfp.weebly.com/the-ctu-13-dataset-a-lab=
eled-dataset-with-botnet-normal-and-background-traffic.html">http://mcfp.we=
ebly.com/the-ctu-13-dataset-a-labeled-dataset-with-botnet-normal-and-backgr=
ound-traffic.html</a><div><br></div><div>--dmm</div><div><br></div></div>

--94eb2c111b8cf5c8200537f814e3--


From nobody Tue Jul 19 01:41:48 2016
Return-Path: <jiangsheng@huawei.com>
X-Original-To: nmlrg@ietfa.amsl.com
Delivered-To: nmlrg@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id EB44812D105 for <nmlrg@ietfa.amsl.com>; Tue, 19 Jul 2016 01:41:45 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -5.488
X-Spam-Level: 
X-Spam-Status: No, score=-5.488 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RP_MATCHES_RCVD=-1.287, SPF_PASS=-0.001] autolearn=ham autolearn_force=no
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id QMp6aotfSFhs for <nmlrg@ietfa.amsl.com>; Tue, 19 Jul 2016 01:41:38 -0700 (PDT)
Received: from szxga01-in.huawei.com (szxga01-in.huawei.com [58.251.152.64]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 6956012DAB4 for <nmlrg@irtf.org>; Tue, 19 Jul 2016 01:41:28 -0700 (PDT)
Received: from 172.24.1.47 (EHLO NKGEML413-HUB.china.huawei.com) ([172.24.1.47]) by szxrg01-dlp.huawei.com (MOS 4.3.7-GA FastPath queued) with ESMTP id DNZ09736; Tue, 19 Jul 2016 16:41:23 +0800 (CST)
Received: from NKGEML515-MBX.china.huawei.com ([fe80::a54a:89d2:c471:ff]) by NKGEML413-HUB.china.huawei.com ([10.98.56.74]) with mapi id 14.03.0235.001; Tue, 19 Jul 2016 16:41:21 +0800
From: Sheng Jiang <jiangsheng@huawei.com>
To: "nmlrg@irtf.org" <nmlrg@irtf.org>
Thread-Topic: =?Windows-1252?Q?NetworkML_2016_workshop, _co-located_with_IEEE_ICNP'16, _N?= =?Windows-1252?Q?ov._8=9611,_2016,_Singapore?=
Thread-Index: AdHhmVMZmV0tMC/iRTGDcPxMHhsGRw==
Date: Tue, 19 Jul 2016 08:41:21 +0000
Message-ID: <5D36713D8A4E7348A7E10DF7437A4B927CAC61CC@NKGEML515-MBX.china.huawei.com>
Accept-Language: en-GB, zh-CN, en-US
Content-Language: en-GB
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
x-originating-ip: [10.200.218.254]
Content-Type: text/plain; charset="Windows-1252"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-CFilter-Loop: Reflected
X-Mirapoint-Virus-RAPID-Raw: score=unknown(0), refid=str=0001.0A0B0207.578DE7B4.0036, ss=1, re=0.000, recu=0.000, reip=0.000,  cl=1, cld=1, fgs=0, ip=0.0.0.0, so=2013-06-18 04:22:30, dmn=2013-03-21 17:37:32
X-Mirapoint-Loop-Id: ddd1b757c2803ddb8cea071a5486a628
Archived-At: <https://mailarchive.ietf.org/arch/msg/nmlrg/SMH7glVfdYr9apld0ffpTLD8mFs>
Subject: [Nmlrg] =?windows-1252?q?=3D=3FWindows-1252=3FQ=3FNetworkML=5F201?= =?windows-1252?q?6=5Fworkshop=2C_=5Fco-located=5Fwith=5FIEEE=5FICNP=2716?= =?windows-1252?q?=2C_=5FN=3F=3D_ov=2E_8=9611=2C_2016=2C_Singapore?=
X-BeenThere: nmlrg@irtf.org
X-Mailman-Version: 2.1.17
Precedence: list
List-Id: Network Machine Learning Research Group <nmlrg.irtf.org>
List-Unsubscribe: <https://www.irtf.org/mailman/options/nmlrg>, <mailto:nmlrg-request@irtf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/nmlrg/>
List-Post: <mailto:nmlrg@irtf.org>
List-Help: <mailto:nmlrg-request@irtf.org?subject=help>
List-Subscribe: <https://www.irtf.org/mailman/listinfo/nmlrg>, <mailto:nmlrg-request@irtf.org?subject=subscribe>
X-List-Received-Date: Tue, 19 Jul 2016 08:41:46 -0000

Hi, NMLRGers,=0A=
=0A=
This workshop targets very similar area with our NML RG. It is one of the f=
irst workshop that is dedicated to machine learning in computer networks.=
=0A=
=0A=
IEEE ICNP Workshop on Machine Learning in Computer Networks (NetworkML 2016=
)=0A=
Co-located with IEEE ICNP'16, Nov. 8=9611, 2016, Singapore.=0A=
http://networkml.github.io/=0A=
=0A=
The paper submission deadline to NetworkML 2016 is extended to July 27, 23:=
59HKT.=0A=
=0A=
Topics including, but not limited to, the following:=0A=
=0A=
- Protocol design and optimization using machine learning=0A=
- Resource allocation for shared/virtualized networks using machine learnin=
g=0A=
- Fault-tolerant network protocols using machine learning=0A=
- Machine learning aided network management=0A=
- Experiences and best-practices using machine learning in operational netw=
orks=0A=
- Security, performance, and monitoring applications using machine learning=
=0A=
- Implications and challenges brought by computer networks to machine learn=
ing theory and algorithms=0A=
- Data-driven network architecture design=0A=
- Application-driven network architecture design=0A=
- Data analytics for network information mining=0A=
- Deep learning and reinforcement learning in network control=0A=
- Learning-based network optimization=0A=
=0A=
Regards,=0A=
=0A=
Sheng=

