Subject: Re: [nmrg] 26th nmrg meeting minutes uploaded
From: "Ersue, Mehmet (NSN - DE/Munich)" <[email-masked]>
To: "ext Simon Leinen" <[email-masked]>, <[email-masked]>
Date: Fri, 20 Aug 2010 22:16:39 +0200
List-Id: Network Management Research Group <nmrg.ibr.cs.tu-bs.de>

Q is question and C is most likely one of the chairs 
(but which one ;-)

Cheers,
Mehmet

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]
> bs.de] On Behalf Of ext Simon Leinen
> Sent: Friday, August 20, 2010 2:20 PM
> To: [email-masked]
> Subject: Re: [nmrg] 26th nmrg meeting minutes uploaded
> 
> Juergen Schoenwaelder writes:
> > I just uploaded the minutes of the 26th NMRG meeting that took place
> > during the 78th IETF meeting:
> 
> > http://www.ietf.org/proceedings/78/minutes/NMRG.txt
> 
> > Many thanks to the note takes and the minutes editors.
> 
> Thanks!
> 
> But who are "Q" and "C"?
> --
> Simon.
> --
> !! This message is brought to you via the `nmrg' mailing list.
> !! Please do not reply to this message to unsubscribe. To unsubscribe
> or adjust
> !! your settings, send a mail message to
<[email-masked]>
> !! or look at https://mail.ibr.cs.tu-bs.de/mailman/listinfo/nmrg.

From: Simon Leinen <[email-masked]>
To: [email-masked]
Date: Fri, 20 Aug 2010 14:19:32 +0200
Subject: Re: [nmrg] 26th nmrg meeting minutes uploaded

Juergen Schoenwaelder writes:
> I just uploaded the minutes of the 26th NMRG meeting that took place
> during the 78th IETF meeting:

> http://www.ietf.org/proceedings/78/minutes/NMRG.txt

> Many thanks to the note takes and the minutes editors.

Thanks!

But who are "Q" and "C"?
-- 
Simon.

Date: Fri, 20 Aug 2010 11:10:32 +0200
From: Juergen Schoenwaelder <[email-masked]>
To: [email-masked]
Mail-Followup-To: [email-masked]
Subject: [nmrg] 26th nmrg meeting minutes uploaded
Reply-To: Juergen Schoenwaelder <[email-masked]>

Hi,

I just uploaded the minutes of the 26th NMRG meeting that took place
during the 78th IETF meeting:

http://www.ietf.org/proceedings/78/minutes/NMRG.txt

Many thanks to the note takes and the minutes editors.

/js

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1, 28759 Bremen, Germany
Fax:   +49 421 200 3103         <http://www.jacobs-university.de/>

Date: Sun, 8 Aug 2010 17:46:55 +0530
From: Hari Narayanan <[email-masked]>
To: "[email-masked]" <[email-masked]>
Subject: Re: [nmrg] SNMP OID Compression

--005045015d4b3cae1e048d4ee0d8
Content-Type: text/plain; charset=ISO-8859-1

I am trying understand GetColumn proposal by Dave Perkin (
http://www.ietf.org/proceedings/48/SLIDES/snmpv3-getcols) in the context of
my current Net SNMP developments on OID compression. Pl. send me the
pointers if this has already been discussed, otherwise I would like to know
more about GetColumn, specifically the following questions:

   - GetColumn proposes to address Overshoot with agent side detection and
   holes with "noSuchInstance" exception. Is there any issue in using the same
   type of solutions for GetBulk besides compatibility?
   - I am assuming that the request generator is keeping a complete copy of
   its request to construct the OID for 2nd and subsequent column objects. What
   about a mid stream application like Wireshark? There is no differentiating
   sub-ids in the encoding. Even if we assume these are consecutive columns of
   a table, they do not have enough information to construct the OIDs.

Regards ... Hari T.S. Narayanan

On Wed, Aug 4, 2010 at 12:41 PM, Juergen Schoenwaelder <
[email-masked]> wrote:

> On Tue, Aug 03, 2010 at 06:20:49PM +0200, Hari Narayanan wrote:
>
> > OID Compression has been already discussed in two drafts ? Delta
> > compression and Prefix compression. At present, I have worked on a
> > simple form of prefix compression where all the requested objects
> > are at the same level of MIB tree. First object in the request is
> > fully specified in PDU varbind, rest of the objects are specified
> > only with their respective suffix where they differ from the first
> > object.
>
> It would be nice to know how much simpler your compression scheme is
> compared to delta compression, which works with arbitrary sequences of
> OIDs since it does not make any assumptions of the OIDs in the
> payload. It may look more complicated at first sight but I believe the
> code to implement delta compression really is not that difficult.
>
> > To continue this investigation, I have also added additional code to
> > Net SNMP 5.5 to support this simplified prefix compression. This
> > enables me to measure code size, image size, response time, bytes
> > saved, and CPU cycles required etc. The additional code required is
> > about 400 lines (including comment and white space). These 400 lines
> > of code are sufficient to enable both agent and application to
> > support OID compression. The agent image is larger by about 4Kbytes.
> > I am yet to measure the response time rigorously, but I notice that
> > for getbulk request with large number of objects the response time
> > is better with OID compression. The code that encodes objid takes
> > less time because it has to code only the suffix part for all but
> > the first object. It is now easy to compare the legacy code with the
> > new code in ever aspect. There is only one API function to access
> > and use OID compression.  There is also command line option to
> > request OID compression.
>
> Several studies have reported that the time/cycles spent on BER
> encoding/decoding is not significant compared to the overall
> time/cycles it takes to process a request (see e.g. [4]). While this
> is already true for insecure versions of SNMP, if you add
> authentication and encryption, encoding/decoding overhead is not at
> all relevant. Back in the days (> 10 years ago) when OID compression
> was more heavily discussed, the main objective was to pack more data
> into response PDUs, in particular allowing GetBulk requests to ship
> more varbinds in responses. Since there is an overshoot issue (unless
> the manager can estimate the number of instances and provide
> reasonable max-repetitions), a GetRange PDU was proposed as well,
> original idea in [5], more detailed description here:
>
> http://tools.ietf.org/html/draft-irtf-nmrg-snmp-getrange-00
>
> Nowadays, interest is probably more coming from links that do have
> very small frame sizes and where fragmentation should be avoided (such
> as 802.15.4 networks) and where CPU cycles can potentially be saved
> when security is enabled.
>
> > I am also doing a profiling on a large collection of MIBs (from 250
> > vendors) to understand how and where OID compression could be taken
> > advantage off. This profiling includes average object depth, average
> > object value size, percentage of table objects, number fixed size
> > object, number of variable size objects, maximum depth, etc.
>
> Such studies based on an analysis of MIB modules have been done before
> [1]. There have also been efforts to collect real-world SNMP traces in
> order to analyze how deployed applications actually use SNMP [2]. A
> more general survey of SNMP performance studies can be found in [3].
>
> I am sending these references just in case they help to put your work
> into context.
>
> /js
>
> [1] J. Schonwalder: Characterization of SNMP MIB Modules. 9th
>    IFIP/IEEE International Symposium on Integrated Network
>    Management, pages 615-628, Nice, May 2005.
>
> [2] J. Schonwalder, A. Pras, M. Harvan, J. Schippers, R. van de Meent:
>    SNMP Traffic Analysis: Approaches, Tools, and First Results. 10th
>    IFIP/IEEE International Symposium on Integrated Network
>    Management, Munich, May 2007.
>
> [3] L. Andrey, O. Festor, A. Lahmadi, A. Pras, J. Schonwalder: Survey
>    of SNMP Performance Analysis Studies. International Journal of
>    Network Management 19(6), Wiley, November 2009.
>
> [4] Aiko Pras, Thomas Drevers, Remco van de Meent, Dirk Quartel:
>    Comparing the Performance of SNMP and Web services-based
>    Management. Transactions on Network and Service Management,
>    1(2), IEEE, 2004.
>
> [5] Malowidzki, M., "GetBulk Worth Fixing", Simple Times 10(1),
>    December 2002.
>
> --
> Juergen Schoenwaelder           Jacobs University Bremen gGmbH
> Phone: +49 421 200 3587         Campus Ring 1, 28759 Bremen, Germany
> Fax:   +49 421 200 3103         <http://www.jacobs-university.de/>
>

--005045015d4b3cae1e048d4ee0d8
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<div>I am trying understand GetColumn proposal by Dave Perkin (<a href="http://www.ietf.org/proceedings/48/SLIDES/snmpv3-getcols">http://www.ietf.org/proceedings/48/SLIDES/snmpv3-getcols</a>) in the context of my current Net SNMP developments on OID compression. Pl. send me the pointers if this has already been discussed, otherwise I would like to know more about GetColumn, specifically the following questions:</div>
<div><ul><li>GetColumn proposes to address Overshoot with agent side detection and holes with &quot;noSuchInstance&quot; exception. Is there any issue in using the same type of solutions for GetBulk besides compatibility?</li>
<li>I am assuming that the request generator is keeping a complete copy of its request to construct the OID for 2nd and subsequent column objects. What about a mid stream application like Wireshark? There is no differentiating sub-ids in the encoding. Even if we assume these are consecutive columns of a table, they do not have enough information to construct the OIDs.</li>
</ul></div><div>Regards ... Hari T.S. Narayanan</div><br><div class="gmail_quote">On Wed, Aug 4, 2010 at 12:41 PM, Juergen Schoenwaelder <span dir="ltr">&lt;<a href="mailto:[email-masked]">[email-masked]</a>&gt;</span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><div class="im">On Tue, Aug 03, 2010 at 06:20:49PM +0200, Hari Narayanan wrote:<br>
<br>
&gt; OID Compression has been already discussed in two drafts ? Delta<br>
&gt; compression and Prefix compression. At present, I have worked on a<br>
&gt; simple form of prefix compression where all the requested objects<br>
&gt; are at the same level of MIB tree. First object in the request is<br>
&gt; fully specified in PDU varbind, rest of the objects are specified<br>
&gt; only with their respective suffix where they differ from the first<br>
&gt; object.<br>
<br>
</div>It would be nice to know how much simpler your compression scheme is<br>
compared to delta compression, which works with arbitrary sequences of<br>
OIDs since it does not make any assumptions of the OIDs in the<br>
payload. It may look more complicated at first sight but I believe the<br>
code to implement delta compression really is not that difficult.<br>
<div class="im"><br>
&gt; To continue this investigation, I have also added additional code to<br>
&gt; Net SNMP 5.5 to support this simplified prefix compression. This<br>
&gt; enables me to measure code size, image size, response time, bytes<br>
&gt; saved, and CPU cycles required etc. The additional code required is<br>
&gt; about 400 lines (including comment and white space). These 400 lines<br>
&gt; of code are sufficient to enable both agent and application to<br>
&gt; support OID compression. The agent image is larger by about 4Kbytes.<br>
&gt; I am yet to measure the response time rigorously, but I notice that<br>
&gt; for getbulk request with large number of objects the response time<br>
&gt; is better with OID compression. The code that encodes objid takes<br>
&gt; less time because it has to code only the suffix part for all but<br>
&gt; the first object. It is now easy to compare the legacy code with the<br>
&gt; new code in ever aspect. There is only one API function to access<br>
&gt; and use OID compression.  There is also command line option to<br>
&gt; request OID compression.<br>
<br>
</div>Several studies have reported that the time/cycles spent on BER<br>
encoding/decoding is not significant compared to the overall<br>
time/cycles it takes to process a request (see e.g. [4]). While this<br>
is already true for insecure versions of SNMP, if you add<br>
authentication and encryption, encoding/decoding overhead is not at<br>
all relevant. Back in the days (&gt; 10 years ago) when OID compression<br>
was more heavily discussed, the main objective was to pack more data<br>
into response PDUs, in particular allowing GetBulk requests to ship<br>
more varbinds in responses. Since there is an overshoot issue (unless<br>
the manager can estimate the number of instances and provide<br>
reasonable max-repetitions), a GetRange PDU was proposed as well,<br>
original idea in [5], more detailed description here:<br>
<br>
<a href="http://tools.ietf.org/html/draft-irtf-nmrg-snmp-getrange-00" target="_blank">http://tools.ietf.org/html/draft-irtf-nmrg-snmp-getrange-00</a><br>
<br>
Nowadays, interest is probably more coming from links that do have<br>
very small frame sizes and where fragmentation should be avoided (such<br>
as 802.15.4 networks) and where CPU cycles can potentially be saved<br>
when security is enabled.<br>
<div class="im"><br>
&gt; I am also doing a profiling on a large collection of MIBs (from 250<br>
&gt; vendors) to understand how and where OID compression could be taken<br>
&gt; advantage off. This profiling includes average object depth, average<br>
&gt; object value size, percentage of table objects, number fixed size<br>
&gt; object, number of variable size objects, maximum depth, etc.<br>
<br>
</div>Such studies based on an analysis of MIB modules have been done before<br>
[1]. There have also been efforts to collect real-world SNMP traces in<br>
order to analyze how deployed applications actually use SNMP [2]. A<br>
more general survey of SNMP performance studies can be found in [3].<br>
<br>
I am sending these references just in case they help to put your work<br>
into context.<br>
<br>
/js<br>
<br>
[1] J. Schonwalder: Characterization of SNMP MIB Modules. 9th<br>
    IFIP/IEEE International Symposium on Integrated Network<br>
    Management, pages 615-628, Nice, May 2005.<br>
<br>
[2] J. Schonwalder, A. Pras, M. Harvan, J. Schippers, R. van de Meent:<br>
    SNMP Traffic Analysis: Approaches, Tools, and First Results. 10th<br>
    IFIP/IEEE International Symposium on Integrated Network<br>
    Management, Munich, May 2007.<br>
<br>
[3] L. Andrey, O. Festor, A. Lahmadi, A. Pras, J. Schonwalder: Survey<br>
    of SNMP Performance Analysis Studies. International Journal of<br>
    Network Management 19(6), Wiley, November 2009.<br>
<br>
[4] Aiko Pras, Thomas Drevers, Remco van de Meent, Dirk Quartel:<br>
    Comparing the Performance of SNMP and Web services-based<br>
    Management. Transactions on Network and Service Management,<br>
    1(2), IEEE, 2004.<br>
<br>
[5] Malowidzki, M., &quot;GetBulk Worth Fixing&quot;, Simple Times 10(1),<br>
    December 2002.<br>
<font color="#888888"><br>
--<br>
Juergen Schoenwaelder           Jacobs University Bremen gGmbH<br>
Phone: +49 421 200 3587         Campus Ring 1, 28759 Bremen, Germany<br>
Fax:   +49 421 200 3103         &lt;<a href="http://www.jacobs-university.de/" target="_blank">http://www.jacobs-university.de/</a>&gt;<br>
</font></blockquote></div><br>

--005045015d4b3cae1e048d4ee0d8--

Date: Thu, 5 Aug 2010 20:26:28 +0530
From: Hari Narayanan <[email-masked]>
To: David Harrington <[email-masked]>
Cc: [email-masked], Juergen Schoenwaelder <[email-masked]>
Subject: Re: [nmrg] SNMP OID Compression

--00504501400c561194048d14c142
Content-Type: text/plain; charset=ISO-8859-1

David,
Circa 1997, Nortel developed the first integrated NMS using SPECTRUM for
their enterprise customers. I was the project lead for that and accumulated
lot of air miles between NH and San Jose. Cabletron and Nortel had a joint
stall at Interop, Atlanta to showcase this. It is so long back that I am not
able recall any of the names of my Cabletron contacts.
=
There are a wide spectrum of SNMP developers - at one one end of this
spectrum we see developers who are simply coding applications for functional
spec, at the other end we see developers who are trying to optimize every
CPU cycle (at the agent side) because every resource is scarce and agent is
one of the least priority tasks. OID compression is an additional option for
the latter folks. Eventually, I see every SNMP developer to make use of this
once it is available and awareness is created.  This is for future
developers and it is not for retrofitting unless it makes sense.

OID compression is a zero overhead feature and it should not interefere with
object order in varbind list. Compression is that last function to execute
before a PDU is sent and decompression is the first function to execute when
a PDU is received. Thus, it is least likely to affect any other optimization
at the agent and application side. Mid stream debugging and sniffing tools
are required to be rewritten for the compressed PDU.  I am currently trying
to update Wireshark to recognize my OID compressed PDU.

This optimization addresses the table counter objects specifically. These
objects are frequently requested and there are several rows of these counter
objects. In general, it is usefull for requesting groups of related objects
that are changing frequently at the agent side.

Regards ... Hari T.S. Narayanan

On Mon, Jul 5, 2010 at 1:14 AM, David Harrington <[email-masked]>wrote:

> Hi,
>
> I used to work on the Spectrum network management application (years
> ago).
>
> Juergen mentioned real world SNMP traces. My concern with these traces
> is with the sample networks, which tended to be academic. I think
> academic networks can be very different than commercial networks in
> the nature of their management practices. I know Spectrum sold for
> $100K+. I am not sure how many academic departments spend $100K+ on
> their management platforms.
>
> I am of the impression that how individual applications collect
> objects is likely to be the single biggest factor in SNMP performance.
> Much like software profiling, optimizing the collection of a MIB
> object that is never or seldom collected yields little benefit.
> Optimizing the collection of a single object that is collected very
> frequently will yield better results. In software profiling, it is
> generally better to optimize printf() than initialize_application().
>
> I suggest that a potentially fruitful area of research would be to
> study how the major SNMP NMS platforms behave before being extended
> for additional MIB modules. These platforms constantly check certain
> objects (like sysUpTime) to determine connectivity, detect
> discontinuities, update synchronization, monitor interface changes,
> etc. They check these routinely every few minutes throughout the day,
> for all devices. This generates much more traffic to optimize than
> operator-initiated reads using command line tools.
>
> In my experience, which is now pretty dated, most MIB module
> extensions are treated with custom polling instructions defined by the
> operator, so they vary by deployment. Otherwise, the extension MIBs
> are handled by MIB browser functionality - they are often read only on
> demand.
>
> If an NMS platform has add-on applications, the application will have
> certain objects it will monitor (possibly using traps) to determine
> when a significant event occurs, and then it will launch a series of
> special reads to gather more info on the event, possibly on a
> vendor/model-specific basis.
>
> I would recommend avoiding any proposals that make varbind order
> important. Spectrum optimizes (used to optimize) SNMPv1 reads by
> reordering the varbinds, and they use a polling engine that might
> combine requested polls from multiple applications into a single
> varbind to the device. (I don't know if they still do any of that
> given SNMPv3 user-based security.) You also might have threading and
> subagent issues.
>
> Spectrum also used to optimize data collection by determining, with
> administrator and historical input, how frequently an object needed to
> be actually read from the device, or whether an existing database
> value, possibly collected for a different application, that is less
> than <x> seconds old would suffice. Counters are the datatypes that
> most frequently must be refreshed by an actual read, so optimizing for
> counters would likely yield bigger benefits than optimizing static
> strings.
>
> As Juergen points out, in the face of other optimization techniques,
> OID compression might not be that important. It may be important for
> resource-contrained devices, but you'll want to be careful not to
> interfere with existing optimization techniques used by the
> application side.
>
> dbh
>
> > -----Original Message-----
> > From: [email-masked]
> > [mailto:[email-masked]] On Behalf Of Juergen
> > Schoenwaelder
> > Sent: Wednesday, August 04, 2010 9:11 AM
> > To: Hari Narayanan
> > Cc: [email-masked]
> > Subject: Re: [nmrg] SNMP OID Compression
> >
> > On Tue, Aug 03, 2010 at 06:20:49PM +0200, Hari Narayanan wrote:
> >
> > > OID Compression has been already discussed in two drafts ? Delta
> > > compression and Prefix compression. At present, I have worked on a
>
> > > simple form of prefix compression where all the requested
> > objects are
> > > at the same level of MIB tree. First object in the request is
> fully
> > > specified in PDU varbind, rest of the objects are specified
> > only with
> > > their respective suffix where they differ from the first object.
> >
> > It would be nice to know how much simpler your compression
> > scheme is compared to delta compression, which works with
> > arbitrary sequences of OIDs since it does not make any
> > assumptions of the OIDs in the payload. It may look more
> > complicated at first sight but I believe the code to
> > implement delta compression really is not that difficult.
> >
> > > To continue this investigation, I have also added
> > additional code to
> > > Net SNMP 5.5 to support this simplified prefix compression. This
> > > enables me to measure code size, image size, response time, bytes
> > > saved, and CPU cycles required etc. The additional code required
> is
> > > about 400 lines (including comment and white space). These
> > 400 lines
> > > of code are sufficient to enable both agent and application
> > to support
> > > OID compression. The agent image is larger by about 4Kbytes.
> > > I am yet to measure the response time rigorously, but I notice
> that
> > > for getbulk request with large number of objects the
> > response time is
> > > better with OID compression. The code that encodes objid takes
> less
> > > time because it has to code only the suffix part for all
> > but the first
> > > object. It is now easy to compare the legacy code with the
> > new code in
> > > ever aspect. There is only one API function to access and use OID
> > > compression.  There is also command line option to request OID
> > > compression.
> >
> > Several studies have reported that the time/cycles spent on
> > BER encoding/decoding is not significant compared to the
> > overall time/cycles it takes to process a request (see e.g.
> > [4]). While this is already true for insecure versions of
> > SNMP, if you add authentication and encryption,
> > encoding/decoding overhead is not at all relevant. Back in
> > the days (> 10 years ago) when OID compression was more
> > heavily discussed, the main objective was to pack more data
> > into response PDUs, in particular allowing GetBulk requests
> > to ship more varbinds in responses. Since there is an
> > overshoot issue (unless the manager can estimate the number
> > of instances and provide reasonable max-repetitions), a
> > GetRange PDU was proposed as well, original idea in [5], more
> > detailed description here:
> >
> > http://tools.ietf.org/html/draft-irtf-nmrg-snmp-getrange-00
> >
> > Nowadays, interest is probably more coming from links that do
> > have very small frame sizes and where fragmentation should be
> > avoided (such as 802.15.4 networks) and where CPU cycles can
> > potentially be saved when security is enabled.
> >
> > > I am also doing a profiling on a large collection of MIBs (from
> 250
> > > vendors) to understand how and where OID compression could be
> taken
> > > advantage off. This profiling includes average object
> > depth, average
> > > object value size, percentage of table objects, number fixed size
> > > object, number of variable size objects, maximum depth, etc.
> >
> > Such studies based on an analysis of MIB modules have been
> > done before [1]. There have also been efforts to collect
> > real-world SNMP traces in order to analyze how deployed
> > applications actually use SNMP [2]. A more general survey of
> > SNMP performance studies can be found in [3].
> >
> > I am sending these references just in case they help to put
> > your work into context.
> >
> > /js
> >
> > [1] J. Schonwalder: Characterization of SNMP MIB Modules. 9th
> >     IFIP/IEEE International Symposium on Integrated Network
> >     Management, pages 615-628, Nice, May 2005.
> >
> > [2] J. Schonwalder, A. Pras, M. Harvan, J. Schippers, R. van de
> Meent:
> >     SNMP Traffic Analysis: Approaches, Tools, and First Results.
> 10th
> >     IFIP/IEEE International Symposium on Integrated Network
> >     Management, Munich, May 2007.
> >
> > [3] L. Andrey, O. Festor, A. Lahmadi, A. Pras, J. Schonwalder:
> Survey
> >     of SNMP Performance Analysis Studies. International Journal of
> >     Network Management 19(6), Wiley, November 2009.
> >
> > [4] Aiko Pras, Thomas Drevers, Remco van de Meent, Dirk Quartel:
> >     Comparing the Performance of SNMP and Web services-based
> >     Management. Transactions on Network and Service Management,
> >     1(2), IEEE, 2004.
> >
> > [5] Malowidzki, M., "GetBulk Worth Fixing", Simple Times 10(1),
> >     December 2002.
> >
> > --
> > Juergen Schoenwaelder           Jacobs University Bremen gGmbH
> > Phone: +49 421 200 3587         Campus Ring 1, 28759 Bremen, Germany
> > Fax:   +49 421 200 3103         <http://www.jacobs-university.de/>
> > --
> > !! This message is brought to you via the `nmrg' mailing list.
> > !! Please do not reply to this message to unsubscribe. To
> > unsubscribe or adjust !! your settings, send a mail message
> > to <[email-masked]> !! or look at
> > https://mail.ibr.cs.tu-bs.de/mailman/listinfo/nmrg.
>
>

--00504501400c561194048d14c142
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<div>David,</div><div>Circa 1997, Nortel developed the first integrated NMS using SPECTRUM for their enterprise customers. I was the project lead for that and accumulated lot of air miles between NH and San Jose. Cabletron and Nortel had a joint stall at Interop, Atlanta to showcase this. It is so long back that I am not able recall any of the names of my Cabletron contacts.</div>
<div>===</div><div><br></div><div>There are a wide spectrum of SNMP developers - at one one end of this spectrum we see developers who are simply coding applications for functional spec, at the other end we see developers who are trying to optimize every CPU cycle (at the agent side) because every resource is scarce and agent is one of the least priority tasks. OID compression is an additional option for the latter folks. Eventually, I see every SNMP developer to make use of this once it is available and awareness is created.  This is for future developers and it is not for retrofitting unless it makes sense.</div>
<div><br></div><div>OID compression is a zero overhead feature and it should not interefere with object order in varbind list. Compression is that last function to execute before a PDU is sent and decompression is the first function to execute when a PDU is received. Thus, it is least likely to affect any other optimization at the agent and application side. Mid stream debugging and sniffing tools are required to be rewritten for the compressed PDU.  I am currently trying to update Wireshark to recognize my OID compressed PDU.</div>
<div><br></div><div>This optimization addresses the table counter objects specifically. These objects are frequently requested and there are several rows of these counter objects. In general, it is usefull for requesting groups of related objects that are changing frequently at the agent side.</div>
<div><br></div><div>Regards ... Hari T.S. Narayanan</div><div><div><br></div><div><br><div class="gmail_quote">On Mon, Jul 5, 2010 at 1:14 AM, David Harrington <span dir="ltr">&lt;<a href="mailto:[email-masked]" target="_blank">[email-masked]</a>&gt;</span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi,<br>
<br>
I used to work on the Spectrum network management application (years<br>
ago).<br>
<br>
Juergen mentioned real world SNMP traces. My concern with these traces<br>
is with the sample networks, which tended to be academic. I think<br>
academic networks can be very different than commercial networks in<br>
the nature of their management practices. I know Spectrum sold for<br>
$100K+. I am not sure how many academic departments spend $100K+ on<br>
their management platforms.<br>
<br>
I am of the impression that how individual applications collect<br>
objects is likely to be the single biggest factor in SNMP performance.<br>
Much like software profiling, optimizing the collection of a MIB<br>
object that is never or seldom collected yields little benefit.<br>
Optimizing the collection of a single object that is collected very<br>
frequently will yield better results. In software profiling, it is<br>
generally better to optimize printf() than initialize_application().<br>
<br>
I suggest that a potentially fruitful area of research would be to<br>
study how the major SNMP NMS platforms behave before being extended<br>
for additional MIB modules. These platforms constantly check certain<br>
objects (like sysUpTime) to determine connectivity, detect<br>
discontinuities, update synchronization, monitor interface changes,<br>
etc. They check these routinely every few minutes throughout the day,<br>
for all devices. This generates much more traffic to optimize than<br>
operator-initiated reads using command line tools.<br>
<br>
In my experience, which is now pretty dated, most MIB module<br>
extensions are treated with custom polling instructions defined by the<br>
operator, so they vary by deployment. Otherwise, the extension MIBs<br>
are handled by MIB browser functionality - they are often read only on<br>
demand.<br>
<br>
If an NMS platform has add-on applications, the application will have<br>
certain objects it will monitor (possibly using traps) to determine<br>
when a significant event occurs, and then it will launch a series of<br>
special reads to gather more info on the event, possibly on a<br>
vendor/model-specific basis.<br>
<br>
I would recommend avoiding any proposals that make varbind order<br>
important. Spectrum optimizes (used to optimize) SNMPv1 reads by<br>
reordering the varbinds, and they use a polling engine that might<br>
combine requested polls from multiple applications into a single<br>
varbind to the device. (I don&#39;t know if they still do any of that<br>
given SNMPv3 user-based security.) You also might have threading and<br>
subagent issues.<br>
<br>
Spectrum also used to optimize data collection by determining, with<br>
administrator and historical input, how frequently an object needed to<br>
be actually read from the device, or whether an existing database<br>
value, possibly collected for a different application, that is less<br>
than &lt;x&gt; seconds old would suffice. Counters are the datatypes that<br>
most frequently must be refreshed by an actual read, so optimizing for<br>
counters would likely yield bigger benefits than optimizing static<br>
strings.<br>
<br>
As Juergen points out, in the face of other optimization techniques,<br>
OID compression might not be that important. It may be important for<br>
resource-contrained devices, but you&#39;ll want to be careful not to<br>
interfere with existing optimization techniques used by the<br>
application side.<br>
<br>
dbh<br>
<div><div></div><div><br>
&gt; -----Original Message-----<br>
&gt; From: <a href="mailto:[email-masked]" target="_blank">[email-masked]</a><br>
&gt; [mailto:<a href="mailto:[email-masked]" target="_blank">[email-masked]</a>] On Behalf Of Juergen<br>
&gt; Schoenwaelder<br>
&gt; Sent: Wednesday, August 04, 2010 9:11 AM<br>
&gt; To: Hari Narayanan<br>
&gt; Cc: <a href="mailto:[email-masked]" target="_blank">[email-masked]</a><br>
&gt; Subject: Re: [nmrg] SNMP OID Compression<br>
&gt;<br>
&gt; On Tue, Aug 03, 2010 at 06:20:49PM +0200, Hari Narayanan wrote:<br>
&gt;<br>
&gt; &gt; OID Compression has been already discussed in two drafts ? Delta<br>
&gt; &gt; compression and Prefix compression. At present, I have worked on a<br>
<br>
&gt; &gt; simple form of prefix compression where all the requested<br>
&gt; objects are<br>
&gt; &gt; at the same level of MIB tree. First object in the request is<br>
fully<br>
&gt; &gt; specified in PDU varbind, rest of the objects are specified<br>
&gt; only with<br>
&gt; &gt; their respective suffix where they differ from the first object.<br>
&gt;<br>
&gt; It would be nice to know how much simpler your compression<br>
&gt; scheme is compared to delta compression, which works with<br>
&gt; arbitrary sequences of OIDs since it does not make any<br>
&gt; assumptions of the OIDs in the payload. It may look more<br>
&gt; complicated at first sight but I believe the code to<br>
&gt; implement delta compression really is not that difficult.<br>
&gt;<br>
&gt; &gt; To continue this investigation, I have also added<br>
&gt; additional code to<br>
&gt; &gt; Net SNMP 5.5 to support this simplified prefix compression. This<br>
&gt; &gt; enables me to measure code size, image size, response time, bytes<br>
&gt; &gt; saved, and CPU cycles required etc. The additional code required<br>
is<br>
&gt; &gt; about 400 lines (including comment and white space). These<br>
&gt; 400 lines<br>
&gt; &gt; of code are sufficient to enable both agent and application<br>
&gt; to support<br>
&gt; &gt; OID compression. The agent image is larger by about 4Kbytes.<br>
&gt; &gt; I am yet to measure the response time rigorously, but I notice<br>
that<br>
&gt; &gt; for getbulk request with large number of objects the<br>
&gt; response time is<br>
&gt; &gt; better with OID compression. The code that encodes objid takes<br>
less<br>
&gt; &gt; time because it has to code only the suffix part for all<br>
&gt; but the first<br>
&gt; &gt; object. It is now easy to compare the legacy code with the<br>
&gt; new code in<br>
&gt; &gt; ever aspect. There is only one API function to access and use OID<br>
&gt; &gt; compression.  There is also command line option to request OID<br>
&gt; &gt; compression.<br>
&gt;<br>
&gt; Several studies have reported that the time/cycles spent on<br>
&gt; BER encoding/decoding is not significant compared to the<br>
&gt; overall time/cycles it takes to process a request (see e.g.<br>
&gt; [4]). While this is already true for insecure versions of<br>
&gt; SNMP, if you add authentication and encryption,<br>
&gt; encoding/decoding overhead is not at all relevant. Back in<br>
&gt; the days (&gt; 10 years ago) when OID compression was more<br>
&gt; heavily discussed, the main objective was to pack more data<br>
&gt; into response PDUs, in particular allowing GetBulk requests<br>
&gt; to ship more varbinds in responses. Since there is an<br>
&gt; overshoot issue (unless the manager can estimate the number<br>
&gt; of instances and provide reasonable max-repetitions), a<br>
&gt; GetRange PDU was proposed as well, original idea in [5], more<br>
&gt; detailed description here:<br>
&gt;<br>
&gt; <a href="http://tools.ietf.org/html/draft-irtf-nmrg-snmp-getrange-00" target="_blank">http://tools.ietf.org/html/draft-irtf-nmrg-snmp-getrange-00</a><br>
&gt;<br>
&gt; Nowadays, interest is probably more coming from links that do<br>
&gt; have very small frame sizes and where fragmentation should be<br>
&gt; avoided (such as 802.15.4 networks) and where CPU cycles can<br>
&gt; potentially be saved when security is enabled.<br>
&gt;<br>
&gt; &gt; I am also doing a profiling on a large collection of MIBs (from<br>
250<br>
&gt; &gt; vendors) to understand how and where OID compression could be<br>
taken<br>
&gt; &gt; advantage off. This profiling includes average object<br>
&gt; depth, average<br>
&gt; &gt; object value size, percentage of table objects, number fixed size<br>
&gt; &gt; object, number of variable size objects, maximum depth, etc.<br>
&gt;<br>
&gt; Such studies based on an analysis of MIB modules have been<br>
&gt; done before [1]. There have also been efforts to collect<br>
&gt; real-world SNMP traces in order to analyze how deployed<br>
&gt; applications actually use SNMP [2]. A more general survey of<br>
&gt; SNMP performance studies can be found in [3].<br>
&gt;<br>
&gt; I am sending these references just in case they help to put<br>
&gt; your work into context.<br>
&gt;<br>
&gt; /js<br>
&gt;<br>
&gt; [1] J. Schonwalder: Characterization of SNMP MIB Modules. 9th<br>
&gt;     IFIP/IEEE International Symposium on Integrated Network<br>
&gt;     Management, pages 615-628, Nice, May 2005.<br>
&gt;<br>
&gt; [2] J. Schonwalder, A. Pras, M. Harvan, J. Schippers, R. van de<br>
Meent:<br>
&gt;     SNMP Traffic Analysis: Approaches, Tools, and First Results.<br>
10th<br>
&gt;     IFIP/IEEE International Symposium on Integrated Network<br>
&gt;     Management, Munich, May 2007.<br>
&gt;<br>
&gt; [3] L. Andrey, O. Festor, A. Lahmadi, A. Pras, J. Schonwalder:<br>
Survey<br>
&gt;     of SNMP Performance Analysis Studies. International Journal of<br>
&gt;     Network Management 19(6), Wiley, November 2009.<br>
&gt;<br>
&gt; [4] Aiko Pras, Thomas Drevers, Remco van de Meent, Dirk Quartel:<br>
&gt;     Comparing the Performance of SNMP and Web services-based<br>
&gt;     Management. Transactions on Network and Service Management,<br>
&gt;     1(2), IEEE, 2004.<br>
&gt;<br>
&gt; [5] Malowidzki, M., &quot;GetBulk Worth Fixing&quot;, Simple Times 10(1),<br>
&gt;     December 2002.<br>
&gt;<br>
&gt; --<br>
&gt; Juergen Schoenwaelder           Jacobs University Bremen gGmbH<br>
&gt; Phone: +49 421 200 3587         Campus Ring 1, 28759 Bremen, Germany<br>
&gt; Fax:   +49 421 200 3103         &lt;<a href="http://www.jacobs-university.de/" target="_blank">http://www.jacobs-university.de/</a>&gt;<br>
</div></div>&gt; --<br>
&gt; !! This message is brought to you via the `nmrg&#39; mailing list.<br>
&gt; !! Please do not reply to this message to unsubscribe. To<br>
&gt; unsubscribe or adjust !! your settings, send a mail message<br>
&gt; to &lt;<a href="mailto:[email-masked]" target="_blank">[email-masked]</a>&gt; !! or look at<br>
&gt; <a href="https://mail.ibr.cs.tu-bs.de/mailman/listinfo/nmrg" target="_blank">https://mail.ibr.cs.tu-bs.de/mailman/listinfo/nmrg</a>.<br>
<br>
</blockquote></div><br>
</div></div>

--00504501400c561194048d14c142--

Sender: [email-masked]
Date: Wed, 4 Aug 2010 21:32:26 -0400
From: Mark Ellison <[email-masked]>
To: [email-masked]
Cc: David Harrington <[email-masked]>, Juergen Schoenwaelder <[email-masked]>
Subject: Re: [nmrg] SNMP OID Compression

--0016e6409c0cdb6573048d09850e
Content-Type: text/plain; charset=ISO-8859-1

Hi,

This subject is interesting.  I have worked with the SNMP since 1990 and
believe there might be a rather simple approach to (OID) compression.

It occurs to me that the target area to be compressed resides solely within
the Scoped PDU.

Prior to sending, might there be benefit to simply run the Scoped PDU
through a (lossless data) compression algorithm prior to transformation via
privacy/encryption and calculation of authentication hashes?

Such an approach would be impartial to the frequency of OID prefixes or
instances present.  It would simply condense the most frequent occurrences
of subid patterns.

Regards,

Mark

On Sun, Jul 4, 2010 at 3:44 PM, David Harrington <[email-masked]>wrote:

> Hi,
>
> I used to work on the Spectrum network management application (years
> ago).
>
> Juergen mentioned real world SNMP traces. My concern with these traces
> is with the sample networks, which tended to be academic. I think
> academic networks can be very different than commercial networks in
> the nature of their management practices. I know Spectrum sold for
> $100K+. I am not sure how many academic departments spend $100K+ on
> their management platforms.
>
> I am of the impression that how individual applications collect
> objects is likely to be the single biggest factor in SNMP performance.
> Much like software profiling, optimizing the collection of a MIB
> object that is never or seldom collected yields little benefit.
> Optimizing the collection of a single object that is collected very
> frequently will yield better results. In software profiling, it is
> generally better to optimize printf() than initialize_application().
>
> I suggest that a potentially fruitful area of research would be to
> study how the major SNMP NMS platforms behave before being extended
> for additional MIB modules. These platforms constantly check certain
> objects (like sysUpTime) to determine connectivity, detect
> discontinuities, update synchronization, monitor interface changes,
> etc. They check these routinely every few minutes throughout the day,
> for all devices. This generates much more traffic to optimize than
> operator-initiated reads using command line tools.
>
> In my experience, which is now pretty dated, most MIB module
> extensions are treated with custom polling instructions defined by the
> operator, so they vary by deployment. Otherwise, the extension MIBs
> are handled by MIB browser functionality - they are often read only on
> demand.
>
> If an NMS platform has add-on applications, the application will have
> certain objects it will monitor (possibly using traps) to determine
> when a significant event occurs, and then it will launch a series of
> special reads to gather more info on the event, possibly on a
> vendor/model-specific basis.
>
> I would recommend avoiding any proposals that make varbind order
> important. Spectrum optimizes (used to optimize) SNMPv1 reads by
> reordering the varbinds, and they use a polling engine that might
> combine requested polls from multiple applications into a single
> varbind to the device. (I don't know if they still do any of that
> given SNMPv3 user-based security.) You also might have threading and
> subagent issues.
>
> Spectrum also used to optimize data collection by determining, with
> administrator and historical input, how frequently an object needed to
> be actually read from the device, or whether an existing database
> value, possibly collected for a different application, that is less
> than <x> seconds old would suffice. Counters are the datatypes that
> most frequently must be refreshed by an actual read, so optimizing for
> counters would likely yield bigger benefits than optimizing static
> strings.
>
> As Juergen points out, in the face of other optimization techniques,
> OID compression might not be that important. It may be important for
> resource-contrained devices, but you'll want to be careful not to
> interfere with existing optimization techniques used by the
> application side.
>
> dbh
>
> > -----Original Message-----
> > From: [email-masked]
> > [mailto:[email-masked]] On Behalf Of Juergen
> > Schoenwaelder
> > Sent: Wednesday, August 04, 2010 9:11 AM
> > To: Hari Narayanan
> > Cc: [email-masked]
> > Subject: Re: [nmrg] SNMP OID Compression
> >
> > On Tue, Aug 03, 2010 at 06:20:49PM +0200, Hari Narayanan wrote:
> >
> > > OID Compression has been already discussed in two drafts ? Delta
> > > compression and Prefix compression. At present, I have worked on a
>
> > > simple form of prefix compression where all the requested
> > objects are
> > > at the same level of MIB tree. First object in the request is
> fully
> > > specified in PDU varbind, rest of the objects are specified
> > only with
> > > their respective suffix where they differ from the first object.
> >
> > It would be nice to know how much simpler your compression
> > scheme is compared to delta compression, which works with
> > arbitrary sequences of OIDs since it does not make any
> > assumptions of the OIDs in the payload. It may look more
> > complicated at first sight but I believe the code to
> > implement delta compression really is not that difficult.
> >
> > > To continue this investigation, I have also added
> > additional code to
> > > Net SNMP 5.5 to support this simplified prefix compression. This
> > > enables me to measure code size, image size, response time, bytes
> > > saved, and CPU cycles required etc. The additional code required
> is
> > > about 400 lines (including comment and white space). These
> > 400 lines
> > > of code are sufficient to enable both agent and application
> > to support
> > > OID compression. The agent image is larger by about 4Kbytes.
> > > I am yet to measure the response time rigorously, but I notice
> that
> > > for getbulk request with large number of objects the
> > response time is
> > > better with OID compression. The code that encodes objid takes
> less
> > > time because it has to code only the suffix part for all
> > but the first
> > > object. It is now easy to compare the legacy code with the
> > new code in
> > > ever aspect. There is only one API function to access and use OID
> > > compression.  There is also command line option to request OID
> > > compression.
> >
> > Several studies have reported that the time/cycles spent on
> > BER encoding/decoding is not significant compared to the
> > overall time/cycles it takes to process a request (see e.g.
> > [4]). While this is already true for insecure versions of
> > SNMP, if you add authentication and encryption,
> > encoding/decoding overhead is not at all relevant. Back in
> > the days (> 10 years ago) when OID compression was more
> > heavily discussed, the main objective was to pack more data
> > into response PDUs, in particular allowing GetBulk requests
> > to ship more varbinds in responses. Since there is an
> > overshoot issue (unless the manager can estimate the number
> > of instances and provide reasonable max-repetitions), a
> > GetRange PDU was proposed as well, original idea in [5], more
> > detailed description here:
> >
> > http://tools.ietf.org/html/draft-irtf-nmrg-snmp-getrange-00
> >
> > Nowadays, interest is probably more coming from links that do
> > have very small frame sizes and where fragmentation should be
> > avoided (such as 802.15.4 networks) and where CPU cycles can
> > potentially be saved when security is enabled.
> >
> > > I am also doing a profiling on a large collection of MIBs (from
> 250
> > > vendors) to understand how and where OID compression could be
> taken
> > > advantage off. This profiling includes average object
> > depth, average
> > > object value size, percentage of table objects, number fixed size
> > > object, number of variable size objects, maximum depth, etc.
> >
> > Such studies based on an analysis of MIB modules have been
> > done before [1]. There have also been efforts to collect
> > real-world SNMP traces in order to analyze how deployed
> > applications actually use SNMP [2]. A more general survey of
> > SNMP performance studies can be found in [3].
> >
> > I am sending these references just in case they help to put
> > your work into context.
> >
> > /js
> >
> > [1] J. Schonwalder: Characterization of SNMP MIB Modules. 9th
> >     IFIP/IEEE International Symposium on Integrated Network
> >     Management, pages 615-628, Nice, May 2005.
> >
> > [2] J. Schonwalder, A. Pras, M. Harvan, J. Schippers, R. van de
> Meent:
> >     SNMP Traffic Analysis: Approaches, Tools, and First Results.
> 10th
> >     IFIP/IEEE International Symposium on Integrated Network
> >     Management, Munich, May 2007.
> >
> > [3] L. Andrey, O. Festor, A. Lahmadi, A. Pras, J. Schonwalder:
> Survey
> >     of SNMP Performance Analysis Studies. International Journal of
> >     Network Management 19(6), Wiley, November 2009.
> >
> > [4] Aiko Pras, Thomas Drevers, Remco van de Meent, Dirk Quartel:
> >     Comparing the Performance of SNMP and Web services-based
> >     Management. Transactions on Network and Service Management,
> >     1(2), IEEE, 2004.
> >
> > [5] Malowidzki, M., "GetBulk Worth Fixing", Simple Times 10(1),
> >     December 2002.
> >
> > --
> > Juergen Schoenwaelder           Jacobs University Bremen gGmbH
> > Phone: +49 421 200 3587         Campus Ring 1, 28759 Bremen, Germany
> > Fax:   +49 421 200 3103         <http://www.jacobs-university.de/>
> > --
> > !! This message is brought to you via the `nmrg' mailing list.
> > !! Please do not reply to this message to unsubscribe. To
> > unsubscribe or adjust !! your settings, send a mail message
> > to <[email-masked]> !! or look at
> > https://mail.ibr.cs.tu-bs.de/mailman/listinfo/nmrg.
>
> --
> !! This message is brought to you via the `nmrg' mailing list.
> !! Please do not reply to this message to unsubscribe. To unsubscribe or
> adjust
> !! your settings, send a mail message to <[email-masked]>
> !! or look at https://mail.ibr.cs.tu-bs.de/mailman/listinfo/nmrg.
>

--0016e6409c0cdb6573048d09850e
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Hi,<br><br>This subject is interesting.  I have worked with the SNMP since 1990 and believe there might be a rather simple approach to (OID) compression.<br><br>It occurs to me that the target area to be compressed resides solely within the Scoped PDU. <br>
<br>Prior to sending, might there be benefit to simply run the Scoped PDU through a (lossless data) compression algorithm prior to transformation via privacy/encryption and calculation of authentication hashes?<br><br>Such an approach would be impartial to the frequency of OID prefixes or instances present.  It would simply condense the most frequent occurrences of subid patterns.<br>
<br>Regards,<br><br>Mark<br><br><div class="gmail_quote">On Sun, Jul 4, 2010 at 3:44 PM, David Harrington <span dir="ltr">&lt;<a href="mailto:[email-masked]">[email-masked]</a>&gt;</span> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Hi,<br>
<br>
I used to work on the Spectrum network management application (years<br>
ago).<br>
<br>
Juergen mentioned real world SNMP traces. My concern with these traces<br>
is with the sample networks, which tended to be academic. I think<br>
academic networks can be very different than commercial networks in<br>
the nature of their management practices. I know Spectrum sold for<br>
$100K+. I am not sure how many academic departments spend $100K+ on<br>
their management platforms.<br>
<br>
I am of the impression that how individual applications collect<br>
objects is likely to be the single biggest factor in SNMP performance.<br>
Much like software profiling, optimizing the collection of a MIB<br>
object that is never or seldom collected yields little benefit.<br>
Optimizing the collection of a single object that is collected very<br>
frequently will yield better results. In software profiling, it is<br>
generally better to optimize printf() than initialize_application().<br>
<br>
I suggest that a potentially fruitful area of research would be to<br>
study how the major SNMP NMS platforms behave before being extended<br>
for additional MIB modules. These platforms constantly check certain<br>
objects (like sysUpTime) to determine connectivity, detect<br>
discontinuities, update synchronization, monitor interface changes,<br>
etc. They check these routinely every few minutes throughout the day,<br>
for all devices. This generates much more traffic to optimize than<br>
operator-initiated reads using command line tools.<br>
<br>
In my experience, which is now pretty dated, most MIB module<br>
extensions are treated with custom polling instructions defined by the<br>
operator, so they vary by deployment. Otherwise, the extension MIBs<br>
are handled by MIB browser functionality - they are often read only on<br>
demand.<br>
<br>
If an NMS platform has add-on applications, the application will have<br>
certain objects it will monitor (possibly using traps) to determine<br>
when a significant event occurs, and then it will launch a series of<br>
special reads to gather more info on the event, possibly on a<br>
vendor/model-specific basis.<br>
<br>
I would recommend avoiding any proposals that make varbind order<br>
important. Spectrum optimizes (used to optimize) SNMPv1 reads by<br>
reordering the varbinds, and they use a polling engine that might<br>
combine requested polls from multiple applications into a single<br>
varbind to the device. (I don&#39;t know if they still do any of that<br>
given SNMPv3 user-based security.) You also might have threading and<br>
subagent issues.<br>
<br>
Spectrum also used to optimize data collection by determining, with<br>
administrator and historical input, how frequently an object needed to<br>
be actually read from the device, or whether an existing database<br>
value, possibly collected for a different application, that is less<br>
than &lt;x&gt; seconds old would suffice. Counters are the datatypes that<br>
most frequently must be refreshed by an actual read, so optimizing for<br>
counters would likely yield bigger benefits than optimizing static<br>
strings.<br>
<br>
As Juergen points out, in the face of other optimization techniques,<br>
OID compression might not be that important. It may be important for<br>
resource-contrained devices, but you&#39;ll want to be careful not to<br>
interfere with existing optimization techniques used by the<br>
application side.<br>
<br>
dbh<br>
<div><div></div><div class="h5"><br>
&gt; -----Original Message-----<br>
&gt; From: <a href="mailto:[email-masked]">[email-masked]</a><br>
&gt; [mailto:<a href="mailto:[email-masked]">[email-masked]</a>] On Behalf Of Juergen<br>
&gt; Schoenwaelder<br>
&gt; Sent: Wednesday, August 04, 2010 9:11 AM<br>
&gt; To: Hari Narayanan<br>
&gt; Cc: <a href="mailto:[email-masked]">[email-masked]</a><br>
&gt; Subject: Re: [nmrg] SNMP OID Compression<br>
&gt;<br>
&gt; On Tue, Aug 03, 2010 at 06:20:49PM +0200, Hari Narayanan wrote:<br>
&gt;<br>
&gt; &gt; OID Compression has been already discussed in two drafts ? Delta<br>
&gt; &gt; compression and Prefix compression. At present, I have worked on a<br>
<br>
&gt; &gt; simple form of prefix compression where all the requested<br>
&gt; objects are<br>
&gt; &gt; at the same level of MIB tree. First object in the request is<br>
fully<br>
&gt; &gt; specified in PDU varbind, rest of the objects are specified<br>
&gt; only with<br>
&gt; &gt; their respective suffix where they differ from the first object.<br>
&gt;<br>
&gt; It would be nice to know how much simpler your compression<br>
&gt; scheme is compared to delta compression, which works with<br>
&gt; arbitrary sequences of OIDs since it does not make any<br>
&gt; assumptions of the OIDs in the payload. It may look more<br>
&gt; complicated at first sight but I believe the code to<br>
&gt; implement delta compression really is not that difficult.<br>
&gt;<br>
&gt; &gt; To continue this investigation, I have also added<br>
&gt; additional code to<br>
&gt; &gt; Net SNMP 5.5 to support this simplified prefix compression. This<br>
&gt; &gt; enables me to measure code size, image size, response time, bytes<br>
&gt; &gt; saved, and CPU cycles required etc. The additional code required<br>
is<br>
&gt; &gt; about 400 lines (including comment and white space). These<br>
&gt; 400 lines<br>
&gt; &gt; of code are sufficient to enable both agent and application<br>
&gt; to support<br>
&gt; &gt; OID compression. The agent image is larger by about 4Kbytes.<br>
&gt; &gt; I am yet to measure the response time rigorously, but I notice<br>
that<br>
&gt; &gt; for getbulk request with large number of objects the<br>
&gt; response time is<br>
&gt; &gt; better with OID compression. The code that encodes objid takes<br>
less<br>
&gt; &gt; time because it has to code only the suffix part for all<br>
&gt; but the first<br>
&gt; &gt; object. It is now easy to compare the legacy code with the<br>
&gt; new code in<br>
&gt; &gt; ever aspect. There is only one API function to access and use OID<br>
&gt; &gt; compression.  There is also command line option to request OID<br>
&gt; &gt; compression.<br>
&gt;<br>
&gt; Several studies have reported that the time/cycles spent on<br>
&gt; BER encoding/decoding is not significant compared to the<br>
&gt; overall time/cycles it takes to process a request (see e.g.<br>
&gt; [4]). While this is already true for insecure versions of<br>
&gt; SNMP, if you add authentication and encryption,<br>
&gt; encoding/decoding overhead is not at all relevant. Back in<br>
&gt; the days (&gt; 10 years ago) when OID compression was more<br>
&gt; heavily discussed, the main objective was to pack more data<br>
&gt; into response PDUs, in particular allowing GetBulk requests<br>
&gt; to ship more varbinds in responses. Since there is an<br>
&gt; overshoot issue (unless the manager can estimate the number<br>
&gt; of instances and provide reasonable max-repetitions), a<br>
&gt; GetRange PDU was proposed as well, original idea in [5], more<br>
&gt; detailed description here:<br>
&gt;<br>
&gt; <a href="http://tools.ietf.org/html/draft-irtf-nmrg-snmp-getrange-00" target="_blank">http://tools.ietf.org/html/draft-irtf-nmrg-snmp-getrange-00</a><br>
&gt;<br>
&gt; Nowadays, interest is probably more coming from links that do<br>
&gt; have very small frame sizes and where fragmentation should be<br>
&gt; avoided (such as 802.15.4 networks) and where CPU cycles can<br>
&gt; potentially be saved when security is enabled.<br>
&gt;<br>
&gt; &gt; I am also doing a profiling on a large collection of MIBs (from<br>
250<br>
&gt; &gt; vendors) to understand how and where OID compression could be<br>
taken<br>
&gt; &gt; advantage off. This profiling includes average object<br>
&gt; depth, average<br>
&gt; &gt; object value size, percentage of table objects, number fixed size<br>
&gt; &gt; object, number of variable size objects, maximum depth, etc.<br>
&gt;<br>
&gt; Such studies based on an analysis of MIB modules have been<br>
&gt; done before [1]. There have also been efforts to collect<br>
&gt; real-world SNMP traces in order to analyze how deployed<br>
&gt; applications actually use SNMP [2]. A more general survey of<br>
&gt; SNMP performance studies can be found in [3].<br>
&gt;<br>
&gt; I am sending these references just in case they help to put<br>
&gt; your work into context.<br>
&gt;<br>
&gt; /js<br>
&gt;<br>
&gt; [1] J. Schonwalder: Characterization of SNMP MIB Modules. 9th<br>
&gt;     IFIP/IEEE International Symposium on Integrated Network<br>
&gt;     Management, pages 615-628, Nice, May 2005.<br>
&gt;<br>
&gt; [2] J. Schonwalder, A. Pras, M. Harvan, J. Schippers, R. van de<br>
Meent:<br>
&gt;     SNMP Traffic Analysis: Approaches, Tools, and First Results.<br>
10th<br>
&gt;     IFIP/IEEE International Symposium on Integrated Network<br>
&gt;     Management, Munich, May 2007.<br>
&gt;<br>
&gt; [3] L. Andrey, O. Festor, A. Lahmadi, A. Pras, J. Schonwalder:<br>
Survey<br>
&gt;     of SNMP Performance Analysis Studies. International Journal of<br>
&gt;     Network Management 19(6), Wiley, November 2009.<br>
&gt;<br>
&gt; [4] Aiko Pras, Thomas Drevers, Remco van de Meent, Dirk Quartel:<br>
&gt;     Comparing the Performance of SNMP and Web services-based<br>
&gt;     Management. Transactions on Network and Service Management,<br>
&gt;     1(2), IEEE, 2004.<br>
&gt;<br>
&gt; [5] Malowidzki, M., &quot;GetBulk Worth Fixing&quot;, Simple Times 10(1),<br>
&gt;     December 2002.<br>
&gt;<br>
&gt; --<br>
&gt; Juergen Schoenwaelder           Jacobs University Bremen gGmbH<br>
&gt; Phone: +49 421 200 3587         Campus Ring 1, 28759 Bremen, Germany<br>
&gt; Fax:   +49 421 200 3103         &lt;<a href="http://www.jacobs-university.de/" target="_blank">http://www.jacobs-university.de/</a>&gt;<br>
&gt; --<br>
&gt; !! This message is brought to you via the `nmrg&#39; mailing list.<br>
&gt; !! Please do not reply to this message to unsubscribe. To<br>
&gt; unsubscribe or adjust !! your settings, send a mail message<br>
&gt; to &lt;<a href="mailto:[email-masked]">[email-masked]</a>&gt; !! or look at<br>
&gt; <a href="https://mail.ibr.cs.tu-bs.de/mailman/listinfo/nmrg" target="_blank">https://mail.ibr.cs.tu-bs.de/mailman/listinfo/nmrg</a>.<br>
<br>
--<br>
!! This message is brought to you via the `nmrg&#39; mailing list.<br>
!! Please do not reply to this message to unsubscribe. To unsubscribe or adjust<br>
!! your settings, send a mail message to &lt;<a href="mailto:[email-masked]">[email-masked]</a>&gt;<br>
!! or look at <a href="https://mail.ibr.cs.tu-bs.de/mailman/listinfo/nmrg" target="_blank">https://mail.ibr.cs.tu-bs.de/mailman/listinfo/nmrg</a>.<br>
</div></div></blockquote></div><br>

--0016e6409c0cdb6573048d09850e--

From: "David Harrington" <[email-masked]>
To: "'Juergen Schoenwaelder'" <[email-masked]>, "'Hari Narayanan'" <[email-masked]>
Cc: [email-masked]
Subject: Re: [nmrg] SNMP OID Compression
Date: Wed, 04 Aug 2010 19:44:59 -0000

Hi,

I used to work on the Spectrum network management application (years
ago).

Juergen mentioned real world SNMP traces. My concern with these traces
is with the sample networks, which tended to be academic. I think
academic networks can be very different than commercial networks in
the nature of their management practices. I know Spectrum sold for
$100K+. I am not sure how many academic departments spend $100K+ on
their management platforms.

I am of the impression that how individual applications collect
objects is likely to be the single biggest factor in SNMP performance.
Much like software profiling, optimizing the collection of a MIB
object that is never or seldom collected yields little benefit.
Optimizing the collection of a single object that is collected very
frequently will yield better results. In software profiling, it is
generally better to optimize printf() than initialize_application().

I suggest that a potentially fruitful area of research would be to
study how the major SNMP NMS platforms behave before being extended
for additional MIB modules. These platforms constantly check certain
objects (like sysUpTime) to determine connectivity, detect
discontinuities, update synchronization, monitor interface changes,
etc. They check these routinely every few minutes throughout the day,
for all devices. This generates much more traffic to optimize than
operator-initiated reads using command line tools.

In my experience, which is now pretty dated, most MIB module
extensions are treated with custom polling instructions defined by the
operator, so they vary by deployment. Otherwise, the extension MIBs
are handled by MIB browser functionality - they are often read only on
demand.

If an NMS platform has add-on applications, the application will have
certain objects it will monitor (possibly using traps) to determine
when a significant event occurs, and then it will launch a series of
special reads to gather more info on the event, possibly on a
vendor/model-specific basis.

I would recommend avoiding any proposals that make varbind order
important. Spectrum optimizes (used to optimize) SNMPv1 reads by
reordering the varbinds, and they use a polling engine that might
combine requested polls from multiple applications into a single
varbind to the device. (I don't know if they still do any of that
given SNMPv3 user-based security.) You also might have threading and
subagent issues.

Spectrum also used to optimize data collection by determining, with
administrator and historical input, how frequently an object needed to
be actually read from the device, or whether an existing database
value, possibly collected for a different application, that is less
than <x> seconds old would suffice. Counters are the datatypes that
most frequently must be refreshed by an actual read, so optimizing for
counters would likely yield bigger benefits than optimizing static
strings.

As Juergen points out, in the face of other optimization techniques,
OID compression might not be that important. It may be important for
resource-contrained devices, but you'll want to be careful not to
interfere with existing optimization techniques used by the
application side.

dbh

> -----Original Message-----
> From: [email-masked] 
> [mailto:[email-masked]] On Behalf Of Juergen 
> Schoenwaelder
> Sent: Wednesday, August 04, 2010 9:11 AM
> To: Hari Narayanan
> Cc: [email-masked]
> Subject: Re: [nmrg] SNMP OID Compression
> 
> On Tue, Aug 03, 2010 at 06:20:49PM +0200, Hari Narayanan wrote:
>  
> > OID Compression has been already discussed in two drafts ? Delta 
> > compression and Prefix compression. At present, I have worked on a

> > simple form of prefix compression where all the requested 
> objects are 
> > at the same level of MIB tree. First object in the request is
fully 
> > specified in PDU varbind, rest of the objects are specified 
> only with 
> > their respective suffix where they differ from the first object.
> 
> It would be nice to know how much simpler your compression 
> scheme is compared to delta compression, which works with 
> arbitrary sequences of OIDs since it does not make any 
> assumptions of the OIDs in the payload. It may look more 
> complicated at first sight but I believe the code to 
> implement delta compression really is not that difficult.
> 
> > To continue this investigation, I have also added 
> additional code to 
> > Net SNMP 5.5 to support this simplified prefix compression. This 
> > enables me to measure code size, image size, response time, bytes 
> > saved, and CPU cycles required etc. The additional code required
is 
> > about 400 lines (including comment and white space). These 
> 400 lines 
> > of code are sufficient to enable both agent and application 
> to support 
> > OID compression. The agent image is larger by about 4Kbytes.
> > I am yet to measure the response time rigorously, but I notice
that 
> > for getbulk request with large number of objects the 
> response time is 
> > better with OID compression. The code that encodes objid takes
less 
> > time because it has to code only the suffix part for all 
> but the first 
> > object. It is now easy to compare the legacy code with the 
> new code in 
> > ever aspect. There is only one API function to access and use OID 
> > compression.  There is also command line option to request OID 
> > compression.
> 
> Several studies have reported that the time/cycles spent on 
> BER encoding/decoding is not significant compared to the 
> overall time/cycles it takes to process a request (see e.g. 
> [4]). While this is already true for insecure versions of 
> SNMP, if you add authentication and encryption, 
> encoding/decoding overhead is not at all relevant. Back in 
> the days (> 10 years ago) when OID compression was more 
> heavily discussed, the main objective was to pack more data 
> into response PDUs, in particular allowing GetBulk requests 
> to ship more varbinds in responses. Since there is an 
> overshoot issue (unless the manager can estimate the number 
> of instances and provide reasonable max-repetitions), a 
> GetRange PDU was proposed as well, original idea in [5], more 
> detailed description here:
> 
> http://tools.ietf.org/html/draft-irtf-nmrg-snmp-getrange-00
> 
> Nowadays, interest is probably more coming from links that do 
> have very small frame sizes and where fragmentation should be 
> avoided (such as 802.15.4 networks) and where CPU cycles can 
> potentially be saved when security is enabled.
> 
> > I am also doing a profiling on a large collection of MIBs (from
250
> > vendors) to understand how and where OID compression could be
taken 
> > advantage off. This profiling includes average object 
> depth, average 
> > object value size, percentage of table objects, number fixed size 
> > object, number of variable size objects, maximum depth, etc.
> 
> Such studies based on an analysis of MIB modules have been 
> done before [1]. There have also been efforts to collect 
> real-world SNMP traces in order to analyze how deployed 
> applications actually use SNMP [2]. A more general survey of 
> SNMP performance studies can be found in [3].
> 
> I am sending these references just in case they help to put 
> your work into context.
> 
> /js
> 
> [1] J. Schonwalder: Characterization of SNMP MIB Modules. 9th
>     IFIP/IEEE International Symposium on Integrated Network
>     Management, pages 615-628, Nice, May 2005.
> 
> [2] J. Schonwalder, A. Pras, M. Harvan, J. Schippers, R. van de
Meent:
>     SNMP Traffic Analysis: Approaches, Tools, and First Results.
10th
>     IFIP/IEEE International Symposium on Integrated Network
>     Management, Munich, May 2007.
> 
> [3] L. Andrey, O. Festor, A. Lahmadi, A. Pras, J. Schonwalder:
Survey
>     of SNMP Performance Analysis Studies. International Journal of
>     Network Management 19(6), Wiley, November 2009.
> 
> [4] Aiko Pras, Thomas Drevers, Remco van de Meent, Dirk Quartel:
>     Comparing the Performance of SNMP and Web services-based
>     Management. Transactions on Network and Service Management,
>     1(2), IEEE, 2004.
> 
> [5] Malowidzki, M., "GetBulk Worth Fixing", Simple Times 10(1),
>     December 2002.
> 
> -- 
> Juergen Schoenwaelder           Jacobs University Bremen gGmbH
> Phone: +49 421 200 3587         Campus Ring 1, 28759 Bremen, Germany
> Fax:   +49 421 200 3103         <http://www.jacobs-university.de/>
> --
> !! This message is brought to you via the `nmrg' mailing list.
> !! Please do not reply to this message to unsubscribe. To 
> unsubscribe or adjust !! your settings, send a mail message 
> to <[email-masked]> !! or look at 
> https://mail.ibr.cs.tu-bs.de/mailman/listinfo/nmrg.

Date: Wed, 4 Aug 2010 14:54:32 +0530
From: Hari Narayanan <[email-masked]>
To: Juergen Schoenwaelder <[email-masked]>,  Hari Narayanan <[email-masked]>, "[email-masked]" <[email-masked]>
Subject: Re: [nmrg] SNMP OID Compression

--0022152d6eb1843a55048cfc00e3
Content-Type: text/plain; charset=ISO-8859-1

Thanks JS,
I do not expect the complexity of Delta compression to be significantly
different. Now I understand Net SNMP better, I plan to do that in the near
future. The additional complexity comes only in coding compression for three
cases: Single Identifier, Range of Identifiers, and Truncation. I will have
some concrete answers for you when I complete it.

Keeping the compression overhead is a requirement to justify the other
benefits. The main objective in my opinion is to save packets instead of
bytes.  In the worst case, with OID compression feature, we reduce SNMP
traffic volume. In the best case, we also relieve control processor  from
management activity. OID Compression can reduce the CPU cycles at an agent
to 50% (assuming one response does the job of two) for certain requests. For
a typical table object at a depth of 14 if the response contains about 60
objects then one request will do the job of 2. This is perhaps, the real
selling point for OID compression, though it requires the developers to be
aware of this feature and table profile.

Let me look at the references that you listed.

Regards .... Hari T S Narayanan

On Wed, Aug 4, 2010 at 12:41 PM, Juergen Schoenwaelder <
[email-masked]> wrote:

> On Tue, Aug 03, 2010 at 06:20:49PM +0200, Hari Narayanan wrote:
>
> > OID Compression has been already discussed in two drafts ? Delta
> > compression and Prefix compression. At present, I have worked on a
> > simple form of prefix compression where all the requested objects
> > are at the same level of MIB tree. First object in the request is
> > fully specified in PDU varbind, rest of the objects are specified
> > only with their respective suffix where they differ from the first
> > object.
>
> It would be nice to know how much simpler your compression scheme is
> compared to delta compression, which works with arbitrary sequences of
> OIDs since it does not make any assumptions of the OIDs in the
> payload. It may look more complicated at first sight but I believe the
> code to implement delta compression really is not that difficult.
>
> > To continue this investigation, I have also added additional code to
> > Net SNMP 5.5 to support this simplified prefix compression. This
> > enables me to measure code size, image size, response time, bytes
> > saved, and CPU cycles required etc. The additional code required is
> > about 400 lines (including comment and white space). These 400 lines
> > of code are sufficient to enable both agent and application to
> > support OID compression. The agent image is larger by about 4Kbytes.
> > I am yet to measure the response time rigorously, but I notice that
> > for getbulk request with large number of objects the response time
> > is better with OID compression. The code that encodes objid takes
> > less time because it has to code only the suffix part for all but
> > the first object. It is now easy to compare the legacy code with the
> > new code in ever aspect. There is only one API function to access
> > and use OID compression.  There is also command line option to
> > request OID compression.
>
> Several studies have reported that the time/cycles spent on BER
> encoding/decoding is not significant compared to the overall
> time/cycles it takes to process a request (see e.g. [4]). While this
> is already true for insecure versions of SNMP, if you add
> authentication and encryption, encoding/decoding overhead is not at
> all relevant. Back in the days (> 10 years ago) when OID compression
> was more heavily discussed, the main objective was to pack more data
> into response PDUs, in particular allowing GetBulk requests to ship
> more varbinds in responses. Since there is an overshoot issue (unless
> the manager can estimate the number of instances and provide
> reasonable max-repetitions), a GetRange PDU was proposed as well,
> original idea in [5], more detailed description here:
>
> http://tools.ietf.org/html/draft-irtf-nmrg-snmp-getrange-00
>
> Nowadays, interest is probably more coming from links that do have
> very small frame sizes and where fragmentation should be avoided (such
> as 802.15.4 networks) and where CPU cycles can potentially be saved
> when security is enabled.
>
> > I am also doing a profiling on a large collection of MIBs (from 250
> > vendors) to understand how and where OID compression could be taken
> > advantage off. This profiling includes average object depth, average
> > object value size, percentage of table objects, number fixed size
> > object, number of variable size objects, maximum depth, etc.
>
> Such studies based on an analysis of MIB modules have been done before
> [1]. There have also been efforts to collect real-world SNMP traces in
> order to analyze how deployed applications actually use SNMP [2]. A
> more general survey of SNMP performance studies can be found in [3].
>
> I am sending these references just in case they help to put your work
> into context.
>
> /js
>
> [1] J. Schonwalder: Characterization of SNMP MIB Modules. 9th
>    IFIP/IEEE International Symposium on Integrated Network
>    Management, pages 615-628, Nice, May 2005.
>
> [2] J. Schonwalder, A. Pras, M. Harvan, J. Schippers, R. van de Meent:
>    SNMP Traffic Analysis: Approaches, Tools, and First Results. 10th
>    IFIP/IEEE International Symposium on Integrated Network
>    Management, Munich, May 2007.
>
> [3] L. Andrey, O. Festor, A. Lahmadi, A. Pras, J. Schonwalder: Survey
>    of SNMP Performance Analysis Studies. International Journal of
>    Network Management 19(6), Wiley, November 2009.
>
> [4] Aiko Pras, Thomas Drevers, Remco van de Meent, Dirk Quartel:
>    Comparing the Performance of SNMP and Web services-based
>    Management. Transactions on Network and Service Management,
>    1(2), IEEE, 2004.
>
> [5] Malowidzki, M., "GetBulk Worth Fixing", Simple Times 10(1),
>    December 2002.
>
> --
> Juergen Schoenwaelder           Jacobs University Bremen gGmbH
> Phone: +49 421 200 3587         Campus Ring 1, 28759 Bremen, Germany
> Fax:   +49 421 200 3103         <http://www.jacobs-university.de/>
>

--0022152d6eb1843a55048cfc00e3
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Thanks JS,<div>I do not expect the complexity of Delta compression to be significantly different. Now I understand Net SNMP better, I plan to do that in the near future. The additional complexity comes only in coding compression for three cases: Single Identifier, Range of Identifiers, and Truncation. I will have some concrete answers for you when I complete it.</div>
<div><br></div><div>Keeping the compression overhead is a requirement to justify the other benefits. The main objective in my opinion is to save packets instead of bytes.  In the worst case, with OID compression feature, we reduce SNMP traffic volume. In the best case, we also relieve control processor  from management activity. OID Compression can reduce the CPU cycles at an agent to 50% (assuming one response does the job of two) for certain requests. For a typical table object at a depth of 14 if the response contains about 60 objects then one request will do the job of 2. This is perhaps, the real selling point for OID compression, though it requires the developers to be aware of this feature and table profile.</div>
<div><br></div><div>Let me look at the references that you listed.</div><div><br></div><div>Regards .... Hari T S Narayanan</div><div><br><div class="gmail_quote">On Wed, Aug 4, 2010 at 12:41 PM, Juergen Schoenwaelder <span dir="ltr">&lt;<a href="mailto:[email-masked]">[email-masked]</a>&gt;</span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><div class="im">On Tue, Aug 03, 2010 at 06:20:49PM +0200, Hari Narayanan wrote:<br>
<br>
&gt; OID Compression has been already discussed in two drafts ? Delta<br>
&gt; compression and Prefix compression. At present, I have worked on a<br>
&gt; simple form of prefix compression where all the requested objects<br>
&gt; are at the same level of MIB tree. First object in the request is<br>
&gt; fully specified in PDU varbind, rest of the objects are specified<br>
&gt; only with their respective suffix where they differ from the first<br>
&gt; object.<br>
<br>
</div>It would be nice to know how much simpler your compression scheme is<br>
compared to delta compression, which works with arbitrary sequences of<br>
OIDs since it does not make any assumptions of the OIDs in the<br>
payload. It may look more complicated at first sight but I believe the<br>
code to implement delta compression really is not that difficult.<br>
<div class="im"><br>
&gt; To continue this investigation, I have also added additional code to<br>
&gt; Net SNMP 5.5 to support this simplified prefix compression. This<br>
&gt; enables me to measure code size, image size, response time, bytes<br>
&gt; saved, and CPU cycles required etc. The additional code required is<br>
&gt; about 400 lines (including comment and white space). These 400 lines<br>
&gt; of code are sufficient to enable both agent and application to<br>
&gt; support OID compression. The agent image is larger by about 4Kbytes.<br>
&gt; I am yet to measure the response time rigorously, but I notice that<br>
&gt; for getbulk request with large number of objects the response time<br>
&gt; is better with OID compression. The code that encodes objid takes<br>
&gt; less time because it has to code only the suffix part for all but<br>
&gt; the first object. It is now easy to compare the legacy code with the<br>
&gt; new code in ever aspect. There is only one API function to access<br>
&gt; and use OID compression.  There is also command line option to<br>
&gt; request OID compression.<br>
<br>
</div>Several studies have reported that the time/cycles spent on BER<br>
encoding/decoding is not significant compared to the overall<br>
time/cycles it takes to process a request (see e.g. [4]). While this<br>
is already true for insecure versions of SNMP, if you add<br>
authentication and encryption, encoding/decoding overhead is not at<br>
all relevant. Back in the days (&gt; 10 years ago) when OID compression<br>
was more heavily discussed, the main objective was to pack more data<br>
into response PDUs, in particular allowing GetBulk requests to ship<br>
more varbinds in responses. Since there is an overshoot issue (unless<br>
the manager can estimate the number of instances and provide<br>
reasonable max-repetitions), a GetRange PDU was proposed as well,<br>
original idea in [5], more detailed description here:<br>
<br>
<a href="http://tools.ietf.org/html/draft-irtf-nmrg-snmp-getrange-00" target="_blank">http://tools.ietf.org/html/draft-irtf-nmrg-snmp-getrange-00</a><br>
<br>
Nowadays, interest is probably more coming from links that do have<br>
very small frame sizes and where fragmentation should be avoided (such<br>
as 802.15.4 networks) and where CPU cycles can potentially be saved<br>
when security is enabled.<br>
<div class="im"><br>
&gt; I am also doing a profiling on a large collection of MIBs (from 250<br>
&gt; vendors) to understand how and where OID compression could be taken<br>
&gt; advantage off. This profiling includes average object depth, average<br>
&gt; object value size, percentage of table objects, number fixed size<br>
&gt; object, number of variable size objects, maximum depth, etc.<br>
<br>
</div>Such studies based on an analysis of MIB modules have been done before<br>
[1]. There have also been efforts to collect real-world SNMP traces in<br>
order to analyze how deployed applications actually use SNMP [2]. A<br>
more general survey of SNMP performance studies can be found in [3].<br>
<br>
I am sending these references just in case they help to put your work<br>
into context.<br>
<br>
/js<br>
<br>
[1] J. Schonwalder: Characterization of SNMP MIB Modules. 9th<br>
    IFIP/IEEE International Symposium on Integrated Network<br>
    Management, pages 615-628, Nice, May 2005.<br>
<br>
[2] J. Schonwalder, A. Pras, M. Harvan, J. Schippers, R. van de Meent:<br>
    SNMP Traffic Analysis: Approaches, Tools, and First Results. 10th<br>
    IFIP/IEEE International Symposium on Integrated Network<br>
    Management, Munich, May 2007.<br>
<br>
[3] L. Andrey, O. Festor, A. Lahmadi, A. Pras, J. Schonwalder: Survey<br>
    of SNMP Performance Analysis Studies. International Journal of<br>
    Network Management 19(6), Wiley, November 2009.<br>
<br>
[4] Aiko Pras, Thomas Drevers, Remco van de Meent, Dirk Quartel:<br>
    Comparing the Performance of SNMP and Web services-based<br>
    Management. Transactions on Network and Service Management,<br>
    1(2), IEEE, 2004.<br>
<br>
[5] Malowidzki, M., &quot;GetBulk Worth Fixing&quot;, Simple Times 10(1),<br>
    December 2002.<br>
<font color="#888888"><br>
--<br>
Juergen Schoenwaelder           Jacobs University Bremen gGmbH<br>
Phone: +49 421 200 3587         Campus Ring 1, 28759 Bremen, Germany<br>
Fax:   +49 421 200 3103         &lt;<a href="http://www.jacobs-university.de/" target="_blank">http://www.jacobs-university.de/</a>&gt;<br>
</font></blockquote></div><br></div>

--0022152d6eb1843a55048cfc00e3--

Date: Wed, 4 Aug 2010 09:11:22 +0200
From: Juergen Schoenwaelder <[email-masked]>
To: Hari Narayanan <[email-masked]>
Mail-Followup-To: Hari Narayanan <[email-masked]>, "[email-masked]" <[email-masked]>
Cc: "[email-masked]" <[email-masked]>
Subject: Re: [nmrg] SNMP OID Compression
Reply-To: Juergen Schoenwaelder <[email-masked]>

On Tue, Aug 03, 2010 at 06:20:49PM +0200, Hari Narayanan wrote:
 
> OID Compression has been already discussed in two drafts ? Delta
> compression and Prefix compression. At present, I have worked on a
> simple form of prefix compression where all the requested objects
> are at the same level of MIB tree. First object in the request is
> fully specified in PDU varbind, rest of the objects are specified
> only with their respective suffix where they differ from the first
> object.

It would be nice to know how much simpler your compression scheme is
compared to delta compression, which works with arbitrary sequences of
OIDs since it does not make any assumptions of the OIDs in the
payload. It may look more complicated at first sight but I believe the
code to implement delta compression really is not that difficult.

> To continue this investigation, I have also added additional code to
> Net SNMP 5.5 to support this simplified prefix compression. This
> enables me to measure code size, image size, response time, bytes
> saved, and CPU cycles required etc. The additional code required is
> about 400 lines (including comment and white space). These 400 lines
> of code are sufficient to enable both agent and application to
> support OID compression. The agent image is larger by about 4Kbytes.
> I am yet to measure the response time rigorously, but I notice that
> for getbulk request with large number of objects the response time
> is better with OID compression. The code that encodes objid takes
> less time because it has to code only the suffix part for all but
> the first object. It is now easy to compare the legacy code with the
> new code in ever aspect. There is only one API function to access
> and use OID compression.  There is also command line option to
> request OID compression.

Several studies have reported that the time/cycles spent on BER
encoding/decoding is not significant compared to the overall
time/cycles it takes to process a request (see e.g. [4]). While this
is already true for insecure versions of SNMP, if you add
authentication and encryption, encoding/decoding overhead is not at
all relevant. Back in the days (> 10 years ago) when OID compression
was more heavily discussed, the main objective was to pack more data
into response PDUs, in particular allowing GetBulk requests to ship
more varbinds in responses. Since there is an overshoot issue (unless
the manager can estimate the number of instances and provide
reasonable max-repetitions), a GetRange PDU was proposed as well,
original idea in [5], more detailed description here:

http://tools.ietf.org/html/draft-irtf-nmrg-snmp-getrange-00

Nowadays, interest is probably more coming from links that do have
very small frame sizes and where fragmentation should be avoided (such
as 802.15.4 networks) and where CPU cycles can potentially be saved
when security is enabled.

> I am also doing a profiling on a large collection of MIBs (from 250
> vendors) to understand how and where OID compression could be taken
> advantage off. This profiling includes average object depth, average
> object value size, percentage of table objects, number fixed size
> object, number of variable size objects, maximum depth, etc.

Such studies based on an analysis of MIB modules have been done before
[1]. There have also been efforts to collect real-world SNMP traces in
order to analyze how deployed applications actually use SNMP [2]. A
more general survey of SNMP performance studies can be found in [3].

I am sending these references just in case they help to put your work
into context.

/js

[1] J. Schonwalder: Characterization of SNMP MIB Modules. 9th
    IFIP/IEEE International Symposium on Integrated Network
    Management, pages 615-628, Nice, May 2005.

[2] J. Schonwalder, A. Pras, M. Harvan, J. Schippers, R. van de Meent:
    SNMP Traffic Analysis: Approaches, Tools, and First Results. 10th
    IFIP/IEEE International Symposium on Integrated Network
    Management, Munich, May 2007.

[3] L. Andrey, O. Festor, A. Lahmadi, A. Pras, J. Schonwalder: Survey
    of SNMP Performance Analysis Studies. International Journal of
    Network Management 19(6), Wiley, November 2009.

[4] Aiko Pras, Thomas Drevers, Remco van de Meent, Dirk Quartel:
    Comparing the Performance of SNMP and Web services-based
    Management. Transactions on Network and Service Management,
    1(2), IEEE, 2004.

[5] Malowidzki, M., "GetBulk Worth Fixing", Simple Times 10(1),
    December 2002.

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1, 28759 Bremen, Germany
Fax:   +49 421 200 3103         <http://www.jacobs-university.de/>

Date: Tue, 3 Aug 2010 21:50:49 +0530
From: Hari Narayanan <[email-masked]>
To: [email-masked]
Subject: [nmrg] SNMP OID Compression

--0016361e87ce5ff6da048cedb39d
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: quoted-printable

I have been working on the OID compression for the last few months. The
objective of my work is to understand the overall benefit of it.  If the OID
Compression is simple then it could reduce the management traffic, improve
the response time, and more crucially reduce the CPU cycles required at the
Agent side. In certain cases a single compressed request-response can do the
job of 3 request-response.

OID Compression has been already discussed in two drafts –
*Delta*compression and
*Prefix* compression. At present, I have worked on a simple form of prefix
compression where all the requested objects are at the same level of MIB
tree. First object in the request is fully specified in PDU varbind, rest of
the objects are specified only with their respective suffix where they
differ from the first object.

A simple analytical model is developed to understand the potential benefits
and their limitations as functions of object depth (in MIB tree), number of
objects requested, and their relationship, and object value size. This model
has been submitted for review. I am not sure if I can share the content of
this paper here; I will find out from the journal where I submitted this
work.

To continue this investigation, I have also added additional code to Net
SNMP 5.5 to support this simplified prefix compression. This enables me to
measure code size, image size, response time, bytes saved, and CPU cycles
required etc. The additional code required is about 400 lines (including
comment and white space). These 400 lines of code are sufficient to enable
both agent and application to support OID compression. The agent image is
larger by about 4Kbytes.  I am yet to measure the response time rigorously,
but I notice that for getbulk request with large number of objects the
response time is better with OID compression. The code that encodes objid
takes less time because it has to code only the suffix part for all but the
first object. It is now easy to compare the legacy code with the new code in
ever aspect. There is only one API function to access and use OID
compression.  There is also command line option to request OID compression.

I am also doing a profiling on a large collection of MIBs  (from 250
vendors) to understand how and where OID compression could be taken
advantage off. This profiling includes average object depth, average object
value size, percentage of table objects, number fixed size object, number of
variable size objects, maximum depth, etc.

I will be more than happy to share the above code and other findings with
you if there is interest, let me know.

In summary, I see there are benefits with simple but non-trivial effort and
no run time overhead.

Regards … Hari TS Narayanan

Netprowise

Chennai

India

--0016361e87ce5ff6da048cedb39d
Content-Type: text/html; charset=windows-1252
Content-Transfer-Encoding: quoted-printable

<p class="MsoNormal" style="text-align:justify">I have been working on the OID
compression for the last few months. The objective of my work is to understand
the overall benefit of it.<span style="mso-spacerun:yes">  </span>If the OID
Compression is simple then it could reduce the management traffic, improve the
response time, and more crucially reduce the CPU cycles required at the Agent
side. In certain cases a single compressed request-response can do the job of 3
request-response. </p>

<p class="MsoNormal" style="text-align:justify"> </p>

<p class="MsoNormal" style="text-align:justify">OID Compression has been already
discussed in two drafts – <i style="mso-bidi-font-style:normal">Delta</i>
compression and <i style="mso-bidi-font-style:normal">Prefix</i> compression.
At present, I have worked on a simple form of prefix compression where all the
requested objects are at the same level of MIB tree. First object in the
request is fully specified in PDU varbind, rest of the objects are specified only
with their respective suffix where they differ from the first object. </p>

<p class="MsoNormal" style="text-align:justify"> </p>

<p class="MsoNormal" style="text-align:justify">A simple analytical model is
developed to understand the potential benefits and their limitations as
functions of object depth (in MIB tree), number of objects requested, and their
relationship, and object value size. This model has been submitted for review.
I am not sure if I can share the content of this paper here; I will find out
from the journal where I submitted this work. </p>

<p class="MsoNormal" style="text-align:justify"> </p>

<p class="MsoNormal" style="text-align:justify">To continue this investigation, I
have also added additional code to Net SNMP 5.5 to support this simplified
prefix compression. This enables me to measure code size, image size, response
time, bytes saved, and CPU cycles required etc. The additional code required is
about 400 lines (including comment and white space). These 400 lines of code are
sufficient to enable both agent and application to support OID compression. The
agent image is larger by about 4Kbytes.<span style="mso-spacerun:yes"> 
</span>I am yet to measure the response time rigorously, but I notice that for
getbulk request with large number of objects the response time is better with
OID compression. The code that encodes objid takes less time because it has to
code only the suffix part for all but the first object. It is now easy to compare the legacy code with the new code in ever aspect. There is only one API
function to access and use OID compression. <span style="mso-spacerun:yes"> There is also command line option to request OID compression.</span></p>

<p class="MsoNormal" style="text-align:justify"> </p>

<p class="MsoNormal" style="text-align:justify">I am also doing a profiling on a
large collection of MIBs <span style="mso-spacerun:yes"> </span>(from 250
vendors) to understand how and where OID compression could be taken advantage
off. This profiling includes average object depth, average object value size,
percentage of table objects, number fixed size object, number of variable size
objects, maximum depth, etc.</p>

<p class="MsoNormal" style="text-align:justify"> </p>

<p class="MsoNormal" style="text-align:justify">I will be more than happy to
share the above code and other findings with you if there is interest, let me
know. </p>

<p class="MsoNormal" style="text-align:justify"> </p>

<p class="MsoNormal" style="text-align:justify">In summary, I see there are benefits
with simple but non-trivial effort and no run time overhead.</p>

<p class="MsoNormal" style="text-align:justify"> </p>

<p class="MsoNormal" style="text-align:justify">Regards … Hari TS Narayanan</p><p class="MsoNormal" style="text-align:justify"><br></p><p class="MsoNormal" style="text-align:justify">Netprowise</p><p class="MsoNormal" style="text-align:justify">
Chennai</p><p class="MsoNormal" style="text-align:justify">India </p>

--0016361e87ce5ff6da048cedb39d--