Subject: [core] ABNF help

From: Zach Shelby <[email-masked]>
Date: Tue, 3 May 2011 11:32:27 +0300

I am trying to add decimal range limitations to the core-link-format ct= and sz= attributes. If anyone knows the best way to do that in ABNF I would appreciate some help. Current ABNF:

   link-extension    = ( "rt" "=" quoted-string )
   link-extension    = ( "if" "=" quoted-string )
   link-extension    = ( "ct" "=" integer )
   link-extension    = ( "sz" "=" integer )
   integer           = 1*DIGIT

Needs to be updated so that ct has a digit range of 0-65535 and sz has a digit range of 0-4294967295. 

Thanks,
Zach

-- 
Zach Shelby, Chief Nerd, Sensinode Ltd.
http://zachshelby.org  - My blog "On the Internet of Things"
http://6lowpan.net - My book "6LoWPAN: The Wireless Embedded Internet"
Mobile: +358 40 7796297

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 3 May 2011 10:52:51 +0200

On May 3, 2011, at 10:32, Zach Shelby wrote:

> Needs to be updated so that ct has a digit range of 0-65535

One way to do this is shown in a number of RFC including RFC 3986:

      IPv4address = dec-octet "." dec-octet "." dec-octet "." dec-octet

      dec-octet   = DIGIT                 ; 0-9
                  / %x31-39 DIGIT         ; 10-99
                  / "1" 2DIGIT            ; 100-199
                  / "2" %x30-34 DIGIT     ; 200-249
                  / "25" %x30-35          ; 250-255

This extends in a straightforward way to any range of numbers.
(Note that the technique used also nicely takes care of disallowing leading zeros.)

Is it reasonable to go to the trouble to encode this range limit in the ABNF?
(Doesn't make a difference to a small implementation, which will probably do the range check in a different way.)

> and sz has a digit range of 0-4294967295. 

It is not even clear to me that we have a natural limit of < 4 GiB.
Why not leave that open?  If a node has an mp4 of the royal wedding, it might not fit anyway :-)

In summary, I certainly would not bother to make this change for sz, and I probably wouldn't for ct.

Gruesse, Carsten

---

From: Alexey Melnikov <[email-masked]>
Date: Tue, 03 May 2011 09:58:50 +0100

Hi Carsten,

>(Note that the technique used also nicely takes care of disallowing leading zeros.)
Right.

Stating restrictions in an ABNF comment is also acceptable, IMHO.

>In summary, I certainly would not bother to make this change for sz, and I probably wouldn't for ct.
If I were to implement this and I were to store these values as integers 
(as opposed to strings), I would need to know the possible upper limit.

---

From: Zach Shelby <[email-masked]>
Date: Tue, 3 May 2011 12:01:13 +0300

> (Doesn't make a difference to a small implementation, which will probably do the range check in a different way.)

Probably true. I was just working on Alexey's link-format review comments and this was one of them. I can just include the range in the text description of course.

> Why not leave that open?  If a node has an mp4 of the royal wedding, it might not fit anyway :-)

So you are a royalist? :-) You are probably right, we can just leave the range for sz open.

> In summary, I certainly would not bother to make this change for sz, and I probably wouldn't for ct.

Regarding ct I would prefer to keep the 0-65535 range in the text as this is the valid range for the CoAP media type code. 

Zach

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 3 May 2011 11:32:26 +0200

On May 3, 2011, at 10:58, Alexey Melnikov wrote:

> If I were to implement this and I were to store these values as integers (as opposed to strings), I would need to know the possible upper limit.

This depends on what kind of integers you have...

But seriously, for a producer of this format, the size of the resources will fit nicely with its integer processing capabilities.  For a consumer, the only thing that may be needed is one representation for "big", i.e. of a size that is so large that it is no longer relevant for this kind of implementation.  So not having a limit does not hurt.

Not hard-wiring one limit is also the right thing to do, as it prevents Content-Size-style disasters when that one limit turns out to be overtaken by events.

Note that link-format may be used beyond CoAP, so having videos within the resources is not as unrealistic as one may think from the context of this WG.

Gruesse, Carsten

---

From: Alexey Melnikov <[email-masked]>
Date: Tue, 03 May 2011 10:38:17 +0100

>This depends on what kind of integers you have...
Usual 32bit/64bit C types which I don't want to overflow, if I am not 
careful.

>But seriously, for a producer of this format, the size of the resources will fit nicely with its integer processing capabilities.  For a consumer, the only thing that may be needed is one representa…
I don't want a naive consumer to dump core because the producer decided 
to generate a 128bit size. And guess what, some malicious producers 
would do this on purpose to deny service.

In IMAP I know that all sizes are limited to 32bit. If you want more, 
you need to defined & negotiate an extension.

>Not hard-wiring one limit is also the right thing to do, as it prevents Content-Size-style disasters when that one limit turns out to be overtaken by events.
If you choose your size carefully, I doubt that this will happen in our 
lifetime.

>Note that link-format may be used beyond CoAP, so having videos within the resources is not as unrealistic as one may think from the context of this WG.
Well, Ok. But I don't think this changes the gist of my argument.

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 3 May 2011 11:54:49 +0200

> I don't want a naive consumer to dump core because the producer decided to generate a 128bit size. And guess what, some malicious producers would do this on purpose to deny service.

OK, so let's add something like:

	Note that there is no defined upper limit to the value of the sz attribute.
	Implementations MUST be prepared to accept large values.  
        Implementation note: One implementation strategy is to convert any value larger than a reasonable size limit for this implementation to a special value "Big", which in further processing would indicate that a size value was given that was so big that it cannot be processed by this implementation.

> In IMAP I know that all sizes are limited to 32bit. If you want more, you need to defined & negotiate an extension.

But in IMAP you actually need to process the values, as this is a protocol that allows you to operate on the objects.  Link-format is descriptive only (and there is no IMAP connection to negotiate anything on, either).

> If you choose your size carefully, I doubt that this will happen in our lifetime.

1208925819614629174706175 (for an 80-bit integer)?
I'm not sure that helps anyone...

Gruesse, Carsten

---

From: Zach Shelby <[email-masked]>
Date: Tue, 3 May 2011 13:36:18 +0300

>        Implementation note: One implementation strategy is to convert any value larger than a reasonable size limit for this implementation to a special value "Big", which in further processing wou…

I would be OK with text like this. A constrained implementation would work as described anyways. 

Zach

---

From: Alexey Melnikov <[email-masked]>
Date: Tue, 03 May 2011 11:51:18 +0100

>        Implementation note: One implementation strategy is to convert any value larger than a reasonable size limit for this implementation to a special value "Big", which in further processing wou…
I can live with this.

Can the document can also contain an example with a big value, because 
implementors frequently implement from examples...

>1208925819614629174706175 (for an 80-bit integer)?
Good example :-).

---

From: Zach Shelby <[email-masked]>
Date: Tue, 3 May 2011 14:06:09 +0300

On May 3, 2011, at 1:51 PM, Alexey Melnikov wrote:

>> 	Implementations MUST be prepared to accept large values.         Implementation note: One implementation strategy is to convert any value larger than a reasonable size limit for this implementation to a special value "Big", which in further processing would indicate that a size value was given that was so big that it cannot be processed by this implementation.
>> 
> I can live with this.
> 
> Can the document can also contain an example with a big value, because implementors frequently implement from examples...

Done.

Zach

---

From: Cullen Jennings <[email-masked]>
Date: Thu, 5 May 2011 11:46:52 -0600

> So you are a royalist? :-) You are probably right, we can just leave the range for sz open.

I don't care if you set the limit at 128 bits but I really prefer some limit. When I go to implement this, I need to assume something. I understand the idea of just having some way to represent "big" numbers but that does not work well for many situations. For example, firewalls often check that messages are compliant with the syntax and "reasonable", they will choose to enforce what they believe is a "reasonable" upper limit. If you specify one, they will use that, if you don't specify one, they will choose something you may not like. Same for someone writing a library. I can easily imaging many library implementers thinking they could use 16 bits for this while others might choose 32. 

As far as the ABNF, I'd be fine with a comment such as 

integer32           = 1*DIGIT ; Must be in range of 0-4294967295 with no leading zero

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 5 May 2011 20:49:52 +0200

> I really prefer some limit. 

Note that there has been a brief discussion about artificial limitations like this going on in httpbis.

http://lists.w3.org/Archives/Public/ietf-http-wg/2011AprJun/0137.html

(Unfortunately, that then quickly ratholed into a discussion about the User-Agent header field.)

I don't think there is any proposal to change

     Content-Length = 1*DIGIT

in draft-ietf-httpbis-p1-messaging.

I believe we should follow HTTP's lead here.
Any artificial limitation we come up with will be so high that most people will need to do the "Big" thing anyway.  (If we have to, I'd stick with 1208925819614629174706175, which takes 288 years to transfer on a Pbit/s link, which sounds about right.  For this century.  9223372036854775807 might do in a pinch.)

(But, yes, I'd like to see the link-format ABNF for ct and sz as

     integer = "0" 
             / %x31-39 *DIGIT 

please -- "integer" could also be renamed to a term for non-negative integers, such as "cardinal".)

Gruesse, Carsten