Subject: [drinks] Discussion on UETF-8 from CORE working group

From: Dean Willis <[email-masked]>
Date: Wed, 27 Jul 2011 09:20:08 -0400

One of the open questions in SPPP is relay a meta-question about language encoding. What textual representation shall we use? We could stay with plain old 7-bit ASCII (which would be my preference), but I'm expecting international users to not like that idea very much.

The "obvious alternative" is to say UTF-8. But I'm learning that thee's a lot more to do than just say "use UTF-8".

The CORE working group (Constrained Restful Environments) is close to publishing the Constrained Application Protocol: http://www.ietf.org/id/draft-ietf-core-coap-07.txt

Zach Shelby, lead author of CoAP, said during today's CORE meeting that the UTF-8 discussion had proven to be much harder to get right than he had expected, but that it's pretty much done in CoAP.

Perhaps we could re-use the approach (and possibly even the text) from CoAP?

--
Dean

---

From: Alexander Mayrhofer <[email-masked]>
Date: Wed, 27 Jul 2011 15:33:02 +0200

> The "obvious alternative" is to say UTF-8. But I'm learning that
thee's a lot
> more to do than just say "use UTF-8".
> 
> The CORE working group (Constrained Restful Environments) is close to
> publishing the Constrained Application Protocol:
> http://www.ietf.org/id/draft-ietf-core-coap-07.txt
> 
> Zach Shelby, lead author of CoAP, said during today's CORE meeting
that the
> UTF-8 discussion had proven to be much harder to get right than he had
> expected, but that it's pretty much done in CoAP.
> 

I think we should definitely look at this, particularly because it
*could* be a base protocol for another transport mapping of DRINKS.

However, i think that by using the SOAP-XML-HTTP stack, we have a big
advantage compared to CoAP, since character encoding etc. is "done" for
that stack, while the CoAP folks essentially had to start from zero.

Do you see any action item on either the protocol or transport document
out of what your heard in CORE?

Alex

---

From: Dean Willis <[email-masked]>
Date: Fri, 29 Jul 2011 10:08:06 -0400

> out of what your heard in CORE?

well, I'm not an internationalization expert. I might even be described as internationalization-phobic (If it's not written in Tejano, it's foreign!). All I know about it is that for every protocol I've worked on, it's been much harder than anybody expected.

While our WSDL is described in UTF-8, I don't see anything else in the spec that tells us what kind of encoding the registry data is in. Four OpenSPP, we've made no real effort to support internationalization. All the hard string matching is being done inside MySQL, so we sort of have a "whatever the database does" model going on. Or at least I haven't seen any string-prep code being invoked. Note that this means I probably need to add internationalized test-cases to the product backlog.

So, unless I'm mistaken and we inherited something from one of the other specs, I don't know that we have a complete specification on whether or not we're using UTF-8 or UTF-16 or 7-bit ASCII or ASCII plus a codepage or multiple codepages (or whatever) and what the comparison rules, etc. are supposed to be. 

--
Dean

---

From: Alexander Mayrhofer <[email-masked]>
Date: Fri, 29 Jul 2011 16:11:12 +0200

> So, unless I'm mistaken and we inherited something from one of the
other
> specs, I don't know that we have a complete specification on whether
or not
> we're using UTF-8 or UTF-16 or 7-bit ASCII or ASCII plus a codepage or
> multiple codepages (or whatever) and what the comparison rules, etc.
are
> supposed to be.

We certainly have open issues here. I'm currently looking at the "Data
Validation" issue, and collecting a list of fields and what their
constraints might be. I will include internationalization considerations
in that list, and look for what other similar procotols do with regards
to Internationalization.

Alex

---

From: "Zhang, Ning" <[email-masked]>
Date: Fri, 29 Jul 2011 11:21:02 -0400

For the protocol, UTF-8 encoding is fine for I18N.

The issues are mainly to be dealt in the implementation.

I had some experience on UTF-8 with C++/Java, Oracle/Postgres/MySQL before.

The XML parser such as Xerces in C++/Java would always store strings extracted from XML messages in Unicode.

Depending on the database support and setup, the underlying C/Java code may need to do some translations before putting a string data into the database, or after getting data from the database.

For example, if the db set up is UTF-8, and the db driver does not support Unicode to UTF-8 conversion automatically, the application code would need to convert the string in UTF-8.

Additionally, there is an issue related to table column length. For example, a Unicode string of 16 chars may be translated into a UTF-8 string of 20 chars. This would require the application code for performing string length validations.

If required, one may need to perform IDN validations based on RFC 5891.

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Dean Willis
Sent: Friday, July 29, 2011 10:08 AM
To: Alexander Mayrhofer
Cc: [email-masked]
Subject: Re: [drinks] Discussion on UETF-8 from CORE working group

--
Dean
_______________________________________________
drinks mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/drinks