From yves@zioup.com  Mon Jun  1 15:27:16 2009
Return-Path: <yves@zioup.com>
X-Original-To: ldapext@core3.amsl.com
Delivered-To: ldapext@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 591143A6B9A for <ldapext@core3.amsl.com>; Mon,  1 Jun 2009 15:27:16 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.247
X-Spam-Level: 
X-Spam-Status: No, score=-1.247 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, J_CHICKENPOX_22=0.6, J_CHICKENPOX_43=0.6, SARE_SUB_ENC_UTF8=0.152]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id KZIPoLsoHCkW for <ldapext@core3.amsl.com>; Mon,  1 Jun 2009 15:27:15 -0700 (PDT)
Received: from idcmail-mo2no.shaw.ca (idcmail-mo2no.shaw.ca [64.59.134.9]) by core3.amsl.com (Postfix) with ESMTP id 20FE83A68EE for <ldapext@ietf.org>; Mon,  1 Jun 2009 15:27:13 -0700 (PDT)
Received: from pd6ml2no-ssvc.prod.shaw.ca ([10.0.153.163]) by pd5mo1no-svcs.prod.shaw.ca with ESMTP; 01 Jun 2009 16:26:06 -0600
X-Cloudmark-SP-Filtered: true
X-Cloudmark-SP-Result: v=1.0 c=0 a=563Ur-DeAAAA:8 a=J6b0bBEkAAAA:8 a=V1iep_2QXOpg2FuPxMIA:9 a=j4bl4BuY_BEz1ctBQ1eEz28yHEgA:4 a=WPV3hXSGBlIA:10 a=Cguc0gE-TcEA:10
Received: from unknown (HELO home.zioup.com) ([68.147.37.46]) by pd6ml2no-dmz.prod.shaw.ca with ESMTP; 01 Jun 2009 16:26:06 -0600
Received: from [192.168.2.209] (83D1.zioup.net [192.168.2.209]) by home.zioup.com (Postfix) with ESMTP id 3B2BF2F03D for <ldapext@ietf.org>; Mon,  1 Jun 2009 16:26:06 -0600 (MDT)
Message-ID: <4A24557D.7030006@zioup.com>
Date: Mon, 01 Jun 2009 16:26:05 -0600
From: Yves Dorfsman <yves@zioup.com>
User-Agent: Thunderbird 2.0.0.21 (X11/20090409)
MIME-Version: 1.0
To: ldapext@ietf.org
References: <49C497F9.7010200@zioup.com>	<CD3905D4-2A25-4C56-8187-3CE10D46C929@isode.com>	<49C870C6.4010803@zioup.com>	<E94B7389-9A6D-4CB6-BB2C-649CCD3FD15B@Isode.com>	<49CB192E.5050105@zioup.com> <49CB211C.6070108@eb2bcom.com> <49CB87FE.1050809@zioup.com> <49CC01DE.6040506@eb2bcom.com>
In-Reply-To: <49CC01DE.6040506@eb2bcom.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Subject: Re: [ldapext] UTF-8 full support in LDIF / LDIF v2
X-BeenThere: ldapext@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: LDAP Extension Working Group <ldapext.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/ldapext>, <mailto:ldapext-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/ldapext>
List-Post: <mailto:ldapext@ietf.org>
List-Help: <mailto:ldapext-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/ldapext>, <mailto:ldapext-request@ietf.org?subject=subscribe>
X-List-Received-Date: Mon, 01 Jun 2009 22:27:16 -0000

Steven Legg wrote:

>>>>> See http://www.xmled.info/drafts/draft-sciberras-xed-eldif-05.txt

I did look at it, personally I find it difficult for humans, for diff'ing 
etc... XML has its place, but so does pure text.


>> Yes I was wondering about that, do we need multi-line values as work 
>> around because schemas aren't precise enough ?
> 
> No, we need them because sheets of paper, computer screens and RFCs are
> not infinitely wide. :-) Human-readability, line breaks and indenting tend
> to go hand-in-hand.


I've been thinking about this and trying a few things. My conclusion is that 
the best solution would be the good old here document.

objectclass: inetOrgPerson
organizationName:<<EOT
The two line
   company
EOT
sn: Jensen



With the following specifications:
Any of the following characters (or sequence in the case of CR+LF) can be 
used as a separator (<SEP>):

LF (U+000A), CR (U+000D), CR+LF (U+000D followed by U+000A), NEL (U+0085), 
FF (U+000C), LS (U+2028), PS U+2029)

Any sequence of characters can be used instead of EOT, but cannot include a 
separator character. The same sequence has to be used at the begining and 
the end.

Any UTF-8 character, except separators, can be used on each line.

Any separator can be used to separate the lines.

The text start after EOT<SEP>, and finishes with the last character before 
<SEP>EOT. The organization name in the example above is exactly two lines, 
the last separator is not part of the text.

No need or possibility to escape characters, no possibility of folding lines  .



-- 
Yves.
http://www.sollers.ca/


From yves@zioup.com  Wed Jun  3 09:11:14 2009
Return-Path: <yves@zioup.com>
X-Original-To: ldapext@core3.amsl.com
Delivered-To: ldapext@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id CCC093A7039 for <ldapext@core3.amsl.com>; Wed,  3 Jun 2009 09:11:14 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.247
X-Spam-Level: 
X-Spam-Status: No, score=-1.247 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, J_CHICKENPOX_22=0.6, J_CHICKENPOX_43=0.6, SARE_SUB_ENC_UTF8=0.152]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id iAtUydlSmBOm for <ldapext@core3.amsl.com>; Wed,  3 Jun 2009 09:11:13 -0700 (PDT)
Received: from idcmail-mo1so.shaw.ca (idcmail-mo1so.shaw.ca [24.71.223.10]) by core3.amsl.com (Postfix) with ESMTP id AF8C13A69A2 for <ldapext@ietf.org>; Wed,  3 Jun 2009 09:11:13 -0700 (PDT)
Received: from pd2ml2so-ssvc.prod.shaw.ca ([10.0.141.134]) by pd3mo1so-svcs.prod.shaw.ca with ESMTP; 03 Jun 2009 10:10:06 -0600
X-Cloudmark-SP-Filtered: true
X-Cloudmark-SP-Result: v=1.0 c=0 a=563Ur-DeAAAA:8 a=J6b0bBEkAAAA:8 a=d8b2TxWtnEydG9VJumsA:9 a=8Hg6qzXlqUNGJCzloruJQjR7GZkA:4 a=WPV3hXSGBlIA:10 a=Cguc0gE-TcEA:10
Received: from unknown (HELO home.zioup.com) ([68.147.37.46]) by pd2ml2so-dmz.prod.shaw.ca with ESMTP; 03 Jun 2009 10:10:06 -0600
Received: from [192.168.2.209] (83D1.zioup.net [192.168.2.209]) by home.zioup.com (Postfix) with ESMTP id D145B1F000 for <ldapext@ietf.org>; Wed,  3 Jun 2009 10:10:05 -0600 (MDT)
Message-ID: <4A26A05D.8040105@zioup.com>
Date: Wed, 03 Jun 2009 10:10:05 -0600
From: Yves Dorfsman <yves@zioup.com>
User-Agent: Thunderbird 2.0.0.21 (X11/20090409)
MIME-Version: 1.0
To: ldapext@ietf.org
References: <49C497F9.7010200@zioup.com>	<CD3905D4-2A25-4C56-8187-3CE10D46C929@isode.com>	<49C870C6.4010803@zioup.com>	<E94B7389-9A6D-4CB6-BB2C-649CCD3FD15B@Isode.com>	<49CB192E.5050105@zioup.com>	<49CB211C.6070108@eb2bcom.com> <49CB87FE.1050809@zioup.com>	<49CC01DE.6040506@eb2bcom.com> <4A24557D.7030006@zioup.com>
In-Reply-To: <4A24557D.7030006@zioup.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Subject: Re: [ldapext] UTF-8 full support in LDIF / LDIF v2
X-BeenThere: ldapext@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: LDAP Extension Working Group <ldapext.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/ldapext>, <mailto:ldapext-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/ldapext>
List-Post: <mailto:ldapext@ietf.org>
List-Help: <mailto:ldapext-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/ldapext>, <mailto:ldapext-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 03 Jun 2009 16:11:14 -0000

I have not received any feed back on this message, and am not sure how to 
interpret that.

Is everybody getting tired of the debate ?
Is the idea of a here document syntax too ridiculous ?
Is UTF-8 support in LDIF not that important ?

Am I the only one thinking xml is not a good replacement for LDIF, if so, 
should we help Steven with the xmled RFC ?

Thanks.


Yves Dorfsman wrote:
> 
> Steven Legg wrote:
> 
>>>>>> See http://www.xmled.info/drafts/draft-sciberras-xed-eldif-05.txt
> 
> I did look at it, personally I find it difficult for humans, for 
> diff'ing etc... XML has its place, but so does pure text.
> 
> 
>>> Yes I was wondering about that, do we need multi-line values as work 
>>> around because schemas aren't precise enough ?
>>
>> No, we need them because sheets of paper, computer screens and RFCs are
>> not infinitely wide. :-) Human-readability, line breaks and indenting 
>> tend
>> to go hand-in-hand.
> 
> 
> I've been thinking about this and trying a few things. My conclusion is 
> that the best solution would be the good old here document.
> 
> objectclass: inetOrgPerson
> organizationName:<<EOT
> The two line
>   company
> EOT
> sn: Jensen
> 
> 
> 
> With the following specifications:
> Any of the following characters (or sequence in the case of CR+LF) can 
> be used as a separator (<SEP>):
> 
> LF (U+000A), CR (U+000D), CR+LF (U+000D followed by U+000A), NEL 
> (U+0085), FF (U+000C), LS (U+2028), PS U+2029)
> 
> Any sequence of characters can be used instead of EOT, but cannot 
> include a separator character. The same sequence has to be used at the 
> begining and the end.
> 
> Any UTF-8 character, except separators, can be used on each line.
> 
> Any separator can be used to separate the lines.
> 
> The text start after EOT<SEP>, and finishes with the last character 
> before <SEP>EOT. The organization name in the example above is exactly 
> two lines, the last separator is not part of the text.
> 
> No need or possibility to escape characters, no possibility of folding 
> lines  .
> 
> 
> 


-- 
Yves.
http://www.sollers.ca/


From hyc@highlandsun.com  Wed Jun  3 16:05:03 2009
Return-Path: <hyc@highlandsun.com>
X-Original-To: ldapext@core3.amsl.com
Delivered-To: ldapext@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 6AA853A6CF6 for <ldapext@core3.amsl.com>; Wed,  3 Jun 2009 16:05:03 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.847
X-Spam-Level: 
X-Spam-Status: No, score=-1.847 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, J_CHICKENPOX_43=0.6, SARE_SUB_ENC_UTF8=0.152]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Sm5enBau49-C for <ldapext@core3.amsl.com>; Wed,  3 Jun 2009 16:05:02 -0700 (PDT)
Received: from mail.highlandsun.com (mail.highlandsun.com [70.87.222.79]) by core3.amsl.com (Postfix) with ESMTP id 535CF3A6844 for <ldapext@ietf.org>; Wed,  3 Jun 2009 16:05:02 -0700 (PDT)
Received: from [127.0.0.1] (localhost [127.0.0.1]) by mail.highlandsun.com (Postfix) with ESMTP id B7B7310F97; Wed,  3 Jun 2009 19:05:03 -0400 (EDT)
Message-ID: <4A27019E.4080900@highlandsun.com>
Date: Wed, 03 Jun 2009 16:05:02 -0700
From: Howard Chu <hyc@highlandsun.com>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; rv:1.9.1b5pre) Gecko/20090517 SeaMonkey/2.0a1pre Firefox/3.0.3
MIME-Version: 1.0
To: Yves Dorfsman <yves@zioup.com>
References: <49C497F9.7010200@zioup.com>	<CD3905D4-2A25-4C56-8187-3CE10D46C929@isode.com>	<49C870C6.4010803@zioup.com>	<E94B7389-9A6D-4CB6-BB2C-649CCD3FD15B@Isode.com>	<49CB192E.5050105@zioup.com>	<49CB211C.6070108@eb2bcom.com>	<49CB87FE.1050809@zioup.com>	<49CC01DE.6040506@eb2bcom.com>	<4A24557D.7030006@zioup.com> <4A26A05D.8040105@zioup.com>
In-Reply-To: <4A26A05D.8040105@zioup.com>
Content-Type: text/plain; charset=us-ascii; format=flowed
Content-Transfer-Encoding: 7bit
Cc: ldapext@ietf.org
Subject: Re: [ldapext] UTF-8 full support in LDIF / LDIF v2
X-BeenThere: ldapext@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: LDAP Extension Working Group <ldapext.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/ldapext>, <mailto:ldapext-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/ldapext>
List-Post: <mailto:ldapext@ietf.org>
List-Help: <mailto:ldapext-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/ldapext>, <mailto:ldapext-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 03 Jun 2009 23:05:03 -0000

Yves Dorfsman wrote:
>
> I have not received any feed back on this message, and am not sure how to
> interpret that.
>
> Is everybody getting tired of the debate ?
> Is the idea of a here document syntax too ridiculous ?

Using the same syntax as the existing Unix shells is problematic, since there 
are already shell scripts that construct LDIF using here-documents. So yes, it 
appears your suggestion is not very well thought out.

This is also the wrong place (or the wrong time) to be discussing multi-line 
values; the purpose of LDIF is to faithfully transmit values stored in a 
directory. As such, whether a value is "multi-line" or not, and how those line 
breaks are represented, is determined solely by the attribute's syntax. 
Anything you discuss here in the LDIF spec is purely cosmetic. E.g., for your 
example below, one can easily write:

organizationName: A Very Long
   Organization name just
   to make a point

but the breaks are immaterial. If you want to talk about actual multi-line 
values with some kind of embedded line-break then you need to define a syntax 
with those properties.

> Is UTF-8 support in LDIF not that important ?

LDIF is just a transfer format, like base64. Any string data carried within it 
is already understood to be in UTF-8. So yes, UTF-8 is important, but it's 
already a given. You're confusing the content with the container.

> Am I the only one thinking xml is not a good replacement for LDIF, if so,
> should we help Steven with the xmled RFC ?

I don't see how the question is related, but IMO, the less XML exists, the better.

> Thanks.
>
>
> Yves Dorfsman wrote:
>>
>> Steven Legg wrote:
>>
>>>>>>> See http://www.xmled.info/drafts/draft-sciberras-xed-eldif-05.txt
>>
>> I did look at it, personally I find it difficult for humans, for
>> diff'ing etc... XML has its place, but so does pure text.
>>
>>
>>>> Yes I was wondering about that, do we need multi-line values as work
>>>> around because schemas aren't precise enough ?
>>>
>>> No, we need them because sheets of paper, computer screens and RFCs are
>>> not infinitely wide. :-) Human-readability, line breaks and indenting
>>> tend
>>> to go hand-in-hand.
>>
>>
>> I've been thinking about this and trying a few things. My conclusion is
>> that the best solution would be the good old here document.
>>
>> objectclass: inetOrgPerson
>> organizationName:<<EOT
>> The two line
>>    company
>> EOT
>> sn: Jensen

-- 
   -- Howard Chu
   CTO, Symas Corp.           http://www.symas.com
   Director, Highland Sun     http://highlandsun.com/hyc/
   Chief Architect, OpenLDAP  http://www.openldap.org/project/

From Kurt.Zeilenga@Isode.com  Wed Jun  3 17:44:59 2009
Return-Path: <Kurt.Zeilenga@Isode.com>
X-Original-To: ldapext@core3.amsl.com
Delivered-To: ldapext@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 3B03C3A6B28 for <ldapext@core3.amsl.com>; Wed,  3 Jun 2009 17:44:59 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.247
X-Spam-Level: 
X-Spam-Status: No, score=-1.247 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, J_CHICKENPOX_22=0.6, J_CHICKENPOX_43=0.6, SARE_SUB_ENC_UTF8=0.152]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id N5vfeSF02ez4 for <ldapext@core3.amsl.com>; Wed,  3 Jun 2009 17:44:58 -0700 (PDT)
Received: from boole.openldap.org (boole.openldap.org [IPv6:2001:4f8:3:ba:219:21ff:fe2f:68d5]) by core3.amsl.com (Postfix) with ESMTP id 054003A6452 for <ldapext@ietf.org>; Wed,  3 Jun 2009 17:44:57 -0700 (PDT)
Received: from [192.168.1.102] (75-141-233-128.dhcp.nv.charter.com [75.141.233.128] (may be forged)) (authenticated bits=0) by boole.openldap.org (8.14.3/8.14.3) with ESMTP id n540ipcB062524; Thu, 4 Jun 2009 00:44:51 GMT (envelope-from Kurt.Zeilenga@Isode.com)
Message-Id: <245BF18B-2066-4E36-9502-16F4A3140D9E@Isode.com>
From: Kurt Zeilenga <Kurt.Zeilenga@Isode.com>
To: Yves Dorfsman <yves@zioup.com>
In-Reply-To: <4A26A05D.8040105@zioup.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v935.3)
Date: Wed, 3 Jun 2009 17:44:50 -0700
References: <49C497F9.7010200@zioup.com>	<CD3905D4-2A25-4C56-8187-3CE10D46C929@isode.com>	<49C870C6.4010803@zioup.com>	<E94B7389-9A6D-4CB6-BB2C-649CCD3FD15B@Isode.com>	<49CB192E.5050105@zioup.com>	<49CB211C.6070108@eb2bcom.com> <49CB87FE.1050809@zioup.com>	<49CC01DE.6040506@eb2bcom.com> <4A24557D.7030006@zioup.com> <4A26A05D.8040105@zioup.com>
X-Mailer: Apple Mail (2.935.3)
Cc: ldapext@ietf.org
Subject: Re: [ldapext] UTF-8 full support in LDIF / LDIF v2
X-BeenThere: ldapext@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: LDAP Extension Working Group <ldapext.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/ldapext>, <mailto:ldapext-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/ldapext>
List-Post: <mailto:ldapext@ietf.org>
List-Help: <mailto:ldapext-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/ldapext>, <mailto:ldapext-request@ietf.org?subject=subscribe>
X-List-Received-Date: Thu, 04 Jun 2009 00:44:59 -0000

On Jun 3, 2009, at 9:10 AM, Yves Dorfsman wrote:
> Is the idea of a here document syntax too ridiculous ?

There are a number of problems with it.  Personally, I think what  
Steven already offered (and likely implemented) is better, though I am  
concerned about line separators.  As Howard comments kind of suggests,  
when you have a value which is multi-lined, it's the syntax that  
controls what line separators are used, not the LDIF.  For instance,  
in some syntaxes, a $ is used to as a line separator.

The problem with your proposal, and Steven's, is that LDIF line  
separators and value line separators are one and the same thing.   
While one might be case occasionally, it cannot be expected to be  
generally the case.

LDIF is first and foremost an interchange format.  Conversion from  
LDAP PDU->LDIF Record->LDAP PDU MUST produce as output the input,  
octet for octet for every "data" component (the DN, every attribute  
description and associated values, etc.).

> Is UTF-8 support in LDIF not that important ?

LDIF being a proper interchange format is important.  UTF-8 support  
(other than being able to interchange values whose syntax is UTF-8  
encoded) is cosmetic.

Adding UTF-8 support does appear to be in support of improving LDIF as  
a proper interchange format.  It seems to be driven by other goals,  
such as trying to make LDIF files displayable.   Given that LDAP does  
not constrain attribute value syntaxes (even directory strings can  
contain arbitrary sequences of Unicode code points), the goal of  
making LDIF files displayable is not terribly feasible.

I note that even today, ASCII LDIF files might not display properly  
without special handling, such as for line separators.  But with  
UTF-8, line separators are only the tip of iceberg of display problems.

I'm not convinced that removing the ASCII restrictions will be a good  
thing.  Not only do I doubt it will have a net positive on  
displayability of LDIF for those who have a displayability goal (I  
don't this goal), I think it will have a net negative impact on  
interoperability and user confusion, such as when the user creates a  
file using one Unicode normalization algorithm, but is trying to set  
values which require a different Unicode normalization value.

> Am I the only one thinking xml is not a good replacement for LDIF,

There already exists a number of XML replacements of LDIF, such as  
DSML...  so I guess at least some do think XML is a good replacement  
for LDIF.

> if so, should we help Steven with the xmled RFC ?

What Steven and Andrew have done is define an extension for LDIF to  
allow XML values to be represented in a human-readable format instead  
of requiring the use use of base64.  Unfortunately his proposal has  
interchange issues (see the I-D's security considerations section).   
This, I think, is a fatal problem with this extension.

-- Kurt


>
>
> Thanks.
>
>
> Yves Dorfsman wrote:
>> Steven Legg wrote:
>>>>>>> See http://www.xmled.info/drafts/draft-sciberras-xed- 
>>>>>>> eldif-05.txt
>> I did look at it, personally I find it difficult for humans, for  
>> diff'ing etc... XML has its place, but so does pure text.
>>>> Yes I was wondering about that, do we need multi-line values as  
>>>> work around because schemas aren't precise enough ?
>>>
>>> No, we need them because sheets of paper, computer screens and  
>>> RFCs are
>>> not infinitely wide. :-) Human-readability, line breaks and  
>>> indenting tend
>>> to go hand-in-hand.
>> I've been thinking about this and trying a few things. My  
>> conclusion is that the best solution would be the good old here  
>> document.
>> objectclass: inetOrgPerson
>> organizationName:<<EOT
>> The two line
>>  company
>> EOT
>> sn: Jensen
>> With the following specifications:
>> Any of the following characters (or sequence in the case of CR+LF)  
>> can be used as a separator (<SEP>):
>> LF (U+000A), CR (U+000D), CR+LF (U+000D followed by U+000A), NEL (U 
>> +0085), FF (U+000C), LS (U+2028), PS U+2029)
>> Any sequence of characters can be used instead of EOT, but cannot  
>> include a separator character. The same sequence has to be used at  
>> the begining and the end.
>> Any UTF-8 character, except separators, can be used on each line.
>> Any separator can be used to separate the lines.
>> The text start after EOT<SEP>, and finishes with the last character  
>> before <SEP>EOT. The organization name in the example above is  
>> exactly two lines, the last separator is not part of the text.
>> No need or possibility to escape characters, no possibility of  
>> folding lines  .
>
>
> -- 
> Yves.
> http://www.sollers.ca/
>
> _______________________________________________
> Ldapext mailing list
> Ldapext@ietf.org
> https://www.ietf.org/mailman/listinfo/ldapext


From jim@willeke.com  Wed Jun  3 17:54:23 2009
Return-Path: <jim@willeke.com>
X-Original-To: ldapext@core3.amsl.com
Delivered-To: ldapext@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 707333A685C for <ldapext@core3.amsl.com>; Wed,  3 Jun 2009 17:54:23 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: 0.668
X-Spam-Level: 
X-Spam-Status: No, score=0.668 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, FM_FORGED_GMAIL=0.622, HTML_MESSAGE=0.001, J_CHICKENPOX_22=0.6, J_CHICKENPOX_43=0.6, MISSING_HEADERS=1.292, SARE_SUB_ENC_UTF8=0.152]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id LXD8bAqu3exk for <ldapext@core3.amsl.com>; Wed,  3 Jun 2009 17:54:22 -0700 (PDT)
Received: from yw-out-2324.google.com (yw-out-2324.google.com [74.125.46.31]) by core3.amsl.com (Postfix) with ESMTP id E4C143A67D2 for <ldapext@ietf.org>; Wed,  3 Jun 2009 17:54:21 -0700 (PDT)
Received: by yw-out-2324.google.com with SMTP id 3so342329ywj.49 for <ldapext@ietf.org>; Wed, 03 Jun 2009 17:54:20 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.151.14.5 with SMTP id r5mr2292758ybi.300.1244076859272; Wed,  03 Jun 2009 17:54:19 -0700 (PDT)
In-Reply-To: <245BF18B-2066-4E36-9502-16F4A3140D9E@Isode.com>
References: <49C497F9.7010200@zioup.com> <49C870C6.4010803@zioup.com> <E94B7389-9A6D-4CB6-BB2C-649CCD3FD15B@Isode.com> <49CB192E.5050105@zioup.com> <49CB211C.6070108@eb2bcom.com> <49CB87FE.1050809@zioup.com> <49CC01DE.6040506@eb2bcom.com> <4A24557D.7030006@zioup.com> <4A26A05D.8040105@zioup.com> <245BF18B-2066-4E36-9502-16F4A3140D9E@Isode.com>
Date: Wed, 3 Jun 2009 20:54:19 -0400
Message-ID: <b662a94e0906031754n217f96c8t55e1e0c34f11bb86@mail.gmail.com>
From: Jim Willeke <jim@willeke.com>
Cc: ldapext@ietf.org
Content-Type: multipart/alternative; boundary=000e0cd6adaa453076046b7b3777
Subject: Re: [ldapext] UTF-8 full support in LDIF / LDIF v2
X-BeenThere: ldapext@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: LDAP Extension Working Group <ldapext.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/ldapext>, <mailto:ldapext-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/ldapext>
List-Post: <mailto:ldapext@ietf.org>
List-Help: <mailto:ldapext-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/ldapext>, <mailto:ldapext-request@ietf.org?subject=subscribe>
X-List-Received-Date: Thu, 04 Jun 2009 00:54:23 -0000

--000e0cd6adaa453076046b7b3777
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

I would agree with Kurt's and Howard's comments.
Perhaps what is really required is a good tool to display and edit the
existing LDIF formatted files that in a view that is deemed acceptable.

-jim
Jim Willeke


On Wed, Jun 3, 2009 at 8:44 PM, Kurt Zeilenga <Kurt.Zeilenga@isode.com>wrote:

>
> On Jun 3, 2009, at 9:10 AM, Yves Dorfsman wrote:
>
>> Is the idea of a here document syntax too ridiculous ?
>>
>
> There are a number of problems with it.  Personally, I think what Steven
> already offered (and likely implemented) is better, though I am concerned
> about line separators.  As Howard comments kind of suggests, when you have a
> value which is multi-lined, it's the syntax that controls what line
> separators are used, not the LDIF.  For instance, in some syntaxes, a $ is
> used to as a line separator.
>
> The problem with your proposal, and Steven's, is that LDIF line separators
> and value line separators are one and the same thing.  While one might be
> case occasionally, it cannot be expected to be generally the case.
>
> LDIF is first and foremost an interchange format.  Conversion from LDAP
> PDU->LDIF Record->LDAP PDU MUST produce as output the input, octet for octet
> for every "data" component (the DN, every attribute description and
> associated values, etc.).
>
>  Is UTF-8 support in LDIF not that important ?
>>
>
> LDIF being a proper interchange format is important.  UTF-8 support (other
> than being able to interchange values whose syntax is UTF-8 encoded) is
> cosmetic.
>
> Adding UTF-8 support does appear to be in support of improving LDIF as a
> proper interchange format.  It seems to be driven by other goals, such as
> trying to make LDIF files displayable.   Given that LDAP does not constrain
> attribute value syntaxes (even directory strings can contain arbitrary
> sequences of Unicode code points), the goal of making LDIF files displayable
> is not terribly feasible.
>
> I note that even today, ASCII LDIF files might not display properly without
> special handling, such as for line separators.  But with UTF-8, line
> separators are only the tip of iceberg of display problems.
>
> I'm not convinced that removing the ASCII restrictions will be a good
> thing.  Not only do I doubt it will have a net positive on displayability of
> LDIF for those who have a displayability goal (I don't this goal), I think
> it will have a net negative impact on interoperability and user confusion,
> such as when the user creates a file using one Unicode normalization
> algorithm, but is trying to set values which require a different Unicode
> normalization value.
>
>  Am I the only one thinking xml is not a good replacement for LDIF,
>>
>
> There already exists a number of XML replacements of LDIF, such as DSML...
>  so I guess at least some do think XML is a good replacement for LDIF.
>
>  if so, should we help Steven with the xmled RFC ?
>>
>
> What Steven and Andrew have done is define an extension for LDIF to allow
> XML values to be represented in a human-readable format instead of requiring
> the use use of base64.  Unfortunately his proposal has interchange issues
> (see the I-D's security considerations section).  This, I think, is a fatal
> problem with this extension.
>
> -- Kurt
>
>
>
>>
>> Thanks.
>>
>>
>> Yves Dorfsman wrote:
>>
>>> Steven Legg wrote:
>>>
>>>> See http://www.xmled.info/drafts/draft-sciberras-xed-eldif-05.txt
>>>>>>>>
>>>>>>> I did look at it, personally I find it difficult for humans, for
>>> diff'ing etc... XML has its place, but so does pure text.
>>>
>>>> Yes I was wondering about that, do we need multi-line values as work
>>>>> around because schemas aren't precise enough ?
>>>>>
>>>>
>>>> No, we need them because sheets of paper, computer screens and RFCs are
>>>> not infinitely wide. :-) Human-readability, line breaks and indenting
>>>> tend
>>>> to go hand-in-hand.
>>>>
>>> I've been thinking about this and trying a few things. My conclusion is
>>> that the best solution would be the good old here document.
>>> objectclass: inetOrgPerson
>>> organizationName:<<EOT
>>> The two line
>>>  company
>>> EOT
>>> sn: Jensen
>>> With the following specifications:
>>> Any of the following characters (or sequence in the case of CR+LF) can be
>>> used as a separator (<SEP>):
>>> LF (U+000A), CR (U+000D), CR+LF (U+000D followed by U+000A), NEL
>>> (U+0085), FF (U+000C), LS (U+2028), PS U+2029)
>>> Any sequence of characters can be used instead of EOT, but cannot include
>>> a separator character. The same sequence has to be used at the begining and
>>> the end.
>>> Any UTF-8 character, except separators, can be used on each line.
>>> Any separator can be used to separate the lines.
>>> The text start after EOT<SEP>, and finishes with the last character
>>> before <SEP>EOT. The organization name in the example above is exactly two
>>> lines, the last separator is not part of the text.
>>> No need or possibility to escape characters, no possibility of folding
>>> lines  .
>>>
>>
>>
>> --
>> Yves.
>> http://www.sollers.ca/
>>
>> _______________________________________________
>> Ldapext mailing list
>> Ldapext@ietf.org
>> https://www.ietf.org/mailman/listinfo/ldapext
>>
>
> _______________________________________________
> Ldapext mailing list
> Ldapext@ietf.org
> https://www.ietf.org/mailman/listinfo/ldapext
>

--000e0cd6adaa453076046b7b3777
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

I would agree with Kurt&#39;s and Howard&#39;s comments.<br>Perhaps what is
really required is a good tool to display and edit the existing LDIF
formatted files that in a view that is deemed acceptable.<br clear=3D"all">=
<br>-jim<br>Jim Willeke<br>
<br><br><div class=3D"gmail_quote">On Wed, Jun 3, 2009 at 8:44 PM, Kurt Zei=
lenga <span dir=3D"ltr">&lt;<a href=3D"mailto:Kurt.Zeilenga@isode.com">Kurt=
.Zeilenga@isode.com</a>&gt;</span> wrote:<br><blockquote class=3D"gmail_quo=
te" style=3D"border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt=
 0.8ex; padding-left: 1ex;">
<br>
On Jun 3, 2009, at 9:10 AM, Yves Dorfsman wrote:<br>
<blockquote class=3D"gmail_quote" style=3D"border-left: 1px solid rgb(204, =
204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Is the idea of a here document syntax too ridiculous ?<br>
</blockquote>
<br>
There are a number of problems with it. =A0Personally, I think what Steven =
already offered (and likely implemented) is better, though I am concerned a=
bout line separators. =A0As Howard comments kind of suggests, when you have=
 a value which is multi-lined, it&#39;s the syntax that controls what line =
separators are used, not the LDIF. =A0For instance, in some syntaxes, a $ i=
s used to as a line separator.<br>

<br>
The problem with your proposal, and Steven&#39;s, is that LDIF line separat=
ors and value line separators are one and the same thing. =A0While one migh=
t be case occasionally, it cannot be expected to be generally the case.<br>

<br>
LDIF is first and foremost an interchange format. =A0Conversion from LDAP P=
DU-&gt;LDIF Record-&gt;LDAP PDU MUST produce as output the input, octet for=
 octet for every &quot;data&quot; component (the DN, every attribute descri=
ption and associated values, etc.).<br>

<br>
<blockquote class=3D"gmail_quote" style=3D"border-left: 1px solid rgb(204, =
204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Is UTF-8 support in LDIF not that important ?<br>
</blockquote>
<br>
LDIF being a proper interchange format is important. =A0UTF-8 support (othe=
r than being able to interchange values whose syntax is UTF-8 encoded) is c=
osmetic.<br>
<br>
Adding UTF-8 support does appear to be in support of improving LDIF as a pr=
oper interchange format. =A0It seems to be driven by other goals, such as t=
rying to make LDIF files displayable. =A0 Given that LDAP does not constrai=
n attribute value syntaxes (even directory strings can contain arbitrary se=
quences of Unicode code points), the goal of making LDIF files displayable =
is not terribly feasible.<br>

<br>
I note that even today, ASCII LDIF files might not display properly without=
 special handling, such as for line separators. =A0But with UTF-8, line sep=
arators are only the tip of iceberg of display problems.<br>
<br>
I&#39;m not convinced that removing the ASCII restrictions will be a good t=
hing. =A0Not only do I doubt it will have a net positive on displayability =
of LDIF for those who have a displayability goal (I don&#39;t this goal), I=
 think it will have a net negative impact on interoperability and user conf=
usion, such as when the user creates a file using one Unicode normalization=
 algorithm, but is trying to set values which require a different Unicode n=
ormalization value.<br>

<br>
<blockquote class=3D"gmail_quote" style=3D"border-left: 1px solid rgb(204, =
204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Am I the only one thinking xml is not a good replacement for LDIF,<br>
</blockquote>
<br>
There already exists a number of XML replacements of LDIF, such as DSML... =
=A0so I guess at least some do think XML is a good replacement for LDIF.<br=
>
<br>
<blockquote class=3D"gmail_quote" style=3D"border-left: 1px solid rgb(204, =
204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
if so, should we help Steven with the xmled RFC ?<br>
</blockquote>
<br>
What Steven and Andrew have done is define an extension for LDIF to allow X=
ML values to be represented in a human-readable format instead of requiring=
 the use use of base64. =A0Unfortunately his proposal has interchange issue=
s (see the I-D&#39;s security considerations section). =A0This, I think, is=
 a fatal problem with this extension.<br>

<br>
-- Kurt<br>
<br>
<br>
<blockquote class=3D"gmail_quote" style=3D"border-left: 1px solid rgb(204, =
204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<br>
<br>
Thanks.<br>
<br>
<br>
Yves Dorfsman wrote:<br>
<blockquote class=3D"gmail_quote" style=3D"border-left: 1px solid rgb(204, =
204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Steven Legg wrote:<br>
<blockquote class=3D"gmail_quote" style=3D"border-left: 1px solid rgb(204, =
204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><blockquote class=
=3D"gmail_quote" style=3D"border-left: 1px solid rgb(204, 204, 204); margin=
: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<blockquote class=3D"gmail_quote" style=3D"border-left: 1px solid rgb(204, =
204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><blockquote class=
=3D"gmail_quote" style=3D"border-left: 1px solid rgb(204, 204, 204); margin=
: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<blockquote class=3D"gmail_quote" style=3D"border-left: 1px solid rgb(204, =
204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
See <a href=3D"http://www.xmled.info/drafts/draft-sciberras-xed-" target=3D=
"_blank">http://www.xmled.info/drafts/draft-sciberras-xed-</a>eldif-05.txt<=
br>
</blockquote></blockquote></blockquote></blockquote></blockquote>
I did look at it, personally I find it difficult for humans, for diff&#39;i=
ng etc... XML has its place, but so does pure text.<br>
<blockquote class=3D"gmail_quote" style=3D"border-left: 1px solid rgb(204, =
204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><blockquote class=
=3D"gmail_quote" style=3D"border-left: 1px solid rgb(204, 204, 204); margin=
: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">

Yes I was wondering about that, do we need multi-line values as work around=
 because schemas aren&#39;t precise enough ?<br>
</blockquote>
<br>
No, we need them because sheets of paper, computer screens and RFCs are<br>
not infinitely wide. :-) Human-readability, line breaks and indenting tend<=
br>
to go hand-in-hand.<br>
</blockquote>
I&#39;ve been thinking about this and trying a few things. My conclusion is=
 that the best solution would be the good old here document.<br>
objectclass: inetOrgPerson<br>
organizationName:&lt;&lt;EOT<br>
The two line<br>
=A0company<br>
EOT<br>
sn: Jensen<br>
With the following specifications:<br>
Any of the following characters (or sequence in the case of CR+LF) can be u=
sed as a separator (&lt;SEP&gt;):<br>
LF (U+000A), CR (U+000D), CR+LF (U+000D followed by U+000A), NEL (U+0085), =
FF (U+000C), LS (U+2028), PS U+2029)<br>
Any sequence of characters can be used instead of EOT, but cannot include a=
 separator character. The same sequence has to be used at the begining and =
the end.<br>
Any UTF-8 character, except separators, can be used on each line.<br>
Any separator can be used to separate the lines.<br>
The text start after EOT&lt;SEP&gt;, and finishes with the last character b=
efore &lt;SEP&gt;EOT. The organization name in the example above is exactly=
 two lines, the last separator is not part of the text.<br>
No need or possibility to escape characters, no possibility of folding line=
s =A0.<br>
</blockquote>
<br>
<br>
-- <br>
Yves.<br>
<a href=3D"http://www.sollers.ca/" target=3D"_blank">http://www.sollers.ca/=
</a><br>
<br>
_______________________________________________<br>
Ldapext mailing list<br>
<a href=3D"mailto:Ldapext@ietf.org" target=3D"_blank">Ldapext@ietf.org</a><=
br>
<a href=3D"https://www.ietf.org/mailman/listinfo/ldapext" target=3D"_blank"=
>https://www.ietf.org/mailman/listinfo/ldapext</a><br>
</blockquote>
<br>
_______________________________________________<br>
Ldapext mailing list<br>
<a href=3D"mailto:Ldapext@ietf.org" target=3D"_blank">Ldapext@ietf.org</a><=
br>
<a href=3D"https://www.ietf.org/mailman/listinfo/ldapext" target=3D"_blank"=
>https://www.ietf.org/mailman/listinfo/ldapext</a><br>
</blockquote></div><br>

--000e0cd6adaa453076046b7b3777--

From steven.legg@eb2bcom.com  Thu Jun  4 00:29:26 2009
Return-Path: <steven.legg@eb2bcom.com>
X-Original-To: ldapext@core3.amsl.com
Delivered-To: ldapext@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 6510B3A6C83 for <ldapext@core3.amsl.com>; Thu,  4 Jun 2009 00:29:26 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: 0.978
X-Spam-Level: 
X-Spam-Status: No, score=0.978 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, FH_HOST_EQ_D_D_D_D=0.765, FH_HOST_EQ_D_D_D_DB=0.888, HOST_EQ_STATIC=1.172, J_CHICKENPOX_22=0.6, SARE_SUB_ENC_UTF8=0.152]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id MWYRLPbC-S6K for <ldapext@core3.amsl.com>; Thu,  4 Jun 2009 00:29:25 -0700 (PDT)
Received: from eb2bcom.com (165.203.232.72.static.reverse.ltdomains.com [72.232.203.165]) by core3.amsl.com (Postfix) with ESMTP id BE96428C1F2 for <ldapext@ietf.org>; Thu,  4 Jun 2009 00:29:17 -0700 (PDT)
Received: from eth3065.vic.adsl.internode.on.net ([150.101.156.248] helo=[192.168.1.182]) by host.eb2bcom.com with esmtpa (Exim 4.69) (envelope-from <steven.legg@eb2bcom.com>) id 1MC7Nw-0006Jp-CA; Thu, 04 Jun 2009 17:29:12 +1000
Message-ID: <4A2777C3.9020005@eb2bcom.com>
Date: Thu, 04 Jun 2009 17:29:07 +1000
From: Steven Legg <steven.legg@eb2bcom.com>
User-Agent: Thunderbird 2.0.0.21 (Windows/20090302)
MIME-Version: 1.0
To: Yves Dorfsman <yves@zioup.com>
References: <49C497F9.7010200@zioup.com>	<CD3905D4-2A25-4C56-8187-3CE10D46C929@isode.com>	<49C870C6.4010803@zioup.com>	<E94B7389-9A6D-4CB6-BB2C-649CCD3FD15B@Isode.com>	<49CB192E.5050105@zioup.com>	<49CB211C.6070108@eb2bcom.com> <49CB87FE.1050809@zioup.com>	<49CC01DE.6040506@eb2bcom.com> <4A24557D.7030006@zioup.com>
In-Reply-To: <4A24557D.7030006@zioup.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-AntiAbuse: This header was added to track abuse, please include it with any abuse report
X-AntiAbuse: Primary Hostname - host.eb2bcom.com
X-AntiAbuse: Original Domain - ietf.org
X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12]
X-AntiAbuse: Sender Address Domain - eb2bcom.com
X-Source: 
X-Source-Args: 
X-Source-Dir: 
Cc: ldapext@ietf.org
Subject: Re: [ldapext] UTF-8 full support in LDIF / LDIF v2
X-BeenThere: ldapext@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: LDAP Extension Working Group <ldapext.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/ldapext>, <mailto:ldapext-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/ldapext>
List-Post: <mailto:ldapext@ietf.org>
List-Help: <mailto:ldapext-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/ldapext>, <mailto:ldapext-request@ietf.org?subject=subscribe>
X-List-Received-Date: Thu, 04 Jun 2009 07:29:26 -0000

Yves,

Yves Dorfsman wrote:
> I've been thinking about this and trying a few things. My conclusion is 
> that the best solution would be the good old here document.
> 
> objectclass: inetOrgPerson
> organizationName:<<EOT
> The two line
>   company
> EOT
> sn: Jensen
> 
> 
> 
> With the following specifications:
> Any of the following characters (or sequence in the case of CR+LF) can 
> be used as a separator (<SEP>):
> 
> LF (U+000A), CR (U+000D), CR+LF (U+000D followed by U+000A), NEL 
> (U+0085), FF (U+000C), LS (U+2028), PS U+2029)
> 
> Any sequence of characters can be used instead of EOT, but cannot 
> include a separator character. The same sequence has to be used at the 
> begining and the end.
> 
> Any UTF-8 character, except separators, can be used on each line.
> 
> Any separator can be used to separate the lines.
> 
> The text start after EOT<SEP>, and finishes with the last character 
> before <SEP>EOT. The organization name in the example above is exactly 
> two lines, the last separator is not part of the text.
> 
> No need or possibility to escape characters, no possibility of folding 
> lines  .

I would be more amenable to a here-document scheme if there were no
special treatment for line separators. That is, if the UTF-8 character
sequence between the end of the introducer and the beginning of the
terminator is the literal directory attribute value without any modification.
That means that any line separators in the attribute value come out
as is, and may be different from the usual LDIF line separator.
Some tools will display these line breaks nicely and some won't, but even
a misaligned display is more accessible than a base64 representation of
the value.

The downside of a here-document scheme is the need to scan the value
before hand to make sure it doesn't contain the terminator sequence,
which becomes awkward for streaming applications, so I'd be inclined to
keep the current scheme as an alternative.

Regards,
Steven

> 
> 
> 

From steven.legg@eb2bcom.com  Thu Jun  4 00:29:45 2009
Return-Path: <steven.legg@eb2bcom.com>
X-Original-To: ldapext@core3.amsl.com
Delivered-To: ldapext@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 718CB28C273 for <ldapext@core3.amsl.com>; Thu,  4 Jun 2009 00:29:45 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: 1.278
X-Spam-Level: *
X-Spam-Status: No, score=1.278 tagged_above=-999 required=5 tests=[AWL=-0.300,  BAYES_00=-2.599, FH_HOST_EQ_D_D_D_D=0.765, FH_HOST_EQ_D_D_D_DB=0.888,  HOST_EQ_STATIC=1.172, J_CHICKENPOX_22=0.6, J_CHICKENPOX_43=0.6, SARE_SUB_ENC_UTF8=0.152]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id yHT7tAcUe9mu for <ldapext@core3.amsl.com>; Thu,  4 Jun 2009 00:29:44 -0700 (PDT)
Received: from eb2bcom.com (165.203.232.72.static.reverse.ltdomains.com [72.232.203.165]) by core3.amsl.com (Postfix) with ESMTP id 1AD243A6904 for <ldapext@ietf.org>; Thu,  4 Jun 2009 00:29:35 -0700 (PDT)
Received: from eth3065.vic.adsl.internode.on.net ([150.101.156.248] helo=[192.168.1.182]) by host.eb2bcom.com with esmtpa (Exim 4.69) (envelope-from <steven.legg@eb2bcom.com>) id 1MC7OH-0006Jz-6K; Thu, 04 Jun 2009 17:29:33 +1000
Message-ID: <4A2777D7.6090309@eb2bcom.com>
Date: Thu, 04 Jun 2009 17:29:27 +1000
From: Steven Legg <steven.legg@eb2bcom.com>
User-Agent: Thunderbird 2.0.0.21 (Windows/20090302)
MIME-Version: 1.0
To: Kurt Zeilenga <Kurt.Zeilenga@Isode.com>
References: <49C497F9.7010200@zioup.com>	<CD3905D4-2A25-4C56-8187-3CE10D46C929@isode.com>	<49C870C6.4010803@zioup.com>	<E94B7389-9A6D-4CB6-BB2C-649CCD3FD15B@Isode.com>	<49CB192E.5050105@zioup.com>	<49CB211C.6070108@eb2bcom.com>	<49CB87FE.1050809@zioup.com>	<49CC01DE.6040506@eb2bcom.com>	<4A24557D.7030006@zioup.com> <4A26A05D.8040105@zioup.com> <245BF18B-2066-4E36-9502-16F4A3140D9E@Isode.com>
In-Reply-To: <245BF18B-2066-4E36-9502-16F4A3140D9E@Isode.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-AntiAbuse: This header was added to track abuse, please include it with any abuse report
X-AntiAbuse: Primary Hostname - host.eb2bcom.com
X-AntiAbuse: Original Domain - ietf.org
X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12]
X-AntiAbuse: Sender Address Domain - eb2bcom.com
X-Source: 
X-Source-Args: 
X-Source-Dir: 
Cc: ldapext@ietf.org
Subject: Re: [ldapext] UTF-8 full support in LDIF / LDIF v2
X-BeenThere: ldapext@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: LDAP Extension Working Group <ldapext.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/ldapext>, <mailto:ldapext-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/ldapext>
List-Post: <mailto:ldapext@ietf.org>
List-Help: <mailto:ldapext-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/ldapext>, <mailto:ldapext-request@ietf.org?subject=subscribe>
X-List-Received-Date: Thu, 04 Jun 2009 07:29:45 -0000

Kurt,

Kurt Zeilenga wrote:
> 
> On Jun 3, 2009, at 9:10 AM, Yves Dorfsman wrote:
>> Is the idea of a here document syntax too ridiculous ?
> 
> There are a number of problems with it.  Personally, I think what Steven 
> already offered (and likely implemented)

I have.

> is better, though I am 
> concerned about line separators.

Me too, but at least it doesn't matter for XML.

 > As Howard comments kind of suggests,
> when you have a value which is multi-lined, it's the syntax that 
> controls what line separators are used, not the LDIF.  For instance, in 
> some syntaxes, a $ is used to as a line separator.
> 
> The problem with your proposal, and Steven's, is that LDIF line 
> separators and value line separators are one and the same thing.  While 
> one might be case occasionally, it cannot be expected to be generally 
> the case.
> 
> LDIF is first and foremost an interchange format.  Conversion from LDAP 
> PDU->LDIF Record->LDAP PDU MUST produce as output the input, octet for 
> octet for every "data" component (the DN, every attribute description 
> and associated values, etc.).

That's highly desirable for directory to directory interchange, but LDIF
is also used for composing data from various data sources to put in a
directory and to extract data from a directory to put in other data
sources. The octet-for-octet preservation usually doesn't apply in these
other cases and the need to turn line-based data such as XML documents
into base64 encodings is a serious impediment, hence the reason Andrew
and I wrote the Internet-draft.

> 
>> Is UTF-8 support in LDIF not that important ?
> 
> LDIF being a proper interchange format is important.  UTF-8 support 
> (other than being able to interchange values whose syntax is UTF-8 
> encoded) is cosmetic.
> 
> Adding UTF-8 support does appear to be in support of improving LDIF as a 
> proper interchange format.  It seems to be driven by other goals, such 
> as trying to make LDIF files displayable.   Given that LDAP does not 
> constrain attribute value syntaxes (even directory strings can contain 
> arbitrary sequences of Unicode code points), the goal of making LDIF 
> files displayable is not terribly feasible.
> 
> I note that even today, ASCII LDIF files might not display properly 
> without special handling, such as for line separators.  But with UTF-8, 
> line separators are only the tip of iceberg of display problems.
> 
> I'm not convinced that removing the ASCII restrictions will be a good 
> thing.  Not only do I doubt it will have a net positive on 
> displayability of LDIF for those who have a displayability goal (I don't 
> this goal), I think it will have a net negative impact on 
> interoperability and user confusion, such as when the user creates a 
> file using one Unicode normalization algorithm, but is trying to set 
> values which require a different Unicode normalization value.

The user is not going to directly enter a base64 encoded value. They
would use a tool that has those same normalization issues to produce
a UTF-8 character stream that has to be passed to another tool to turn
into base64. The issues exist anyway. It is just a question of where.

BTW, it is not my intent to replace LDIFv1 for pure directory to directory
interchange. I just need the option to produce and consume something more
amenable to human editing and batch processing, where appropriate.

Regards,
Steven

> 
>> Am I the only one thinking xml is not a good replacement for LDIF,
> 
> There already exists a number of XML replacements of LDIF, such as 
> DSML...  so I guess at least some do think XML is a good replacement for 
> LDIF.
> 
>> if so, should we help Steven with the xmled RFC ?
> 
> What Steven and Andrew have done is define an extension for LDIF to 
> allow XML values to be represented in a human-readable format instead of 
> requiring the use use of base64.  Unfortunately his proposal has 
> interchange issues (see the I-D's security considerations section).  
> This, I think, is a fatal problem with this extension.
> 
> -- Kurt
> 
> 
>>
>>
>> Thanks.
>>
>>
>> Yves Dorfsman wrote:
>>> Steven Legg wrote:
>>>>>>>> See http://www.xmled.info/drafts/draft-sciberras-xed-eldif-05.txt
>>> I did look at it, personally I find it difficult for humans, for 
>>> diff'ing etc... XML has its place, but so does pure text.
>>>>> Yes I was wondering about that, do we need multi-line values as 
>>>>> work around because schemas aren't precise enough ?
>>>>
>>>> No, we need them because sheets of paper, computer screens and RFCs are
>>>> not infinitely wide. :-) Human-readability, line breaks and 
>>>> indenting tend
>>>> to go hand-in-hand.
>>> I've been thinking about this and trying a few things. My conclusion 
>>> is that the best solution would be the good old here document.
>>> objectclass: inetOrgPerson
>>> organizationName:<<EOT
>>> The two line
>>>  company
>>> EOT
>>> sn: Jensen
>>> With the following specifications:
>>> Any of the following characters (or sequence in the case of CR+LF) 
>>> can be used as a separator (<SEP>):
>>> LF (U+000A), CR (U+000D), CR+LF (U+000D followed by U+000A), NEL 
>>> (U+0085), FF (U+000C), LS (U+2028), PS U+2029)
>>> Any sequence of characters can be used instead of EOT, but cannot 
>>> include a separator character. The same sequence has to be used at 
>>> the begining and the end.
>>> Any UTF-8 character, except separators, can be used on each line.
>>> Any separator can be used to separate the lines.
>>> The text start after EOT<SEP>, and finishes with the last character 
>>> before <SEP>EOT. The organization name in the example above is 
>>> exactly two lines, the last separator is not part of the text.
>>> No need or possibility to escape characters, no possibility of 
>>> folding lines  .
>>
>>
>> -- 
>> Yves.
>> http://www.sollers.ca/
>>
>> _______________________________________________
>> Ldapext mailing list
>> Ldapext@ietf.org
>> https://www.ietf.org/mailman/listinfo/ldapext
> 
> _______________________________________________
> Ldapext mailing list
> Ldapext@ietf.org
> https://www.ietf.org/mailman/listinfo/ldapext

From michael@stroeder.com  Wed Jun 10 03:43:14 2009
Return-Path: <michael@stroeder.com>
X-Original-To: ldapext@core3.amsl.com
Delivered-To: ldapext@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id E4C633A6E32 for <ldapext@core3.amsl.com>; Wed, 10 Jun 2009 03:43:14 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.147
X-Spam-Level: 
X-Spam-Status: No, score=-2.147 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, MIME_8BIT_HEADER=0.3, SARE_SUB_ENC_UTF8=0.152]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 3bqj0Fsme6pr for <ldapext@core3.amsl.com>; Wed, 10 Jun 2009 03:43:14 -0700 (PDT)
Received: from srv1.stroeder.com (srv1.stroeder.com [213.240.180.113]) by core3.amsl.com (Postfix) with ESMTP id 03B8A3A6E2D for <ldapext@ietf.org>; Wed, 10 Jun 2009 03:43:13 -0700 (PDT)
Received: from localhost (localhost [127.0.0.1]) by srv1.stroeder.com (Postfix) with ESMTP id 46F334E155 for <ldapext@ietf.org>; Wed, 10 Jun 2009 12:43:17 +0200 (CEST)
X-Virus-Scanned: amavisd-new at stroeder.com
Received: from srv1.stroeder.com ([127.0.0.1]) by localhost (srv1.stroeder.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id BNhSh+0qFq2q for <ldapext@ietf.org>; Wed, 10 Jun 2009 12:43:15 +0200 (CEST)
Received: from [10.1.0.2] (unknown [10.1.0.2]) by srv1.stroeder.com (Postfix) with ESMTP id 705A74E147 for <ldapext@ietf.org>; Wed, 10 Jun 2009 12:43:14 +0200 (CEST)
Message-ID: <4A2F8E42.2030202@stroeder.com>
Date: Wed, 10 Jun 2009 12:43:14 +0200
From: =?ISO-8859-1?Q?Michael_Str=F6der?= <michael@stroeder.com>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.22) Gecko/20090606 SeaMonkey/1.1.17
MIME-Version: 1.0
To: ldapext@ietf.org
References: <49C497F9.7010200@zioup.com> <49C870C6.4010803@zioup.com>	<E94B7389-9A6D-4CB6-BB2C-649CCD3FD15B@Isode.com>	<49CB192E.5050105@zioup.com> <49CB211C.6070108@eb2bcom.com>	<49CB87FE.1050809@zioup.com> <49CC01DE.6040506@eb2bcom.com>	<4A24557D.7030006@zioup.com> <4A26A05D.8040105@zioup.com>	<245BF18B-2066-4E36-9502-16F4A3140D9E@Isode.com> <b662a94e0906031754n217f96c8t55e1e0c34f11bb86@mail.gmail.com>
In-Reply-To: <b662a94e0906031754n217f96c8t55e1e0c34f11bb86@mail.gmail.com>
X-Enigmail-Version: 0.95.7
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Subject: Re: [ldapext] UTF-8 full support in LDIF / LDIF v2
X-BeenThere: ldapext@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: LDAP Extension Working Group <ldapext.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/ldapext>, <mailto:ldapext-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/ldapext>
List-Post: <mailto:ldapext@ietf.org>
List-Help: <mailto:ldapext-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/ldapext>, <mailto:ldapext-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 10 Jun 2009 10:43:15 -0000

Jim Willeke wrote:
> Perhaps what is really required is a good tool to display and edit the
> existing LDIF formatted files that in a view that is deemed acceptable.

Whatever "good tool" means to you. People tend to strongly disagree on
user interfaces. ;-)

It would boil down to the same problems you have when displaying
atribute values of an entry or presenting an input form for editing an
entry received via LDAP. This is not feasible without making assumptions
about data and/or looking closely at the subschema. But when solely
looking at a LDIF file you usually don't have the accompanying subschema
subentry available.

In my web2ldap you can switch the input form to LDIF (and back to
template/table form) when editing an entry. It accepts LDIF with UTF-8
values (converted from what the browser sent as charset) as input
unfolding multiple lines according to LDIF rules. Attribute values of
syntax DirectoryString containing non-ASCII chars are displayed
base64-encoded. This all is very simple and I'd not consider it to be
completely what's discussed here but at least it helps copying&pasting
(parts of) an LDIF export.

Ciao, Michael.

From yves@zioup.com  Wed Jun 10 22:35:44 2009
Return-Path: <yves@zioup.com>
X-Original-To: ldapext@core3.amsl.com
Delivered-To: ldapext@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 5CE773A67AC for <ldapext@core3.amsl.com>; Wed, 10 Jun 2009 22:35:44 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.847
X-Spam-Level: 
X-Spam-Status: No, score=-1.847 tagged_above=-999 required=5 tests=[AWL=0.600,  BAYES_00=-2.599, SARE_SUB_ENC_UTF8=0.152]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id JXU-e9Mp4tlc for <ldapext@core3.amsl.com>; Wed, 10 Jun 2009 22:35:43 -0700 (PDT)
Received: from idcmail-mo2no.shaw.ca (idcmail-mo2no.shaw.ca [64.59.134.9]) by core3.amsl.com (Postfix) with ESMTP id 421E33A67A7 for <ldapext@ietf.org>; Wed, 10 Jun 2009 22:35:43 -0700 (PDT)
Received: from pd7ml1no-ssvc.prod.shaw.ca ([10.0.153.161]) by pd5mo1no-svcs.prod.shaw.ca with ESMTP; 10 Jun 2009 23:34:46 -0600
X-Cloudmark-SP-Filtered: true
X-Cloudmark-SP-Result: v=1.0 c=0 a=J6b0bBEkAAAA:8 a=HbpCPP7ycFMijxtQ8TcA:9 a=mHGZdTVUfniBvjVhyTLqcvJUpqAA:4 a=WPV3hXSGBlIA:10 a=Cguc0gE-TcEA:10
Received: from unknown (HELO home.zioup.com) ([68.147.37.46]) by pd7ml1no-dmz.prod.shaw.ca with ESMTP; 10 Jun 2009 23:34:46 -0600
Received: from [192.168.2.203] (socrates.zioup.net [192.168.2.203]) by home.zioup.com (Postfix) with ESMTP id D660B2801E for <ldapext@ietf.org>; Wed, 10 Jun 2009 23:34:45 -0600 (MDT)
Message-ID: <4A309775.3080406@zioup.com>
Date: Wed, 10 Jun 2009 23:34:45 -0600
From: Yves Dorfsman <yves@zioup.com>
User-Agent: Thunderbird 2.0.0.21 (X11/20090409)
MIME-Version: 1.0
To: ldapext@ietf.org
References: <49C497F9.7010200@zioup.com>	<CD3905D4-2A25-4C56-8187-3CE10D46C929@isode.com>	<49C870C6.4010803@zioup.com>	<E94B7389-9A6D-4CB6-BB2C-649CCD3FD15B@Isode.com>	<49CB192E.5050105@zioup.com>	<49CB211C.6070108@eb2bcom.com> <49CB87FE.1050809@zioup.com>	<49CC01DE.6040506@eb2bcom.com> <4A24557D.7030006@zioup.com> <4A26A05D.8040105@zioup.com> <245BF18B-2066-4E36-9502-16F4A3140D9E@Isode.com>
In-Reply-To: <245BF18B-2066-4E36-9502-16F4A3140D9E@Isode.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Subject: Re: [ldapext] UTF-8 full support in LDIF / LDIF v2
X-BeenThere: ldapext@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: LDAP Extension Working Group <ldapext.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/ldapext>, <mailto:ldapext-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/ldapext>
List-Post: <mailto:ldapext@ietf.org>
List-Help: <mailto:ldapext-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/ldapext>, <mailto:ldapext-request@ietf.org?subject=subscribe>
X-List-Received-Date: Thu, 11 Jun 2009 05:35:44 -0000

Kurt Zeilenga wrote:
> 
> There are a number of problems with it.  Personally, I think what Steven 
> already offered (and likely implemented) is better, though I am 

My problem with Steven's solution is that it is half LDIF, half XML. As I 
have mentioned earlier, I thing XML has its place, and maybe DSML should be 
fixed or re-invented, but for other application, I find the simplicity of 
LDIF an advantage ; unfortunately, having to base64 encode anything that's 
not 7 bit ASCII takes away some of its simplicity.

> concerned about line separators.  As Howard comments kind of suggests, 
> when you have a value which is multi-lined, 

I have never run into the situation where I needed a multi-line value in an 
LDAP directory and was surprised by the need, but Steven brought this up 
earlier in the thread and said that he has a real-world need for it, and 
that the lack of a syntax for it in my proposition for an updated LDIF 
format was an issue.


> The problem with your proposal, and Steven's, is that LDIF line 
> separators and value line separators are one and the same thing.  While 
> one might be case occasionally, it cannot be expected to be generally 
> the case.

On the contrary, both Steven's solution and mine separate the lines but do 
not impose a line separator. Steven delimits his line with the <item></item> 
syntax, while I let the user choose any line separator out of the half dozen 
that has been used throughout the history of computing.

Our syntaxes are clear enough to let the import process know that those are 
separate lines, and the import process or the LDAP server can choose 
whichever line separator it wants. Making the line separator part of the 
data will create cross-platform issues. The LDAP server or actually the LDAP 
client should choose which line separator to use for its context/platform.


> Adding UTF-8 support does appear to be in support of improving LDIF as a 
> proper interchange format.  It seems to be driven by other goals, such 
> as trying to make LDIF files displayable.   

Yes and no. My main reason for pushing this is diffing. You run into a 
problem and you want to diff the original and the problematic LDIF export of 
your directory. Having half of your LDIF file base64 encoded makes it a lot 
more difficult to pin point the problem. If you are right, that LDIF is 
purely for exchanging information between applications, never to be looked 
at by humans, then why is the current version so human friendly ?


> I'm not convinced that removing the ASCII restrictions will be a good 
> thing.  Not only do I doubt it will have a net positive on 
> displayability of LDIF for those who have a displayability goal (I don't 
> this goal), I think it will have a net negative impact on 
> interoperability and user confusion, such as when the user creates a 
> file using one Unicode normalization algorithm, but is trying to set 
> values which require a different Unicode normalization value.

How so ?
In the current version, you have to encode your Unicode to UTF-8, and then 
encode it again to base64. With my proposal, you would get the exact same 
UTF-8 strings as you do today, but they would not be (or would not have to 
be) encoded in base64.

This is not a rebuttal of your argument, I am truly interested in 
understanding what you mean here (in the same way I was glad somebody 
brought up the issue of Right To Left characters, as I had not thought about 
it). Maybe it is a problem that we can address ?


>> if so, should we help Steven with the xmled RFC ?
> 
> What Steven and Andrew have done is define an extension for LDIF to 
> allow XML values to be represented in a human-readable format instead of 
> requiring the use use of base64.  Unfortunately his proposal has 
> interchange issues (see the I-D's security considerations section).  
> This, I think, is a fatal problem with this extension.

So really this is the issue, should the value of the line separator be part 
of the data, or should everybody (LDIF importers/exporters, LDAP servers, 
LDAP clients) treat multi-line entries as just that, several lines, and 
choose their own line separator ? (in case I wasn't clear earlier, I am in 
favour of the latter).

-- 
Yves.
http://www.sollers.ca/


From yves@zioup.com  Wed Jun 10 22:48:21 2009
Return-Path: <yves@zioup.com>
X-Original-To: ldapext@core3.amsl.com
Delivered-To: ldapext@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 275603A686C for <ldapext@core3.amsl.com>; Wed, 10 Jun 2009 22:48:21 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.047
X-Spam-Level: 
X-Spam-Status: No, score=-2.047 tagged_above=-999 required=5 tests=[AWL=0.400,  BAYES_00=-2.599, SARE_SUB_ENC_UTF8=0.152]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Ic-dLRY3ku6v for <ldapext@core3.amsl.com>; Wed, 10 Jun 2009 22:48:20 -0700 (PDT)
Received: from idcmail-mo1so.shaw.ca (idcmail-mo1so.shaw.ca [24.71.223.10]) by core3.amsl.com (Postfix) with ESMTP id 444043A6868 for <ldapext@ietf.org>; Wed, 10 Jun 2009 22:48:20 -0700 (PDT)
Received: from pd3ml2so-ssvc.prod.shaw.ca ([10.0.141.138]) by pd4mo1so-svcs.prod.shaw.ca with ESMTP; 10 Jun 2009 23:48:12 -0600
X-Cloudmark-SP-Filtered: true
X-Cloudmark-SP-Result: v=1.0 c=0 a=J6b0bBEkAAAA:8 a=p8kvgdrwQCslXSy7le0A:9 a=JbM2doC5_7lPVeHQrsUA:7 a=tZUvUCcAMkTbG0q4BGDjAIEdahEA:4 a=WPV3hXSGBlIA:10 a=Cguc0gE-TcEA:10
Received: from unknown (HELO home.zioup.com) ([68.147.37.46]) by pd3ml2so-dmz.prod.shaw.ca with ESMTP; 10 Jun 2009 23:48:11 -0600
Received: from [192.168.2.203] (socrates.zioup.net [192.168.2.203]) by home.zioup.com (Postfix) with ESMTP id CA4D02801E for <ldapext@ietf.org>; Wed, 10 Jun 2009 23:48:11 -0600 (MDT)
Message-ID: <4A309A9B.1010105@zioup.com>
Date: Wed, 10 Jun 2009 23:48:11 -0600
From: Yves Dorfsman <yves@zioup.com>
User-Agent: Thunderbird 2.0.0.21 (X11/20090409)
MIME-Version: 1.0
To: ldapext@ietf.org
References: <49C497F9.7010200@zioup.com>	<CD3905D4-2A25-4C56-8187-3CE10D46C929@isode.com>	<49C870C6.4010803@zioup.com>	<E94B7389-9A6D-4CB6-BB2C-649CCD3FD15B@Isode.com>	<49CB192E.5050105@zioup.com>	<49CB211C.6070108@eb2bcom.com> <49CB87FE.1050809@zioup.com>	<49CC01DE.6040506@eb2bcom.com> <4A24557D.7030006@zioup.com> <4A2777C3.9020005@eb2bcom.com>
In-Reply-To: <4A2777C3.9020005@eb2bcom.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Subject: Re: [ldapext] UTF-8 full support in LDIF / LDIF v2
X-BeenThere: ldapext@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: LDAP Extension Working Group <ldapext.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/ldapext>, <mailto:ldapext-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/ldapext>
List-Post: <mailto:ldapext@ietf.org>
List-Help: <mailto:ldapext-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/ldapext>, <mailto:ldapext-request@ietf.org?subject=subscribe>
X-List-Received-Date: Thu, 11 Jun 2009 05:48:21 -0000

Steven Legg wrote:
> 
> I would be more amenable to a here-document scheme if there were no
> special treatment for line separators. That is, if the UTF-8 character
> sequence between the end of the introducer and the beginning of the
> terminator is the literal directory attribute value without any 
> modification.

That would be easy to implement, just pick one line separators instead of 
allowing several. But, is this the right way of doing things ?

When you deal with multi-line entries, isn't the LDAP client at the end of 
the chain modifying your line separator into a different one, one 
appropriate for its platform ?

If so, how much more difficult would it be to let every element in the chain 
use whichever line separator they want, and translate as needed ?


> The downside of a here-document scheme is the need to scan the value
> before hand to make sure it doesn't contain the terminator sequence,
> which becomes awkward for streaming applications, so I'd be inclined to
> keep the current scheme as an alternative.

It should be easy to pick a sufficiently complicated word that you know will 
never appear in any of your entries (e.g.: X**ENDOFTEXTDONOTUSE**X), and you 
can still scan for it from your entries and raise an exception in the 
unlikely case that it does appear (no need to scan ahead, still fool proof). 
This is a slight inconvenient, but the advantages of not having to escape 
characters, and of making the file much easier to read make it, in my 
opinion, worth it.

-- 
Yves.
http://www.sollers.ca/


From michael@stroeder.com  Thu Jun 11 08:11:28 2009
Return-Path: <michael@stroeder.com>
X-Original-To: ldapext@core3.amsl.com
Delivered-To: ldapext@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id E073B3A6C9C for <ldapext@core3.amsl.com>; Thu, 11 Jun 2009 08:11:28 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.147
X-Spam-Level: 
X-Spam-Status: No, score=-2.147 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, MIME_8BIT_HEADER=0.3, SARE_SUB_ENC_UTF8=0.152]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id GkY8kxARFSxL for <ldapext@core3.amsl.com>; Thu, 11 Jun 2009 08:11:28 -0700 (PDT)
Received: from srv1.stroeder.com (srv1.stroeder.com [213.240.180.113]) by core3.amsl.com (Postfix) with ESMTP id 013633A6C75 for <ldapext@ietf.org>; Thu, 11 Jun 2009 08:11:27 -0700 (PDT)
Received: from localhost (localhost [127.0.0.1]) by srv1.stroeder.com (Postfix) with ESMTP id 12BD34E06F; Thu, 11 Jun 2009 17:11:28 +0200 (CEST)
X-Virus-Scanned: amavisd-new at stroeder.com
Received: from srv1.stroeder.com ([127.0.0.1]) by localhost (srv1.stroeder.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 4C+jTeNb+fz0; Thu, 11 Jun 2009 17:11:25 +0200 (CEST)
Received: from [10.1.0.2] (unknown [10.1.0.2]) by srv1.stroeder.com (Postfix) with ESMTP id 5C9D64E06D; Thu, 11 Jun 2009 17:11:24 +0200 (CEST)
Message-ID: <4A311E8D.40405@stroeder.com>
Date: Thu, 11 Jun 2009 17:11:09 +0200
From: =?ISO-8859-1?Q?Michael_Str=F6der?= <michael@stroeder.com>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.22) Gecko/20090606 SeaMonkey/1.1.17
MIME-Version: 1.0
To: Yves Dorfsman <yves@zioup.com>
References: <49C497F9.7010200@zioup.com>	<CD3905D4-2A25-4C56-8187-3CE10D46C929@isode.com>	<49C870C6.4010803@zioup.com>	<E94B7389-9A6D-4CB6-BB2C-649CCD3FD15B@Isode.com>	<49CB192E.5050105@zioup.com>	<49CB211C.6070108@eb2bcom.com>	<49CB87FE.1050809@zioup.com>	<49CC01DE.6040506@eb2bcom.com>	<4A24557D.7030006@zioup.com> <4A2777C3.9020005@eb2bcom.com> <4A309A9B.1010105@zioup.com>
In-Reply-To: <4A309A9B.1010105@zioup.com>
X-Enigmail-Version: 0.95.7
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Cc: ldapext@ietf.org
Subject: Re: [ldapext] UTF-8 full support in LDIF / LDIF v2
X-BeenThere: ldapext@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: LDAP Extension Working Group <ldapext.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/ldapext>, <mailto:ldapext-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/ldapext>
List-Post: <mailto:ldapext@ietf.org>
List-Help: <mailto:ldapext-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/ldapext>, <mailto:ldapext-request@ietf.org?subject=subscribe>
X-List-Received-Date: Thu, 11 Jun 2009 15:11:29 -0000

Yves Dorfsman wrote:
> When you deal with multi-line entries,

Let us be more precise here: I guess you meant multi-line attribute values.

> isn't the LDAP client at the end
> of the chain modifying your line separator into a different one, one
> appropriate for its platform ?

The LDAP client is supposed to send on the wire the correct LDAP string
encoding of the attribute value. Local display methods and handling of
user input data within the UI of the client is out-of-scope here.

> If so, how much more difficult would it be to let every element in the
> chain use whichever line separator they want, and translate as needed ?

???

>> The downside of a here-document scheme is the need to scan the value
>> before hand to make sure it doesn't contain the terminator sequence,
>> which becomes awkward for streaming applications, so I'd be inclined to
>> keep the current scheme as an alternative.
> 
> It should be easy to pick a sufficiently complicated word that you know
> will never appear in any of your entries (e.g.:
> X**ENDOFTEXTDONOTUSE**X),

I'm against the here-document scheme. Use decent LDIF module for your
favourite scripting language.

Ciao, Michael.

From michael@stroeder.com  Thu Jun 11 08:12:42 2009
Return-Path: <michael@stroeder.com>
X-Original-To: ldapext@core3.amsl.com
Delivered-To: ldapext@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id EB4D93A6A71 for <ldapext@core3.amsl.com>; Thu, 11 Jun 2009 08:12:42 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.147
X-Spam-Level: 
X-Spam-Status: No, score=-2.147 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, MIME_8BIT_HEADER=0.3, SARE_SUB_ENC_UTF8=0.152]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id gPOERRaEmKT8 for <ldapext@core3.amsl.com>; Thu, 11 Jun 2009 08:12:42 -0700 (PDT)
Received: from srv1.stroeder.com (srv1.stroeder.com [213.240.180.113]) by core3.amsl.com (Postfix) with ESMTP id E7FEF3A691C for <ldapext@ietf.org>; Thu, 11 Jun 2009 08:12:41 -0700 (PDT)
Received: from localhost (localhost [127.0.0.1]) by srv1.stroeder.com (Postfix) with ESMTP id 8273A4E071; Thu, 11 Jun 2009 17:12:43 +0200 (CEST)
X-Virus-Scanned: amavisd-new at stroeder.com
Received: from srv1.stroeder.com ([127.0.0.1]) by localhost (srv1.stroeder.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id f1+kc4wsRbuE; Thu, 11 Jun 2009 17:12:39 +0200 (CEST)
Received: from [10.1.0.2] (unknown [10.1.0.2]) by srv1.stroeder.com (Postfix) with ESMTP id 77F774E070; Thu, 11 Jun 2009 17:12:39 +0200 (CEST)
Message-ID: <4A311ED1.1030202@stroeder.com>
Date: Thu, 11 Jun 2009 17:12:17 +0200
From: =?ISO-8859-1?Q?Michael_Str=F6der?= <michael@stroeder.com>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.22) Gecko/20090606 SeaMonkey/1.1.17
MIME-Version: 1.0
To: Yves Dorfsman <yves@zioup.com>
References: <49C497F9.7010200@zioup.com>	<CD3905D4-2A25-4C56-8187-3CE10D46C929@isode.com>	<49C870C6.4010803@zioup.com>	<E94B7389-9A6D-4CB6-BB2C-649CCD3FD15B@Isode.com>	<49CB192E.5050105@zioup.com>	<49CB211C.6070108@eb2bcom.com>	<49CB87FE.1050809@zioup.com>	<49CC01DE.6040506@eb2bcom.com>	<4A24557D.7030006@zioup.com> <4A26A05D.8040105@zioup.com>	<245BF18B-2066-4E36-9502-16F4A3140D9E@Isode.com> <4A309775.3080406@zioup.com>
In-Reply-To: <4A309775.3080406@zioup.com>
X-Enigmail-Version: 0.95.7
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Cc: ldapext@ietf.org
Subject: Re: [ldapext] UTF-8 full support in LDIF / LDIF v2
X-BeenThere: ldapext@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: LDAP Extension Working Group <ldapext.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/ldapext>, <mailto:ldapext-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/ldapext>
List-Post: <mailto:ldapext@ietf.org>
List-Help: <mailto:ldapext-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/ldapext>, <mailto:ldapext-request@ietf.org?subject=subscribe>
X-List-Received-Date: Thu, 11 Jun 2009 15:12:43 -0000

Yves Dorfsman wrote:
> Kurt Zeilenga wrote:
>> Adding UTF-8 support does appear to be in support of improving LDIF as
>> a proper interchange format.  It seems to be driven by other goals,
>> such as trying to make LDIF files displayable.   
> 
> Yes and no. My main reason for pushing this is diffing. You run into a
> problem and you want to diff the original and the problematic LDIF
> export of your directory.

Sorry, but this is a very bad use-case. Given the fact that there are
decent LDIF modules for various programming languages you can implement
a robust delta-generation of LDIF data. Doing that with line-oriented
text tools is bad practice anyway!

> If you are right, that LDIF is purely for exchanging information
> between applications, never to be looked at by humans, then why is
> the current version so human friendly ?

Most parts of LDIF data is human-readable because most data is ASCII and
most consoles or other apps do not have problems displaying ASCII. This
does not say anything about whether to lift the ASCII restriction or not.

>> I'm not convinced that removing the ASCII restrictions will be a good
>> thing.  Not only do I doubt it will have a net positive on
>> displayability of LDIF for those who have a displayability goal (I
>> don't this goal), I think it will have a net negative impact on
>> interoperability and user confusion, such as when the user creates a
>> file using one Unicode normalization algorithm, but is trying to set
>> values which require a different Unicode normalization value.
> 
> How so ?
> In the current version, you have to encode your Unicode to UTF-8, and
> then encode it again to base64. With my proposal, you would get the
> exact same UTF-8 strings as you do today, but they would not be (or
> would not have to be) encoded in base64.

I agree with Yves here.

> So really this is the issue, should the value of the line separator be
> part of the data, or should everybody (LDIF importers/exporters, LDAP
> servers, LDAP clients) treat multi-line entries as just that, several
> lines, and choose their own line separator ?

Line separators for the attribute value itself is subject of the LDAP
syntax, isn't it? So if e.g. the LDAP syntax DirectoryString allows a
line separator (don't know) it has to be distinguished from the line
separator used for LDIF line folding.

Ciao, Michael.

From Kurt.Zeilenga@Isode.com  Thu Jun 11 19:14:05 2009
Return-Path: <Kurt.Zeilenga@Isode.com>
X-Original-To: ldapext@core3.amsl.com
Delivered-To: ldapext@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 512943A69CD for <ldapext@core3.amsl.com>; Thu, 11 Jun 2009 19:14:05 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.629
X-Spam-Level: 
X-Spam-Status: No, score=-2.629 tagged_above=-999 required=5 tests=[AWL=-0.482, BAYES_00=-2.599, MIME_8BIT_HEADER=0.3, SARE_SUB_ENC_UTF8=0.152]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id rzqeuDXGaULy for <ldapext@core3.amsl.com>; Thu, 11 Jun 2009 19:14:04 -0700 (PDT)
Received: from boole.openldap.org (boole.openldap.org [IPv6:2001:4f8:3:ba:219:21ff:fe2f:68d5]) by core3.amsl.com (Postfix) with ESMTP id 3A5DF3A6903 for <ldapext@ietf.org>; Thu, 11 Jun 2009 19:13:44 -0700 (PDT)
Received: from [192.168.1.102] (75-141-233-128.dhcp.nv.charter.com [75.141.233.128] (may be forged)) (authenticated bits=0) by boole.openldap.org (8.14.3/8.14.3) with ESMTP id n5C2Dknw018621; Fri, 12 Jun 2009 02:13:47 GMT (envelope-from Kurt.Zeilenga@Isode.com)
Message-Id: <DBDCBBE8-BD10-43D4-A788-27354BBBCADB@Isode.com>
From: Kurt Zeilenga <Kurt.Zeilenga@Isode.com>
To: =?ISO-8859-1?Q?Michael_Str=F6der?= <michael@stroeder.com>
In-Reply-To: <4A311ED1.1030202@stroeder.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed; delsp=yes
Content-Transfer-Encoding: quoted-printable
Mime-Version: 1.0 (Apple Message framework v935.3)
Date: Thu, 11 Jun 2009 19:13:46 -0700
References: <49C497F9.7010200@zioup.com>	<CD3905D4-2A25-4C56-8187-3CE10D46C929@isode.com>	<49C870C6.4010803@zioup.com>	<E94B7389-9A6D-4CB6-BB2C-649CCD3FD15B@Isode.com>	<49CB192E.5050105@zioup.com>	<49CB211C.6070108@eb2bcom.com>	<49CB87FE.1050809@zioup.com>	<49CC01DE.6040506@eb2bcom.com>	<4A24557D.7030006@zioup.com> <4A26A05D.8040105@zioup.com>	<245BF18B-2066-4E36-9502-16F4A3140D9E@Isode.com> <4A309775.3080406@zioup.com> <4A311ED1.1030202@stroeder.com>
X-Mailer: Apple Mail (2.935.3)
Cc: ldapext@ietf.org
Subject: Re: [ldapext] UTF-8 full support in LDIF / LDIF v2
X-BeenThere: ldapext@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: LDAP Extension Working Group <ldapext.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/ldapext>, <mailto:ldapext-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/ldapext>
List-Post: <mailto:ldapext@ietf.org>
List-Help: <mailto:ldapext-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/ldapext>, <mailto:ldapext-request@ietf.org?subject=subscribe>
X-List-Received-Date: Fri, 12 Jun 2009 02:14:05 -0000

On Jun 11, 2009, at 8:12 AM, Michael Str=F6der wrote:

>>
>> If you are right, that LDIF is purely for exchanging information
>> between applications, never to be looked at by humans, then why is
>> the current version so human friendly ?
>
> Most parts of LDIF data is human-readable because most data is ASCII =20=

> and
> most consoles or other apps do not have problems displaying ASCII.

I do not concur with this statement.  I find that most applications =20
can generally deal with one line separator convention in US-ASCII =20
files and there are multiple line separator conventions.

> This does not say anything about whether to lift the ASCII =20
> restriction or not.


The US-ASCII problems might suggest that use of Net-ASCII might have =20
been more appropriate than US-ASCII.

-- Kurt



From Kurt.Zeilenga@Isode.com  Thu Jun 11 20:54:12 2009
Return-Path: <Kurt.Zeilenga@Isode.com>
X-Original-To: ldapext@core3.amsl.com
Delivered-To: ldapext@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 573593A6B95 for <ldapext@core3.amsl.com>; Thu, 11 Jun 2009 20:54:12 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.683
X-Spam-Level: 
X-Spam-Status: No, score=-2.683 tagged_above=-999 required=5 tests=[AWL=-0.236, BAYES_00=-2.599, SARE_SUB_ENC_UTF8=0.152]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id sGFx3BqJmVBK for <ldapext@core3.amsl.com>; Thu, 11 Jun 2009 20:54:11 -0700 (PDT)
Received: from boole.openldap.org (boole.openldap.org [IPv6:2001:4f8:3:ba:219:21ff:fe2f:68d5]) by core3.amsl.com (Postfix) with ESMTP id 8F7733A6B83 for <ldapext@ietf.org>; Thu, 11 Jun 2009 20:54:10 -0700 (PDT)
Received: from [192.168.1.102] (75-141-233-128.dhcp.nv.charter.com [75.141.233.128] (may be forged)) (authenticated bits=0) by boole.openldap.org (8.14.3/8.14.3) with ESMTP id n5C3sAZs021551; Fri, 12 Jun 2009 03:54:11 GMT (envelope-from Kurt.Zeilenga@Isode.com)
Message-Id: <82088FD5-8C97-4698-99D3-785ACF0F5ADD@Isode.com>
From: Kurt Zeilenga <Kurt.Zeilenga@Isode.com>
To: Yves Dorfsman <yves@zioup.com>
In-Reply-To: <4A309775.3080406@zioup.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v935.3)
Date: Thu, 11 Jun 2009 20:54:10 -0700
References: <49C497F9.7010200@zioup.com>	<CD3905D4-2A25-4C56-8187-3CE10D46C929@isode.com>	<49C870C6.4010803@zioup.com>	<E94B7389-9A6D-4CB6-BB2C-649CCD3FD15B@Isode.com>	<49CB192E.5050105@zioup.com>	<49CB211C.6070108@eb2bcom.com> <49CB87FE.1050809@zioup.com>	<49CC01DE.6040506@eb2bcom.com> <4A24557D.7030006@zioup.com> <4A26A05D.8040105@zioup.com> <245BF18B-2066-4E36-9502-16F4A3140D9E@Isode.com> <4A309775.3080406@zioup.com>
X-Mailer: Apple Mail (2.935.3)
Cc: ldapext@ietf.org
Subject: Re: [ldapext] UTF-8 full support in LDIF / LDIF v2
X-BeenThere: ldapext@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: LDAP Extension Working Group <ldapext.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/ldapext>, <mailto:ldapext-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/ldapext>
List-Post: <mailto:ldapext@ietf.org>
List-Help: <mailto:ldapext-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/ldapext>, <mailto:ldapext-request@ietf.org?subject=subscribe>
X-List-Received-Date: Fri, 12 Jun 2009 03:54:12 -0000

On Jun 10, 2009, at 10:34 PM, Yves Dorfsman wrote:

> Kurt Zeilenga wrote:
>> There are a number of problems with it.  Personally, I think what  
>> Steven already offered (and likely implemented) is better, though I  
>> am
>
> My problem with Steven's solution is that it is half LDIF, half XML.  
> As I have mentioned earlier, I thing XML has its place, and maybe  
> DSML should be fixed or re-invented, but for other application, I  
> find the simplicity of LDIF an advantage ; unfortunately, having to  
> base64 encode anything that's not 7 bit ASCII takes away some of its  
> simplicity.
>
>> concerned about line separators.  As Howard comments kind of  
>> suggests, when you have a value which is multi-lined,
>
> I have never run into the situation where I needed a multi-line  
> value in an LDAP directory and was surprised by the need, but Steven  
> brought this up earlier in the thread and said that he has a real- 
> world need for it, and that the lack of a syntax for it in my  
> proposition for an updated LDIF format was an issue.

The problem is that what line separators to use is syntax specific (or  
possible attribute value convention specific).  For instance, an LDAP  
syntax could say multiple lines are to be separated by a particular  
set of code points (such as '$') or it could be simple be a convention  
that an attribute uses a particular set of characters.

To convert a LDIF specific line separator to an attribute value line  
separator requires not only knowledge of the LDAP schema, but  
knowledge of the attribute value conventions not expressed in the LDAP  
schema.

LDIF however was designed to allow a mechanical conversion of LDIF to  
LDAP PDUs without such knowledge.  Requiring implementations to have  
additional knowledge is quite problematic.

>
>> The problem with your proposal, and Steven's, is that LDIF line  
>> separators and value line separators are one and the same thing.   
>> While one might be case occasionally, it cannot be expected to be  
>> generally the case.
>
> On the contrary, both Steven's solution and mine separate the lines  
> but do not impose a line separator. Steven delimits his line with  
> the <item></item> syntax,

I think you confuse XML elements with line separators in XML data.   
Two very different things.

Steven's proposal represents line separators in the XML data using the  
<SEP> production.

> while I let the user choose any line separator out of the half dozen  
> that has been used throughout the history of computing.

The problem here is how line separators in LDIF relate to line  
separators in the value.

Your approach assumes that whatever line separator the user chooses to  
use in the LDIF file is valid per the LDAP value syntax and any  
attribute type specific restrictions.

>
> Our syntaxes are clear enough to let the import process know that  
> those are separate lines, and the import process or the LDAP server  
> can choose whichever line separator it wants.

That requires LDAP schema and attribute type specific restrictions  
knowledge.

> Making the line separator part of the data will create cross- 
> platform issues.

Yes, but this is what your proposal seems to do.  (See below).

> The LDAP server or actually the LDAP client should choose which line  
> separator to use for its context/platform.

Today, LDIF line separators (<SEP>) are not part of the LDAP value.

That is,

foo: X
  Y

is equivalent to:

foo: XY

That is, the LDAP value is merely wrapped over multiple LDIF lines.

Now maybe you meant:

foo:<<EOT
X
Y
EOT

to also be equivalent to:

foo: XY

I don't see any value in offering yet another way to line wrap an LDAP  
value.

I took your proposal as representing foo attribute value "X<SEP>Y"  
where <SEP> was the sequence of characters used in the LDIF to  
separate X and Y.  This is problematic.


>
>
>> Adding UTF-8 support does appear to be in support of improving LDIF  
>> as a proper interchange format.  It seems to be driven by other  
>> goals, such as trying to make LDIF files displayable.
>
> Yes and no. My main reason for pushing this is diffing.

Diffing requires knowledge of LDAP schema.   One might store "foo" and  
get back "FOO" (or any other equivalent value) [See LDAP's data  
preservation requirements].

> You run into a problem and you want to diff the original and the  
> problematic LDIF export of your directory.  Having half of your LDIF  
> file base64 encoded makes it a lot more difficult to pin point the  
> problem.

As Michael noted, there exists LDIF diffing tools (most of which are  
likely not schema aware, and hence show equivalent values as being  
different).

> If you are right, that LDIF is purely for exchanging information  
> between applications, never to be looked at by humans, then why is  
> the current version so human friendly ?

I never claimed LDIF would not be looked at by humans.

I have stated that LDIF, with ASCII restrictions, already suffers from  
some display/editing issues namely due line separator issues.  Lifting  
the ASCII restriction will make these matters far worse.  (Line  
separators are the tip of the displayability/editability iceberg.)


>
>
>> I'm not convinced that removing the ASCII restrictions will be a  
>> good thing.  Not only do I doubt it will have a net positive on  
>> displayability of LDIF for those who have a displayability goal (I  
>> don't this goal), I think it will have a net negative impact on  
>> interoperability and user confusion, such as when the user creates  
>> a file using one Unicode normalization algorithm, but is trying to  
>> set values which require a different Unicode normalization value.
>
> How so ?
> In the current version, you have to encode your Unicode to UTF-8,  
> and then encode it again to base64. With my proposal, you would get  
> the exact same UTF-8 strings as you do today, but they would not be  
> (or would not have to be) encoded in base64.

I see two kinds of problems.

1) This would result in LDIF files which programs designed to display  
UTF-8 encoded Unicode text will not be able to display.  There is a  
user expectation that LDIF files be displayable.  With the current  
LDIF format, we do have some display issues (e.g., line separators),  
but they are limited.  If we remove the ASCII restrictions, we'll run  
into a wide range of display issues.

2) Today we have some separation between (non-ASCII) Unicode LDAP  
attribute values and their LDIF representation.  This separation, I  
think, has some value in that it instills LDIF syntactically  
requirements are LDAP attribute value syntax requirements are  
independent of each other.   Removing this separation, I think, will  
lead to user confusion.

> This is not a rebuttal of your argument, I am truly interested in  
> understanding what you mean here (in the same way I was glad  
> somebody brought up the issue of Right To Left characters, as I had  
> not thought about it). Maybe it is a problem that we can address ?

Keep what separation we do have between LDIF representation of an  
attribute value and the LDAP syntax for the attribute value.

>
>
>>> if so, should we help Steven with the xmled RFC ?
>> What Steven and Andrew have done is define an extension for LDIF to  
>> allow XML values to be represented in a human-readable format  
>> instead of requiring the use use of base64.  Unfortunately his  
>> proposal has interchange issues (see the I-D's security  
>> considerations section).  This, I think, is a fatal problem with  
>> this extension.
>
> So really this is the issue, should the value of the line separator  
> be part of the data, or should everybody (LDIF importers/exporters,  
> LDAP servers, LDAP clients) treat multi-line entries as just that,  
> several lines, and choose their own line separator ? (in case I  
> wasn't clear earlier, I am in favour of the latter).

In RFC 2849, <SEP> is never part of the LDAP attribute value.

In ELDIF extension, certain <SEP>s are part of the LDAP attribute  
value.  This, I think, is problematic.

>
> -- 
> Yves.
> http://www.sollers.ca/
>
> _______________________________________________
> Ldapext mailing list
> Ldapext@ietf.org
> https://www.ietf.org/mailman/listinfo/ldapext


From yves@zioup.com  Thu Jun 11 20:59:53 2009
Return-Path: <yves@zioup.com>
X-Original-To: ldapext@core3.amsl.com
Delivered-To: ldapext@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 234413A6A69 for <ldapext@core3.amsl.com>; Thu, 11 Jun 2009 20:59:53 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.147
X-Spam-Level: 
X-Spam-Status: No, score=-2.147 tagged_above=-999 required=5 tests=[AWL=0.300,  BAYES_00=-2.599, SARE_SUB_ENC_UTF8=0.152]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id QPzYrx-O5tW0 for <ldapext@core3.amsl.com>; Thu, 11 Jun 2009 20:59:52 -0700 (PDT)
Received: from idcmail-mo1so.shaw.ca (idcmail-mo1so.shaw.ca [24.71.223.10]) by core3.amsl.com (Postfix) with ESMTP id EB6A13A6BB0 for <ldapext@ietf.org>; Thu, 11 Jun 2009 20:59:51 -0700 (PDT)
Received: from pd3ml1so-ssvc.prod.shaw.ca ([10.0.141.140]) by pd2mo1so-svcs.prod.shaw.ca with ESMTP; 11 Jun 2009 21:58:51 -0600
X-Cloudmark-SP-Filtered: true
X-Cloudmark-SP-Result: v=1.0 c=0 a=J6b0bBEkAAAA:8 a=ZQ6K8o-MQgZShRbjx1AA:9 a=Ho7_nw2w2vIp6zyO6VwA:7 a=FNxj9JAzM0CEFWT3rxm4S2u1OekA:4 a=WPV3hXSGBlIA:10 a=Cguc0gE-TcEA:10
Received: from unknown (HELO home.zioup.com) ([68.147.37.46]) by pd3ml1so-dmz.prod.shaw.ca with ESMTP; 11 Jun 2009 21:58:51 -0600
Received: from [192.168.2.203] (socrates.zioup.net [192.168.2.203]) by home.zioup.com (Postfix) with ESMTP id 4B2314F005 for <ldapext@ietf.org>; Thu, 11 Jun 2009 21:58:51 -0600 (MDT)
Message-ID: <4A31D27B.3090208@zioup.com>
Date: Thu, 11 Jun 2009 21:58:51 -0600
From: Yves Dorfsman <yves@zioup.com>
User-Agent: Thunderbird 2.0.0.21 (X11/20090409)
MIME-Version: 1.0
To: ldapext@ietf.org
References: <49C497F9.7010200@zioup.com>	<CD3905D4-2A25-4C56-8187-3CE10D46C929@isode.com>	<49C870C6.4010803@zioup.com>	<E94B7389-9A6D-4CB6-BB2C-649CCD3FD15B@Isode.com>	<49CB192E.5050105@zioup.com>	<49CB211C.6070108@eb2bcom.com>	<49CB87FE.1050809@zioup.com>	<49CC01DE.6040506@eb2bcom.com>	<4A24557D.7030006@zioup.com> <4A26A05D.8040105@zioup.com>	<245BF18B-2066-4E36-9502-16F4A3140D9E@Isode.com> <4A309775.3080406@zioup.com> <4A311ED1.1030202@stroeder.com>
In-Reply-To: <4A311ED1.1030202@stroeder.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
Subject: Re: [ldapext] UTF-8 full support in LDIF / LDIF v2
X-BeenThere: ldapext@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: LDAP Extension Working Group <ldapext.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/ldapext>, <mailto:ldapext-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/ldapext>
List-Post: <mailto:ldapext@ietf.org>
List-Help: <mailto:ldapext-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/ldapext>, <mailto:ldapext-request@ietf.org?subject=subscribe>
X-List-Received-Date: Fri, 12 Jun 2009 03:59:53 -0000

Ok, let's split the two issue:

First, is it worth amending the standard to allow non-base64 UTF-8 ?
(forget about multi-line attribute value for a moment).

Looking back (way back) at this thread, I am not the only one seeing value here:


Michael Ströder wrote:
>>> I'm not convinced that removing the ASCII restrictions will be a good
>>>  thing.  Not only do I doubt it will have a net positive on 
>>> displayability of LDIF for those who have a displayability goal (I 
>>> don't this goal), I think it will have a net negative impact on 
>>> interoperability and user confusion, such as when the user creates a 
>>> file using one Unicode normalization algorithm, but is trying to set 
>>> values which require a different Unicode normalization value.
>> How so ? In the current version, you have to encode your Unicode to
>> UTF-8, and then encode it again to base64. With my proposal, you would
>> get the exact same UTF-8 strings as you do today, but they would not be
>> (or would not have to be) encoded in base64.
> 
> I agree with Yves here.


Steven Legg wrote:
>> LDIF is first and foremost an interchange format.  Conversion from LDAP
>> PDU->LDIF Record->LDAP PDU MUST produce as output the input, octet for
>> octet for every "data" component (the DN, every attribute description
>> and associated values, etc.).
> 
> That's highly desirable for directory to directory interchange, but LDIF 
> is also used for composing data from various data sources to put in a 
> directory and to extract data from a directory to put in other data 
> sources. The octet-for-octet preservation usually doesn't apply in these 
> other cases and the need to turn line-based data such as XML documents 
> into base64 encodings is a serious impediment, hence the reason Andrew 
> and I wrote the Internet-draft.


Ludovic Poitou wrote:
>> I went back through the mailing list archive, "charset" came up, but I
 >> can't make sense of who started with it.

 > I probably did.
 > In europe there are lots of directory users and administrators that have
 >  non ascii data they need to transform to LDIF.
 > With simple scripts, turning the data to UTF-8 and then to base64 encoded
 > is a pain. Allowing to specify a charset and then letting the tools doing
 > the conversion to UTF-8 automatically could simplify their life.




Can we try to think of what sort of problem non-encoded UTF-8 would create ?
If there is none, than, could we implement a version 2 that does this ?
The people that don't care for it, can carry on using version 1 ?

-- 
Yves.
http://www.sollers.ca/


From Kurt.Zeilenga@Isode.com  Thu Jun 11 22:14:14 2009
Return-Path: <Kurt.Zeilenga@Isode.com>
X-Original-To: ldapext@core3.amsl.com
Delivered-To: ldapext@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 148FF3A67F7 for <ldapext@core3.amsl.com>; Thu, 11 Jun 2009 22:14:14 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.043
X-Spam-Level: 
X-Spam-Status: No, score=-2.043 tagged_above=-999 required=5 tests=[AWL=-0.796, BAYES_00=-2.599, J_CHICKENPOX_13=0.6, J_CHICKENPOX_43=0.6, SARE_SUB_ENC_UTF8=0.152]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id jJjUCSzVerox for <ldapext@core3.amsl.com>; Thu, 11 Jun 2009 22:14:13 -0700 (PDT)
Received: from boole.openldap.org (boole.openldap.org [IPv6:2001:4f8:3:ba:219:21ff:fe2f:68d5]) by core3.amsl.com (Postfix) with ESMTP id C68883A63CB for <ldapext@ietf.org>; Thu, 11 Jun 2009 22:14:12 -0700 (PDT)
Received: from [192.168.1.102] (75-141-233-128.dhcp.nv.charter.com [75.141.233.128] (may be forged)) (authenticated bits=0) by boole.openldap.org (8.14.3/8.14.3) with ESMTP id n5C5EDm1023775; Fri, 12 Jun 2009 05:14:14 GMT (envelope-from Kurt.Zeilenga@Isode.com)
Message-Id: <35B2A165-CE5D-4650-AADE-CC233F71470E@Isode.com>
From: Kurt Zeilenga <Kurt.Zeilenga@Isode.com>
To: Yves Dorfsman <yves@zioup.com>
In-Reply-To: <4A31D27B.3090208@zioup.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v935.3)
Date: Thu, 11 Jun 2009 22:14:13 -0700
References: <49C497F9.7010200@zioup.com>	<CD3905D4-2A25-4C56-8187-3CE10D46C929@isode.com>	<49C870C6.4010803@zioup.com>	<E94B7389-9A6D-4CB6-BB2C-649CCD3FD15B@Isode.com>	<49CB192E.5050105@zioup.com>	<49CB211C.6070108@eb2bcom.com>	<49CB87FE.1050809@zioup.com>	<49CC01DE.6040506@eb2bcom.com>	<4A24557D.7030006@zioup.com> <4A26A05D.8040105@zioup.com>	<245BF18B-2066-4E36-9502-16F4A3140D9E@Isode.com> <4A309775.3080406@zioup.com> <4A311ED1.1030202@stroeder.com> <4A31D27B.3090208@zioup.com>
X-Mailer: Apple Mail (2.935.3)
Cc: ldapext@ietf.org
Subject: Re: [ldapext] UTF-8 full support in LDIF / LDIF v2
X-BeenThere: ldapext@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: LDAP Extension Working Group <ldapext.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/ldapext>, <mailto:ldapext-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/ldapext>
List-Post: <mailto:ldapext@ietf.org>
List-Help: <mailto:ldapext-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/ldapext>, <mailto:ldapext-request@ietf.org?subject=subscribe>
X-List-Received-Date: Fri, 12 Jun 2009 05:14:14 -0000

Yves,

I think we need clear problem statements, a proposal for addressing  
each problem, a summary of why the proposal does address the problem  
and a statement of what (known) problems the proposal might introduce.

You have noted that the "diffing problem".  But here it's not clear  
whether a) you are wish to determine how two LDIF files differ, b) you  
wish to determine if two LDIF files represent the same LDAP requests,  
or c) you wish to determine how directory information represented in  
the LDIF represented LDAP requests differ and, if so, how.

For a), one can use file comparison tools to determine how two LDIF  
files differ.

For b), one can use an LDIF comparison tool.  For instance, such a  
tool know that
	a: foo
and
	a:foo
and
	a:f
	 o
	 o
and:
	a::Zm9v

represent the same value for the attribute a.


For c), a schema aware tool is need.  For instance,
	a:foo
and
	b:foo

might be equivalent as b could be an alias for a.

Or
	a:foo
and
	a:FOO

could be different but equivalent values.

Simply having a UTF-8 value encoding option doesn't generally solve  
any diffing problem.  In cases b and c, the LDIF encoding of the value  
doesn't matter.  In case a), you'd be introducing another way for LDIF  
files which represented the same LDAP requests to differ.
It seems to me that you want to use file comparision tools to  
accomplish b, you need to have a single way to represent in LDIF a  
particular LDAP request.  As long as there are multiple ways to  
represent a particular LDAP request in LDIF, file comparison tools are  
inadequate for b.

It may be you are thinking that a human would be better able to  
visually detect certain kinds of differences.  However, this assumes  
that removing the ASCII restriction would produce a readily display  
Unicode text.  That, I believe, is a bad assumption.  For instance,  
say a user diff(1) to LDIF files and get:

% diff -u ?.ldif
--- 1.ldif	2009-06-11 22:08:21.000000000 -0700
+++ 2.ldif	2009-06-11 22:08:47.000000000 -0700
@@ -1 +1 @@
-a: f??
+a:f??

where ? represents character not displayable on the user's screen.   
The user might assume the values here are same when they aren't.

This, I hope, illustrates why general file diff'ing tools, like  
diff(1), are suitable only for case a but not b.

Do you have a more compelling use case?

-- Kurt

From michael@stroeder.com  Fri Jun 12 06:38:13 2009
Return-Path: <michael@stroeder.com>
X-Original-To: ldapext@core3.amsl.com
Delivered-To: ldapext@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 537AA3A6E22 for <ldapext@core3.amsl.com>; Fri, 12 Jun 2009 06:38:13 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.147
X-Spam-Level: 
X-Spam-Status: No, score=-2.147 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, MIME_8BIT_HEADER=0.3, SARE_SUB_ENC_UTF8=0.152]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ul7sj-1Pl1pD for <ldapext@core3.amsl.com>; Fri, 12 Jun 2009 06:38:12 -0700 (PDT)
Received: from srv1.stroeder.com (srv1.stroeder.com [213.240.180.113]) by core3.amsl.com (Postfix) with ESMTP id 561BF3A694C for <ldapext@ietf.org>; Fri, 12 Jun 2009 06:38:12 -0700 (PDT)
Received: from localhost (localhost [127.0.0.1]) by srv1.stroeder.com (Postfix) with ESMTP id 148834E0B8; Fri, 12 Jun 2009 15:38:11 +0200 (CEST)
X-Virus-Scanned: amavisd-new at stroeder.com
Received: from srv1.stroeder.com ([127.0.0.1]) by localhost (srv1.stroeder.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 2hLC+m533MY5; Fri, 12 Jun 2009 15:38:03 +0200 (CEST)
Received: from [10.1.0.2] (unknown [10.1.0.2]) by srv1.stroeder.com (Postfix) with ESMTP id 81A224E0A6; Fri, 12 Jun 2009 15:38:03 +0200 (CEST)
Message-ID: <4A325A40.2050802@stroeder.com>
Date: Fri, 12 Jun 2009 15:38:08 +0200
From: =?UTF-8?B?TWljaGFlbCBTdHLDtmRlcg==?= <michael@stroeder.com>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.22) Gecko/20090606 SeaMonkey/1.1.17
MIME-Version: 1.0
To: Yves Dorfsman <yves@zioup.com>
References: <49C497F9.7010200@zioup.com>	<CD3905D4-2A25-4C56-8187-3CE10D46C929@isode.com>	<49C870C6.4010803@zioup.com>	<E94B7389-9A6D-4CB6-BB2C-649CCD3FD15B@Isode.com>	<49CB192E.5050105@zioup.com>	<49CB211C.6070108@eb2bcom.com>	<49CB87FE.1050809@zioup.com>	<49CC01DE.6040506@eb2bcom.com>	<4A24557D.7030006@zioup.com>	<4A26A05D.8040105@zioup.com>	<245BF18B-2066-4E36-9502-16F4A3140D9E@Isode.com>	<4A309775.3080406@zioup.com> <4A311ED1.1030202@stroeder.com> <4A31D27B.3090208@zioup.com>
In-Reply-To: <4A31D27B.3090208@zioup.com>
X-Enigmail-Version: 0.95.7
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Cc: ldapext@ietf.org
Subject: Re: [ldapext] UTF-8 full support in LDIF / LDIF v2
X-BeenThere: ldapext@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: LDAP Extension Working Group <ldapext.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/ldapext>, <mailto:ldapext-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/ldapext>
List-Post: <mailto:ldapext@ietf.org>
List-Help: <mailto:ldapext-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/ldapext>, <mailto:ldapext-request@ietf.org?subject=subscribe>
X-List-Received-Date: Fri, 12 Jun 2009 13:38:13 -0000

Yves Dorfsman wrote:
> 
> Looking back (way back) at this thread, I am not the only one seeing
> value here:
> 
> Michael Ströder wrote:
>>>> I'm not convinced that removing the ASCII restrictions will be a good
>>>>  thing.  Not only do I doubt it will have a net positive on
>>>> displayability of LDIF for those who have a displayability goal (I
>>>> don't this goal), I think it will have a net negative impact on
>>>> interoperability and user confusion, such as when the user creates a
>>>> file using one Unicode normalization algorithm, but is trying to set
>>>> values which require a different Unicode normalization value.
>>> How so ? In the current version, you have to encode your Unicode to
>>> UTF-8, and then encode it again to base64. With my proposal, you would
>>> get the exact same UTF-8 strings as you do today, but they would not be
>>> (or would not have to be) encoded in base64.
>>
>> I agree with Yves here.

Hmm, I agreed on SASL normalization not being an issue.
But you skipped my strong disagreement about your LDIF diffing use-case.

Excerpt from
http://www.ietf.org/mail-archive/web/ldapext/current/msg01759.html

> Michael Ströder wrote:
>> Yves Dorfsman wrote:
>>> Kurt Zeilenga wrote:
>>>> Adding UTF-8 support does appear to be in support of improving LDIF as
>>>> a proper interchange format.  It seems to be driven by other goals,
>>>> such as trying to make LDIF files displayable.   
>>> Yes and no. My main reason for pushing this is diffing. You run into a
>>> problem and you want to diff the original and the problematic LDIF
>>> export of your directory.
>> 
>> Sorry, but this is a very bad use-case. Given the fact that there are
>> decent LDIF modules for various programming languages you can implement
>> a robust delta-generation of LDIF data. Doing that with line-oriented
>> text tools is bad practice anyway!

Ciao, Michael.

From yves@zioup.com  Sun Jun 14 21:29:09 2009
Return-Path: <yves@zioup.com>
X-Original-To: ldapext@core3.amsl.com
Delivered-To: ldapext@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id D5A603A6ABC for <ldapext@core3.amsl.com>; Sun, 14 Jun 2009 21:29:09 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.463
X-Spam-Level: 
X-Spam-Status: No, score=-1.463 tagged_above=-999 required=5 tests=[AWL=-0.505, BAYES_05=-1.11, SARE_SUB_ENC_UTF8=0.152]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ZQvrHUGgt4RE for <ldapext@core3.amsl.com>; Sun, 14 Jun 2009 21:29:09 -0700 (PDT)
Received: from idcmail-mo1so.shaw.ca (idcmail-mo1so.shaw.ca [24.71.223.10]) by core3.amsl.com (Postfix) with ESMTP id E21763A6A00 for <ldapext@ietf.org>; Sun, 14 Jun 2009 21:29:08 -0700 (PDT)
Received: from pd2ml2so-ssvc.prod.shaw.ca ([10.0.141.134]) by pd4mo1so-svcs.prod.shaw.ca with ESMTP; 14 Jun 2009 22:28:14 -0600
X-Cloudmark-SP-Filtered: true
X-Cloudmark-SP-Result: v=1.0 c=0 a=J6b0bBEkAAAA:8 a=Wi3oSif3GHoQP8-nLj4A:9 a=5fEN0uZE_ghM0kCvpNQTbH0jf6UA:4 a=WPV3hXSGBlIA:10 a=Cguc0gE-TcEA:10
Received: from unknown (HELO home.zioup.com) ([68.147.37.46]) by pd2ml2so-dmz.prod.shaw.ca with ESMTP; 14 Jun 2009 22:28:14 -0600
Received: from [192.168.2.202] (aristotle.zioup.net [192.168.2.202]) by home.zioup.com (Postfix) with ESMTP id 7825C33000 for <ldapext@ietf.org>; Sun, 14 Jun 2009 22:28:14 -0600 (MDT)
Message-ID: <4A35CDDE.8000604@zioup.com>
Date: Sun, 14 Jun 2009 22:28:14 -0600
From: Yves Dorfsman <yves@zioup.com>
User-Agent: Thunderbird 2.0.0.21 (X11/20090409)
MIME-Version: 1.0
To: ldapext@ietf.org
References: <49C497F9.7010200@zioup.com>	<CD3905D4-2A25-4C56-8187-3CE10D46C929@isode.com>	<49C870C6.4010803@zioup.com>	<E94B7389-9A6D-4CB6-BB2C-649CCD3FD15B@Isode.com>	<49CB192E.5050105@zioup.com>	<49CB211C.6070108@eb2bcom.com>	<49CB87FE.1050809@zioup.com>	<49CC01DE.6040506@eb2bcom.com>	<4A24557D.7030006@zioup.com>	<4A26A05D.8040105@zioup.com>	<245BF18B-2066-4E36-9502-16F4A3140D9E@Isode.com>	<4A309775.3080406@zioup.com> <4A311ED1.1030202@stroeder.com> <4A31D27B.3090208@zioup.com> <4A325A40.2050802@stroeder.com>
In-Reply-To: <4A325A40.2050802@stroeder.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
Subject: Re: [ldapext] UTF-8 full support in LDIF / LDIF v2
X-BeenThere: ldapext@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: LDAP Extension Working Group <ldapext.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/ldapext>, <mailto:ldapext-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/ldapext>
List-Post: <mailto:ldapext@ietf.org>
List-Help: <mailto:ldapext-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/ldapext>, <mailto:ldapext-request@ietf.org?subject=subscribe>
X-List-Received-Date: Mon, 15 Jun 2009 04:29:10 -0000

Michael Ströder wrote:
> Yves Dorfsman wrote:
>> Looking back (way back) at this thread, I am not the only one seeing
>> value here:
>>
> 
> Hmm, I agreed on SASL normalization not being an issue.
> But you skipped my strong disagreement about your LDIF diffing use-case.
> 

I'm sorry, I did not try to put any word in your mouth, all I meant was that 
there is more than just me who see value in having non base64 encoded UTF-8.

I know you disagree with me about diffing, but was under the impression you 
saw value somewhere else.

Sorry for the confusion.

-- 
Yves.
http://www.sollers.ca/


From yves@zioup.com  Sun Jun 14 21:46:45 2009
Return-Path: <yves@zioup.com>
X-Original-To: ldapext@core3.amsl.com
Delivered-To: ldapext@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 6B5D33A6AF2 for <ldapext@core3.amsl.com>; Sun, 14 Jun 2009 21:46:45 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.823
X-Spam-Level: 
X-Spam-Status: No, score=-1.823 tagged_above=-999 required=5 tests=[AWL=0.024,  BAYES_00=-2.599, J_CHICKENPOX_43=0.6, SARE_SUB_ENC_UTF8=0.152]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 1l6L7OUG4jGZ for <ldapext@core3.amsl.com>; Sun, 14 Jun 2009 21:46:44 -0700 (PDT)
Received: from idcmail-mo1so.shaw.ca (idcmail-mo1so.shaw.ca [24.71.223.10]) by core3.amsl.com (Postfix) with ESMTP id 4ACBF3A6AC1 for <ldapext@ietf.org>; Sun, 14 Jun 2009 21:46:44 -0700 (PDT)
Received: from pd2ml2so-ssvc.prod.shaw.ca ([10.0.141.134]) by pd3mo1so-svcs.prod.shaw.ca with ESMTP; 14 Jun 2009 22:46:54 -0600
X-Cloudmark-SP-Filtered: true
X-Cloudmark-SP-Result: v=1.0 c=0 a=J6b0bBEkAAAA:8 a=ers-uFzMC9JexpLFdNIA:9 a=mQk49vgzOBfzndDBW-cA:7 a=ochxnFJGnbiLatfgd4CSM1WnkNsA:4 a=WPV3hXSGBlIA:10 a=Cguc0gE-TcEA:10
Received: from unknown (HELO home.zioup.com) ([68.147.37.46]) by pd2ml2so-dmz.prod.shaw.ca with ESMTP; 14 Jun 2009 22:46:54 -0600
Received: from [192.168.2.202] (aristotle.zioup.net [192.168.2.202]) by home.zioup.com (Postfix) with ESMTP id DC34933000 for <ldapext@ietf.org>; Sun, 14 Jun 2009 22:46:53 -0600 (MDT)
Message-ID: <4A35D23D.5040307@zioup.com>
Date: Sun, 14 Jun 2009 22:46:53 -0600
From: Yves Dorfsman <yves@zioup.com>
User-Agent: Thunderbird 2.0.0.21 (X11/20090409)
MIME-Version: 1.0
To: ldapext@ietf.org
References: <49C497F9.7010200@zioup.com>	<CD3905D4-2A25-4C56-8187-3CE10D46C929@isode.com>	<49C870C6.4010803@zioup.com>	<E94B7389-9A6D-4CB6-BB2C-649CCD3FD15B@Isode.com>	<49CB192E.5050105@zioup.com>	<49CB211C.6070108@eb2bcom.com>	<49CB87FE.1050809@zioup.com>	<49CC01DE.6040506@eb2bcom.com>	<4A24557D.7030006@zioup.com> <4A26A05D.8040105@zioup.com>	<245BF18B-2066-4E36-9502-16F4A3140D9E@Isode.com> <4A309775.3080406@zioup.com> <4A311ED1.1030202@stroeder.com> <4A31D27B.3090208@zioup.com> <35B2A165-CE5D-4650-AADE-CC233F71470E@Isode.com>
In-Reply-To: <35B2A165-CE5D-4650-AADE-CC233F71470E@Isode.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Subject: Re: [ldapext] UTF-8 full support in LDIF / LDIF v2
X-BeenThere: ldapext@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: LDAP Extension Working Group <ldapext.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/ldapext>, <mailto:ldapext-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/ldapext>
List-Post: <mailto:ldapext@ietf.org>
List-Help: <mailto:ldapext-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/ldapext>, <mailto:ldapext-request@ietf.org?subject=subscribe>
X-List-Received-Date: Mon, 15 Jun 2009 04:46:45 -0000

Kurt Zeilenga wrote:
> 
> I think we need clear problem statements, a proposal for addressing each 
> problem, a summary of why the proposal does address the problem and a 
> statement of what (known) problems the proposal might introduce.
> 
> You have noted that the "diffing problem".  But here it's not clear 
> whether a) you are wish to determine how two LDIF files differ, b) you 
> wish to determine if two LDIF files represent the same LDAP requests, or 
> c) you wish to determine how directory information represented in the 
> LDIF represented LDAP requests differ and, if so, how.
> 
> For a), one can use file comparison tools to determine how two LDIF 
> files differ.

Yes, but because it displays unreadable characters, it makes it slightly 
more complicated. The better case (than simply diffing) I have given in the 
past is:

-the directory is broken
-you export to LDIF
-compare this LDIF with a previous one from when the directory was working.

I personally find that in such a case, being able to read the values makes 
it simpler and faster.


Other case: People have mentioned scripts that build LDIF file from other 
source, and have mentioned that encoding the values in base64 is an overhead 
they could do without.

> 
> Simply having a UTF-8 value encoding option doesn't generally solve any 
> diffing problem.  In cases b and c, the LDIF encoding of the value 
> doesn't matter.  In case a), you'd be introducing another way for LDIF 
> files which represented the same LDAP requests to differ.

If you always do your export with the same tool, with the same options, then 
this shouldn't be an issue. A narrow case I admit, but this is one specific 
case I was thinking about.


> It may be you are thinking that a human would be better able to visually 
> detect certain kinds of differences.  However, this assumes that 
> removing the ASCII restriction would produce a readily display Unicode 
> text.

On a modern OS setup properly, Unicode text is displayed properly (my 
experience is with UTF-8 on Linux and solaris here).


>  That, I believe, is a bad assumption.  For instance, say a user 
> diff(1) to LDIF files and get:
> 
> % diff -u ?.ldif
> --- 1.ldif    2009-06-11 22:08:21.000000000 -0700
> +++ 2.ldif    2009-06-11 22:08:47.000000000 -0700
> @@ -1 +1 @@
> -a: f??
> +a:f??
> 
> where ? represents character not displayable on the user's screen.  The 
> user might assume the values here are same when they aren't.
> 
> This, I hope, illustrates why general file diff'ing tools, like diff(1), 
> are suitable only for case a but not b.

Ok, but a) is still a valid case.

-- 
Yves.
http://www.sollers.ca/


From hyc@highlandsun.com  Mon Jun 15 11:00:38 2009
Return-Path: <hyc@highlandsun.com>
X-Original-To: ldapext@core3.amsl.com
Delivered-To: ldapext@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id B76BF3A67B3 for <ldapext@core3.amsl.com>; Mon, 15 Jun 2009 11:00:38 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -0.923
X-Spam-Level: 
X-Spam-Status: No, score=-0.923 tagged_above=-999 required=5 tests=[AWL=-0.924, BAYES_50=0.001]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id N3yj5f0SfESi for <ldapext@core3.amsl.com>; Mon, 15 Jun 2009 11:00:37 -0700 (PDT)
Received: from mail.highlandsun.com (mail.highlandsun.com [70.87.222.79]) by core3.amsl.com (Postfix) with ESMTP id 1C5493A67A1 for <ldapext@ietf.org>; Mon, 15 Jun 2009 11:00:37 -0700 (PDT)
Received: from [127.0.0.1] (localhost [127.0.0.1]) by mail.highlandsun.com (Postfix) with ESMTP id BDE2610F96 for <ldapext@ietf.org>; Mon, 15 Jun 2009 14:00:38 -0400 (EDT)
Message-ID: <4A368C40.4020605@highlandsun.com>
Date: Mon, 15 Jun 2009 11:00:32 -0700
From: Howard Chu <hyc@highlandsun.com>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; rv:1.9.1b5pre) Gecko/20090611 SeaMonkey/2.0a1pre Firefox/3.0.3
MIME-Version: 1.0
To: Ldapext <ldapext@ietf.org>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Subject: [ldapext] LDAPCon 2009 Call for Papers
X-BeenThere: ldapext@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: LDAP Extension Working Group <ldapext.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/ldapext>, <mailto:ldapext-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/ldapext>
List-Post: <mailto:ldapext@ietf.org>
List-Help: <mailto:ldapext-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/ldapext>, <mailto:ldapext-request@ietf.org?subject=subscribe>
X-List-Received-Date: Mon, 15 Jun 2009 18:00:38 -0000

Apologies if this gets double-posted...

The 2nd Edition of the International Conference on LDAP (LDAPCon 2009[1]) will 
be held on September 20-21, 2009 in Portland, Oregon, USA. The event is being 
co-hosted by LinuxCon[2] (same place and overlapping time). The 1st 
International Conference on LDAP[3] was held in September 2007 in Germany.
(Some pictures from 1st LDAPCon [4])

A Call For Papers[5] has been raised and the Program Committee asks you to 
submit abstracts by July 1st. So if you're involved with LDAP in interesting 
projects and you want to share your experiences, please check the Call For 
Papers and submit a proposal.

1: http://www.symas.com/ldapcon2009/
2: http://events.linuxfoundation.org/events/linuxcon
3: http://www.guug.de/veranstaltungen/ldapcon2007/index.html
4: http://www.flickr.com/photos/ludovic_p/sets/72157601937159198/detail/
5: http://www.symas.com/ldapcon2009/call-for-papers.shtml

-- 
   -- Howard Chu
   CTO, Symas Corp.           http://www.symas.com
   Director, Highland Sun     http://highlandsun.com/hyc/
   Chief Architect, OpenLDAP  http://www.openldap.org/project/

From michael@stroeder.com  Tue Jun 16 03:19:18 2009
Return-Path: <michael@stroeder.com>
X-Original-To: ldapext@core3.amsl.com
Delivered-To: ldapext@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 2613728C172 for <ldapext@core3.amsl.com>; Tue, 16 Jun 2009 03:19:18 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.147
X-Spam-Level: 
X-Spam-Status: No, score=-2.147 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, MIME_8BIT_HEADER=0.3, SARE_SUB_ENC_UTF8=0.152]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id O3GzZHBd5MKJ for <ldapext@core3.amsl.com>; Tue, 16 Jun 2009 03:19:17 -0700 (PDT)
Received: from srv1.stroeder.com (srv1.stroeder.com [213.240.180.113]) by core3.amsl.com (Postfix) with ESMTP id 3C69C3A6B71 for <ldapext@ietf.org>; Tue, 16 Jun 2009 03:19:17 -0700 (PDT)
Received: from localhost (localhost [127.0.0.1]) by srv1.stroeder.com (Postfix) with ESMTP id D96A64E06F; Tue, 16 Jun 2009 12:18:59 +0200 (CEST)
X-Virus-Scanned: amavisd-new at stroeder.com
Received: from srv1.stroeder.com ([127.0.0.1]) by localhost (srv1.stroeder.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id YWAOy4SBsXAU; Tue, 16 Jun 2009 12:18:56 +0200 (CEST)
Received: from [10.1.0.2] (unknown [10.1.0.2]) by srv1.stroeder.com (Postfix) with ESMTP id 1A0F64E06D; Tue, 16 Jun 2009 12:18:55 +0200 (CEST)
Message-ID: <4A37719E.3010006@stroeder.com>
Date: Tue, 16 Jun 2009 12:19:10 +0200
From: =?UTF-8?B?TWljaGFlbCBTdHLDtmRlcg==?= <michael@stroeder.com>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.22) Gecko/20090606 SeaMonkey/1.1.17
MIME-Version: 1.0
To: Yves Dorfsman <yves@zioup.com>
References: <49C497F9.7010200@zioup.com>	<CD3905D4-2A25-4C56-8187-3CE10D46C929@isode.com>	<49C870C6.4010803@zioup.com>	<E94B7389-9A6D-4CB6-BB2C-649CCD3FD15B@Isode.com>	<49CB192E.5050105@zioup.com>	<49CB211C.6070108@eb2bcom.com>	<49CB87FE.1050809@zioup.com>	<49CC01DE.6040506@eb2bcom.com>	<4A24557D.7030006@zioup.com>	<4A26A05D.8040105@zioup.com>	<245BF18B-2066-4E36-9502-16F4A3140D9E@Isode.com>	<4A309775.3080406@zioup.com>	<4A311ED1.1030202@stroeder.com> <4A31D27B.3090208@zioup.com>	<4A325A40.2050802@stroeder.com> <4A35CDDE.8000604@zioup.com>
In-Reply-To: <4A35CDDE.8000604@zioup.com>
X-Enigmail-Version: 0.95.7
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Cc: ldapext@ietf.org
Subject: Re: [ldapext] UTF-8 full support in LDIF / LDIF v2
X-BeenThere: ldapext@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: LDAP Extension Working Group <ldapext.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/ldapext>, <mailto:ldapext-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/ldapext>
List-Post: <mailto:ldapext@ietf.org>
List-Help: <mailto:ldapext-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/ldapext>, <mailto:ldapext-request@ietf.org?subject=subscribe>
X-List-Received-Date: Tue, 16 Jun 2009 10:19:18 -0000

Yves Dorfsman wrote:
> Michael Ströder wrote:
>> Yves Dorfsman wrote:
>>> Looking back (way back) at this thread, I am not the only one seeing
>>> value here:
>>
>> Hmm, I agreed on SASL normalization not being an issue.
>> But you skipped my strong disagreement about your LDIF diffing use-case.
> 
> I'm sorry, I did not try to put any word in your mouth, all I meant was
> that there is more than just me who see value in having non base64
> encoded UTF-8.
> 
> I know you disagree with me about diffing,

Yes, because that's better implemented by decent LDIF-aware tools.

> but was under the impression
> you saw value somewhere else.

I see some value. As I said my web2ldap accepts single-line UTF-8 input
in an LDIF editor multi-line input field but does not display UTF-8.

But if we run into more complicated issues like multi-line UTF-8 data
I'm against opening a can of worms with here document scheme or similar
complicated things.

Ciao, Michael.

From Kurt.Zeilenga@Isode.com  Tue Jun 16 06:45:56 2009
Return-Path: <Kurt.Zeilenga@Isode.com>
X-Original-To: ldapext@core3.amsl.com
Delivered-To: ldapext@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id C9CF53A6AA4 for <ldapext@core3.amsl.com>; Tue, 16 Jun 2009 06:45:56 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.168
X-Spam-Level: 
X-Spam-Status: No, score=-2.168 tagged_above=-999 required=5 tests=[AWL=-0.321, BAYES_00=-2.599, J_CHICKENPOX_43=0.6, SARE_SUB_ENC_UTF8=0.152]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id fwQTxtpRY53s for <ldapext@core3.amsl.com>; Tue, 16 Jun 2009 06:45:55 -0700 (PDT)
Received: from boole.openldap.org (boole.openldap.org [IPv6:2001:4f8:3:ba:219:21ff:fe2f:68d5]) by core3.amsl.com (Postfix) with ESMTP id 892F43A699E for <ldapext@ietf.org>; Tue, 16 Jun 2009 06:45:55 -0700 (PDT)
Received: from [192.168.1.101] (75-141-233-128.dhcp.nv.charter.com [75.141.233.128] (may be forged)) (authenticated bits=0) by boole.openldap.org (8.14.3/8.14.3) with ESMTP id n5GDk1n1029998; Tue, 16 Jun 2009 13:46:02 GMT (envelope-from Kurt.Zeilenga@Isode.com)
Message-Id: <D437E784-4198-4037-A4EA-0300439C3D2C@Isode.com>
From: Kurt Zeilenga <Kurt.Zeilenga@Isode.com>
To: Yves Dorfsman <yves@zioup.com>
In-Reply-To: <4A35D23D.5040307@zioup.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v935.3)
Date: Tue, 16 Jun 2009 06:46:01 -0700
References: <49C497F9.7010200@zioup.com>	<CD3905D4-2A25-4C56-8187-3CE10D46C929@isode.com>	<49C870C6.4010803@zioup.com>	<E94B7389-9A6D-4CB6-BB2C-649CCD3FD15B@Isode.com>	<49CB192E.5050105@zioup.com>	<49CB211C.6070108@eb2bcom.com>	<49CB87FE.1050809@zioup.com>	<49CC01DE.6040506@eb2bcom.com>	<4A24557D.7030006@zioup.com> <4A26A05D.8040105@zioup.com>	<245BF18B-2066-4E36-9502-16F4A3140D9E@Isode.com> <4A309775.3080406@zioup.com> <4A311ED1.1030202@stroeder.com> <4A31D27B.3090208@zioup.com> <35B2A165-CE5D-4650-AADE-CC233F71470E@Isode.com> <4A35D23D.5040307@zioup.com>
X-Mailer: Apple Mail (2.935.3)
Cc: ldapext@ietf.org
Subject: Re: [ldapext] UTF-8 full support in LDIF / LDIF v2
X-BeenThere: ldapext@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: LDAP Extension Working Group <ldapext.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/ldapext>, <mailto:ldapext-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/ldapext>
List-Post: <mailto:ldapext@ietf.org>
List-Help: <mailto:ldapext-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/ldapext>, <mailto:ldapext-request@ietf.org?subject=subscribe>
X-List-Received-Date: Tue, 16 Jun 2009 13:45:56 -0000

On Jun 14, 2009, at 9:46 PM, Yves Dorfsman wrote:

>
> Kurt Zeilenga wrote:
>> I think we need clear problem statements, a proposal for addressing  
>> each problem, a summary of why the proposal does address the  
>> problem and a statement of what (known) problems the proposal might  
>> introduce.
>> You have noted that the "diffing problem".  But here it's not clear  
>> whether a) you are wish to determine how two LDIF files differ, b)  
>> you wish to determine if two LDIF files represent the same LDAP  
>> requests, or c) you wish to determine how directory information  
>> represented in the LDIF represented LDAP requests differ and, if  
>> so, how.
>> For a), one can use file comparison tools to determine how two LDIF  
>> files differ.
>
> Yes, but because it displays unreadable characters, it makes it  
> slightly more complicated. The better case (than simply diffing) I  
> have given in the past is:
>
> -the directory is broken
> -you export to LDIF
> -compare this LDIF with a previous one from when the directory was  
> working.

You don't need UTF-8 for this.  A simple text diff tool will tell you  
that the base64 differs.
>
> I personally find that in such a case, being able to read the values  
> makes it simpler and faster.

But now you assume you'll be able to read them.  This is a bad  
assumption.  A simple diff tool might show two DIFFERENT values the  
same way, leading the human to believe there is no difference when  
there is a significant difference.  And then there the issue that a  
UTF-8 encoded Unicode file is not well-formed text, and trying to  
treat it as text will be quite problematic.  (see below)

> Other case: People have mentioned scripts that build LDIF file from  
> other source, and have mentioned that encoding the values in base64  
> is an overhead they could do without.

While base64 data is an additional step, it's an additional step that  
well supported today.  If we lift the ASCII restriction now, we'll  
have some implementations that do support it and some that don't, and  
that will cause interop problems.   I cannot support inducing such  
interop problems without a strong justification.

>
>> It may be you are thinking that a human would be better able to  
>> visually detect certain kinds of differences.  However, this  
>> assumes that removing the ASCII restriction would produce a readily  
>> display Unicode text.
>
> On a modern OS setup properly, Unicode text is displayed properly  
> (my experience is with UTF-8 on Linux and solaris here).

The key phrase here is "Unicode text".  And most such display tools  
not only require "well-formed" text, but often cannot display all  
"well-formed" text.  But removing the ASCII restriction does not make  
a LDIF file "Unicode text".  It makes it a series of Unicode code  
points and hence display of it as text will be quite problematic.  And  
even it's displayable, you have the problem that two values might  
display in the same way, making visual diff'ing problematic.

-- Kurt

From yves@zioup.com  Tue Jun 16 07:04:16 2009
Return-Path: <yves@zioup.com>
X-Original-To: ldapext@core3.amsl.com
Delivered-To: ldapext@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 5969B3A69EE for <ldapext@core3.amsl.com>; Tue, 16 Jun 2009 07:04:16 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.126
X-Spam-Level: 
X-Spam-Status: No, score=-2.126 tagged_above=-999 required=5 tests=[AWL=0.321,  BAYES_00=-2.599, SARE_SUB_ENC_UTF8=0.152]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 6fwALmGmTj4F for <ldapext@core3.amsl.com>; Tue, 16 Jun 2009 07:04:15 -0700 (PDT)
Received: from idcmail-mo1so.shaw.ca (idcmail-mo1so.shaw.ca [24.71.223.10]) by core3.amsl.com (Postfix) with ESMTP id 3DDD93A6845 for <ldapext@ietf.org>; Tue, 16 Jun 2009 07:04:14 -0700 (PDT)
Received: from pd2ml2so-ssvc.prod.shaw.ca ([10.0.141.134]) by pd2mo1so-svcs.prod.shaw.ca with ESMTP; 16 Jun 2009 08:02:02 -0600
X-Cloudmark-SP-Filtered: true
X-Cloudmark-SP-Result: v=1.0 c=0 a=J6b0bBEkAAAA:8 a=_58EkhM2xYawsOKcAXwA:9 a=t68x5vRtSjAGiyb9c94A:7 a=MMqGuKnw9QTCFoAXeEsDbWaxSioA:4 a=WPV3hXSGBlIA:10
Received: from unknown (HELO home.zioup.com) ([68.147.37.46]) by pd2ml2so-dmz.prod.shaw.ca with ESMTP; 16 Jun 2009 08:02:02 -0600
Received: from [192.168.2.202] (aristotle.zioup.net [192.168.2.202]) by home.zioup.com (Postfix) with ESMTP id 3D7BA4A1D2 for <ldapext@ietf.org>; Tue, 16 Jun 2009 08:02:02 -0600 (MDT)
Message-ID: <4A37A5D9.4040901@zioup.com>
Date: Tue, 16 Jun 2009 08:02:01 -0600
From: Yves Dorfsman <yves@zioup.com>
User-Agent: Thunderbird 2.0.0.21 (X11/20090409)
MIME-Version: 1.0
To: ldapext@ietf.org
References: <49C497F9.7010200@zioup.com>	<CD3905D4-2A25-4C56-8187-3CE10D46C929@isode.com>	<49C870C6.4010803@zioup.com>	<E94B7389-9A6D-4CB6-BB2C-649CCD3FD15B@Isode.com>	<49CB192E.5050105@zioup.com>	<49CB211C.6070108@eb2bcom.com>	<49CB87FE.1050809@zioup.com>	<49CC01DE.6040506@eb2bcom.com>	<4A24557D.7030006@zioup.com>	<4A26A05D.8040105@zioup.com>	<245BF18B-2066-4E36-9502-16F4A3140D9E@Isode.com>	<4A309775.3080406@zioup.com>	<4A311ED1.1030202@stroeder.com> <4A31D27B.3090208@zioup.com>	<4A325A40.2050802@stroeder.com> <4A35CDDE.8000604@zioup.com> <4A37719E.3010006@stroeder.com>
In-Reply-To: <4A37719E.3010006@stroeder.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
Subject: Re: [ldapext] UTF-8 full support in LDIF / LDIF v2
X-BeenThere: ldapext@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: LDAP Extension Working Group <ldapext.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/ldapext>, <mailto:ldapext-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/ldapext>
List-Post: <mailto:ldapext@ietf.org>
List-Help: <mailto:ldapext-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/ldapext>, <mailto:ldapext-request@ietf.org?subject=subscribe>
X-List-Received-Date: Tue, 16 Jun 2009 14:04:16 -0000

Michael Ströder wrote:
>> but was under the impression
>> you saw value somewhere else.
> 
> I see some value. As I said my web2ldap accepts single-line UTF-8 input
> in an LDIF editor multi-line input field but does not display UTF-8.
> 
> But if we run into more complicated issues like multi-line UTF-8 data
> I'm against opening a can of worms with here document scheme or similar
> complicated things.

So being able to read/write directly in UTF-8 in the LDIF file will 
eliminate one step in the process ? Can you give more details ?


Listing the case we have so far (please argument against if you think a case 
is not valid):

-comparing at the file level by humans, to fix an issue for example ; this 
imply that the user is using a display properly setup for UTF-8 (Yves Dorfsman).

-simplifying scripts to import/export data form/to other format to/from LDIF 
(Steven Legg. Steven).

-simplifying online editors (Michael Stöder).




Steven:
Can you confirm this is what you were saying, otherwise reword ?


Kurt:
My case was the case a) in you email. In my last email I argued that 1) that 
case has value
2) your argument against it, not being able to display UTF-8 properly, is 
solved in modern environment setup for UTF-8


Michael:
Can you give us more explanation of what the problem is in the case of your 
editor. Would it simplify programing the editor ? Is it impossible to allow 
UTF-8 in your editor today, and this change would allow it ?


-- 
Yves.
http://www.sollers.ca/


From Kurt.Zeilenga@Isode.com  Tue Jun 16 08:35:25 2009
Return-Path: <Kurt.Zeilenga@Isode.com>
X-Original-To: ldapext@core3.amsl.com
Delivered-To: ldapext@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id B00F53A6B79 for <ldapext@core3.amsl.com>; Tue, 16 Jun 2009 08:35:25 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.138
X-Spam-Level: 
X-Spam-Status: No, score=-2.138 tagged_above=-999 required=5 tests=[AWL=-0.291, BAYES_00=-2.599, J_CHICKENPOX_25=0.6, SARE_SUB_ENC_UTF8=0.152]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id sNlqIrV+I2bT for <ldapext@core3.amsl.com>; Tue, 16 Jun 2009 08:35:24 -0700 (PDT)
Received: from boole.openldap.org (boole.openldap.org [IPv6:2001:4f8:3:ba:219:21ff:fe2f:68d5]) by core3.amsl.com (Postfix) with ESMTP id AB84A3A6969 for <ldapext@ietf.org>; Tue, 16 Jun 2009 08:35:24 -0700 (PDT)
Received: from [192.168.1.101] (75-141-233-128.dhcp.nv.charter.com [75.141.233.128] (may be forged)) (authenticated bits=0) by boole.openldap.org (8.14.3/8.14.3) with ESMTP id n5GFZTkx035859; Tue, 16 Jun 2009 15:35:29 GMT (envelope-from Kurt.Zeilenga@Isode.com)
Message-Id: <D5DE1CCA-5F00-46D1-8E37-56A0498B13B0@Isode.com>
From: Kurt Zeilenga <Kurt.Zeilenga@Isode.com>
To: Yves Dorfsman <yves@zioup.com>
In-Reply-To: <4A37A5D9.4040901@zioup.com>
Content-Type: multipart/mixed; boundary=Apple-Mail-7-571356693
Mime-Version: 1.0 (Apple Message framework v935.3)
Date: Tue, 16 Jun 2009 08:35:28 -0700
References: <49C497F9.7010200@zioup.com>	<CD3905D4-2A25-4C56-8187-3CE10D46C929@isode.com>	<49C870C6.4010803@zioup.com>	<E94B7389-9A6D-4CB6-BB2C-649CCD3FD15B@Isode.com>	<49CB192E.5050105@zioup.com>	<49CB211C.6070108@eb2bcom.com>	<49CB87FE.1050809@zioup.com>	<49CC01DE.6040506@eb2bcom.com>	<4A24557D.7030006@zioup.com>	<4A26A05D.8040105@zioup.com>	<245BF18B-2066-4E36-9502-16F4A3140D9E@Isode.com>	<4A309775.3080406@zioup.com>	<4A311ED1.1030202@stroeder.com> <4A31D27B.3090208@zioup.com>	<4A325A40.2050802@stroeder.com> <4A35CDDE.8000604@zioup.com> <4A37719E.3010006@stroeder.com> <4A37A5D9.4040901@zioup.com>
X-Mailer: Apple Mail (2.935.3)
Cc: ldapext@ietf.org
Subject: Re: [ldapext] UTF-8 full support in LDIF / LDIF v2
X-BeenThere: ldapext@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: LDAP Extension Working Group <ldapext.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/ldapext>, <mailto:ldapext-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/ldapext>
List-Post: <mailto:ldapext@ietf.org>
List-Help: <mailto:ldapext-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/ldapext>, <mailto:ldapext-request@ietf.org?subject=subscribe>
X-List-Received-Date: Tue, 16 Jun 2009 15:35:25 -0000

--Apple-Mail-7-571356693
Content-Type: text/plain;
	charset=ISO-8859-1;
	format=flowed;
	delsp=yes
Content-Transfer-Encoding: quoted-printable


On Jun 16, 2009, at 7:02 AM, Yves Dorfsman wrote:

> Michael Str=F6der wrote:
>>> but was under the impression
>>> you saw value somewhere else.
>> I see some value. As I said my web2ldap accepts single-line UTF-8 =20
>> input
>> in an LDIF editor multi-line input field but does not display UTF-8.
>> But if we run into more complicated issues like multi-line UTF-8 data
>> I'm against opening a can of worms with here document scheme or =20
>> similar
>> complicated things.
>
> So being able to read/write directly in UTF-8 in the LDIF file will =20=

> eliminate one step in the process ? Can you give more details ?
>
>
> Listing the case we have so far (please argument against if you =20
> think a case is not valid):
>
> -comparing at the file level by humans, to fix an issue for example ;

Again, two UTF-8 strings may display in the same, meaning humans will =20=

NOT be able to visually detect the difference and hence won't be able =20=

to fix the difference.

> this imply that the user is using a display properly setup for UTF-8 =20=

> (Yves Dorfsman).

Again, Unicode display programs are typically designed to display well-=20=

formed text, not arbitrary sequences of Unicode code points.


> -simplifying scripts to import/export data form/to other format to/=20
> from LDIF (Steven Legg. Steven).

As I noted before, Steve's suggestion is problematic in that it =20
doesn't preserve Unicode code points in exchange of LDAP values.

> -simplifying online editors (Michael St=F6der).

I seriously doubt that removing the ASCII restriction actually =20
simplifies anything.  Removing the restriction adds another encoding =20
option, which adds complexity and adds interop problems.

>
> Steven:
> Can you confirm this is what you were saying, otherwise reword ?
>
>
> Kurt:
> My case was the case a) in you email. In my last email I argued that =20=

> 1) that case has value
> 2) your argument against it, not being able to display UTF-8 =20
> properly, is solved in modern environment setup for UTF-8

I think your assumptions about "modern environment setup for UTF-8" =20
are flawed.  Good UTF-8 display programs (that display UTF-8 as text) =20=

generally expect the UTF-8 to be well formed text.   If the input to =20
these programs is not well formed text, the display will not be well =20
formed (though it may not be obvious to the viewer that it's not well-=20=

formed text).  And there are lots of "bad" UTF-8 display programs in =20
wide use, ones which cannot handle even all well formed text (like =20
bidirectional text).  And most every display program suffers from =20
incomplete fonts.  And then there are private use code points, and =20
various other things that even the best UTF-8 display programs have =20
problems with.

Here's a contrived file that illustrates just some of the display =20
problems removing the ASCII restriction will lead to.  (apologies if =20
my MUA associates the wrong MIME information to this, but that just =20
illustrates yet another problem folks will face (wrong MIME =20
information) when exchange such files.)  BTW, none of my well setup =20
Unicode "text" display programs handled this file well... because it's =20=

simply not well-formed text.


--Apple-Mail-7-571356693
Content-Disposition: attachment;
	filename=utf8-ldif.txt
Content-Type: text/plain;
	x-unix-mode=0664;
	name="utf8-ldif.txt"
Content-Transfer-Encoding: quoted-printable

version: 2

dn: cn=3Dfunky
bom:=EF=BB=BF
smiley-face:=E2=98=BA
# only SPACE is special
no-break-space: =20
zero-width-space:=E2=80=8B
word-joiner:=E2=81=A0
ideographic-space:=E3=80=80
zero-width-no-break-space:=EF=BB=BF
# these hyphen differ but may look the same
hyphen-minus:-
hyphen:=E2=80=90
non-breaking-hyphen:=E2=80=91
figure-dash:=E2=80=92
en-dash:=E2=80=93
minus-sign:=E2=88=92
roman-uncia-sign:=F0=90=86=91
# these differ but may look the same
o-diaeresis:=C3=B6
o-diaeresis-decomposed:o=CC=88
# combining character
diaeresis:=CC=88

--Apple-Mail-7-571356693
Content-Type: text/plain;
	charset=US-ASCII;
	format=flowed
Content-Transfer-Encoding: 7bit




-- Kurt
--Apple-Mail-7-571356693--

From yves@zioup.com  Tue Jun 16 08:42:46 2009
Return-Path: <yves@zioup.com>
X-Original-To: ldapext@core3.amsl.com
Delivered-To: ldapext@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id E73903A68A0 for <ldapext@core3.amsl.com>; Tue, 16 Jun 2009 08:42:46 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.866
X-Spam-Level: 
X-Spam-Status: No, score=-1.866 tagged_above=-999 required=5 tests=[AWL=-0.019, BAYES_00=-2.599, J_CHICKENPOX_43=0.6, SARE_SUB_ENC_UTF8=0.152]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id o9nKpO7i+1qF for <ldapext@core3.amsl.com>; Tue, 16 Jun 2009 08:42:45 -0700 (PDT)
Received: from idcmail-mo1so.shaw.ca (idcmail-mo1so.shaw.ca [24.71.223.10]) by core3.amsl.com (Postfix) with ESMTP id 697933A6804 for <ldapext@ietf.org>; Tue, 16 Jun 2009 08:42:45 -0700 (PDT)
Received: from pd3ml1so-ssvc.prod.shaw.ca ([10.0.141.140]) by pd3mo1so-svcs.prod.shaw.ca with ESMTP; 16 Jun 2009 09:40:28 -0600
X-Cloudmark-SP-Filtered: true
X-Cloudmark-SP-Result: v=1.0 c=0 a=8pif782wAAAA:8 a=J6b0bBEkAAAA:8 a=LPnSuM_rbfk-z26NUwMA:9 a=YGNdDo1MwVjEsCSE8UsA:7 a=K2bN6H9ytkVzrCoDf01oOm0_cPEA:4 a=WPV3hXSGBlIA:10
Received: from unknown (HELO home.zioup.com) ([68.147.37.46]) by pd3ml1so-dmz.prod.shaw.ca with ESMTP; 16 Jun 2009 09:40:28 -0600
Received: from [192.168.2.202] (aristotle.zioup.net [192.168.2.202]) by home.zioup.com (Postfix) with ESMTP id 64E8AC120 for <ldapext@ietf.org>; Tue, 16 Jun 2009 09:40:28 -0600 (MDT)
Message-ID: <4A37BCEB.5040103@zioup.com>
Date: Tue, 16 Jun 2009 09:40:27 -0600
From: Yves Dorfsman <yves@zioup.com>
User-Agent: Thunderbird 2.0.0.21 (X11/20090409)
MIME-Version: 1.0
To: ldapext@ietf.org
References: <49C497F9.7010200@zioup.com>	<CD3905D4-2A25-4C56-8187-3CE10D46C929@isode.com>	<49C870C6.4010803@zioup.com>	<E94B7389-9A6D-4CB6-BB2C-649CCD3FD15B@Isode.com>	<49CB192E.5050105@zioup.com>	<49CB211C.6070108@eb2bcom.com>	<49CB87FE.1050809@zioup.com>	<49CC01DE.6040506@eb2bcom.com>	<4A24557D.7030006@zioup.com> <4A26A05D.8040105@zioup.com>	<245BF18B-2066-4E36-9502-16F4A3140D9E@Isode.com> <4A309775.3080406@zioup.com> <4A311ED1.1030202@stroeder.com> <4A31D27B.3090208@zioup.com> <35B2A165-CE5D-4650-AADE-CC233F71470E@Isode.com> <4A35D23D.5040307@zioup.com> <D437E784-4198-4037-A4EA-0300439C3D2C@Isode.com>
In-Reply-To: <D437E784-4198-4037-A4EA-0300439C3D2C@Isode.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
Subject: Re: [ldapext] UTF-8 full support in LDIF / LDIF v2
X-BeenThere: ldapext@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: LDAP Extension Working Group <ldapext.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/ldapext>, <mailto:ldapext-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/ldapext>
List-Post: <mailto:ldapext@ietf.org>
List-Help: <mailto:ldapext-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/ldapext>, <mailto:ldapext-request@ietf.org?subject=subscribe>
X-List-Received-Date: Tue, 16 Jun 2009 15:42:47 -0000

Thanks Kurt, sorry for my asking an answer in another email, it seems we 
were writing at the same time.


Kurt Zeilenga wrote:
>>
>> -the directory is broken
>> -you export to LDIF
>> -compare this LDIF with a previous one from when the directory was 
>> working.
> 
> You don't need UTF-8 for this.  A simple text diff tool will tell you 
> that the base64 differs.

True, diff will tell you that they are different, and where, but then you 
need to decode base64 to find what the text is in the two file to help you 
understand why they are different.

If there were no base64 encryption, then you would know right away, in most 
case, making it a much faster process.


> But now you assume you'll be able to read them.  This is a bad 
> assumption.

I still don't understand why you are saying this. Can you give precise 
examples ?

>  A simple diff tool might show two DIFFERENT values the same 
> way, leading the human to believe there is no difference when there is a 
> significant difference.

So, for example, one file contains U+2026 (ellipsis, "…") while the other 
contains three U_002E (three times the full stop character, "..."), and the 
issue of duplication in Unicode 
(http://en.wikipedia.org/wiki/Duplicate_characters_in_Unicode) ?

Well, what do we do today when diff shows us that there is a difference in 
two ascii files, but our eyes can't see it ? We hex dump of the offending 
line(s), and go "ahah, I've got <CR> here, but <CR><LF> there). On the other 
hand what's the percentage of time you diff files and run into this problem ?

Having those base64 encoded will make the fact that they are different more 
obvious, but won't help us when we're trying to understand why they are 
different, we will still have to go base64 --> UTF-8 --> hex values.

Data can be different because it comes from different sources, in which case 
it is likely that people will use different duplicate of Unicode point to 
express the same thing, but data can also be different because it was 
changed, which is the case I am more concerned with (backup of LDAP servers 
etc...), and in this case, I doubt somebody will change the mu symbol to the 
micro symbol or something similar. It might happen for malicious reasons, 
but I expect that to be a small percentage.


> 
> The key phrase here is "Unicode text".  And most such display tools not 
> only require "well-formed" text, but often cannot display all 
> "well-formed" text.  But removing the ASCII restriction does not make a 
> LDIF file "Unicode text".  It makes it a series of Unicode code points 
> and hence display of it as text will be quite problematic.  

I don't understand what you are saying here. Could you give an example of a 
problematic case, or a link to an explanation ?

Why is a series of Unicode point not text ?
What do you mean by "well-formed text", or actually by text that is not 
well-formed ?

> And even 
> it's displayable, you have the problem that two values might display in 
> the same way, making visual diff'ing problematic.

This is the issue above (Unicode duplication) ? Right ?


 >> Other case: People have mentioned scripts that build LDIF file from
 >> other source, and have mentioned that encoding the values in base64 is
 >> an overhead they could do without.
 >
 > While base64 data is an additional step, it's an additional step that
 > well supported today.  If we lift the ASCII restriction now, we'll have
 > some implementations that do support it and some that don't, and that
 > will cause interop problems.   I cannot support inducing such interop
 > problems without a strong justification.

Adding a new version of the standard, does not remove the old one. You can 
tell people that you only work with version 1 LDIF files. The advantage of 
having a standard, is that the tools will slowly adopt it and will be able 
to deal with it. If we don't create a standard, we run into the situation 
where everybody creates their own format, and create their own tools to 
transform their format to the current LDIF one. If I remember right, Jim 
mentioned that he already uses his version of extended LDIF.


-- 
Yves.
http://www.sollers.ca/


From Kurt.Zeilenga@Isode.com  Tue Jun 16 09:48:18 2009
Return-Path: <Kurt.Zeilenga@Isode.com>
X-Original-To: ldapext@core3.amsl.com
Delivered-To: ldapext@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 1AF363A6C5F for <ldapext@core3.amsl.com>; Tue, 16 Jun 2009 09:48:18 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.814
X-Spam-Level: 
X-Spam-Status: No, score=-1.814 tagged_above=-999 required=5 tests=[AWL=-0.567, BAYES_00=-2.599, J_CHICKENPOX_21=0.6, J_CHICKENPOX_43=0.6, SARE_SUB_ENC_UTF8=0.152]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Mrf2OfbSe6KZ for <ldapext@core3.amsl.com>; Tue, 16 Jun 2009 09:48:16 -0700 (PDT)
Received: from boole.openldap.org (boole.openldap.org [IPv6:2001:4f8:3:ba:219:21ff:fe2f:68d5]) by core3.amsl.com (Postfix) with ESMTP id C7EF03A6BFD for <ldapext@ietf.org>; Tue, 16 Jun 2009 09:48:15 -0700 (PDT)
Received: from [192.168.1.101] (75-141-233-128.dhcp.nv.charter.com [75.141.233.128] (may be forged)) (authenticated bits=0) by boole.openldap.org (8.14.3/8.14.3) with ESMTP id n5GGmHI2067498; Tue, 16 Jun 2009 16:48:19 GMT (envelope-from Kurt.Zeilenga@Isode.com)
Message-Id: <68380E97-521C-4A80-A569-D09F8F626F6F@Isode.com>
From: Kurt Zeilenga <Kurt.Zeilenga@Isode.com>
To: Yves Dorfsman <yves@zioup.com>
In-Reply-To: <4A37BCEB.5040103@zioup.com>
Content-Type: text/plain; charset=WINDOWS-1252; format=flowed; delsp=yes
Content-Transfer-Encoding: quoted-printable
Mime-Version: 1.0 (Apple Message framework v935.3)
Date: Tue, 16 Jun 2009 09:48:17 -0700
References: <49C497F9.7010200@zioup.com>	<CD3905D4-2A25-4C56-8187-3CE10D46C929@isode.com>	<49C870C6.4010803@zioup.com>	<E94B7389-9A6D-4CB6-BB2C-649CCD3FD15B@Isode.com>	<49CB192E.5050105@zioup.com>	<49CB211C.6070108@eb2bcom.com>	<49CB87FE.1050809@zioup.com>	<49CC01DE.6040506@eb2bcom.com>	<4A24557D.7030006@zioup.com> <4A26A05D.8040105@zioup.com>	<245BF18B-2066-4E36-9502-16F4A3140D9E@Isode.com> <4A309775.3080406@zioup.com> <4A311ED1.1030202@stroeder.com> <4A31D27B.3090208@zioup.com> <35B2A165-CE5D-4650-AADE-CC233F71470E@Isode.com> <4A35D23D.5040307@zioup.com> <D437E784-4198-4037-A4EA-0300439C3D2C@Isode.com> <4A37BCEB.5040103@zioup.com>
X-Mailer: Apple Mail (2.935.3)
Cc: ldapext@ietf.org
Subject: Re: [ldapext] UTF-8 full support in LDIF / LDIF v2
X-BeenThere: ldapext@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: LDAP Extension Working Group <ldapext.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/ldapext>, <mailto:ldapext-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/ldapext>
List-Post: <mailto:ldapext@ietf.org>
List-Help: <mailto:ldapext-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/ldapext>, <mailto:ldapext-request@ietf.org?subject=subscribe>
X-List-Received-Date: Tue, 16 Jun 2009 16:48:18 -0000

On Jun 16, 2009, at 8:40 AM, Yves Dorfsman wrote:

>
> Thanks Kurt, sorry for my asking an answer in another email, it =20
> seems we were writing at the same time.
>
>
> Kurt Zeilenga wrote:
>>>
>>> -the directory is broken
>>> -you export to LDIF
>>> -compare this LDIF with a previous one from when the directory was =20=

>>> working.
>> You don't need UTF-8 for this.  A simple text diff tool will tell =20
>> you that the base64 differs.
>
> True, diff will tell you that they are different, and where, but =20
> then you need to decode base64 to find what the text is in the two =20
> file to help you understand why they are different.

You have to compare the values octet by octet to understand why they =20
are different.


> If there were no base64 encryption, then you would know right away, =20=

> in most case, making it a much faster process.

s/encryption/encoding/

I disagree.  Display of UTF-8 as text will hide differences that =20
base64 or octet-by-octet comparison won't hide.

>> But now you assume you'll be able to read them.  This is a bad =20
>> assumption.
>
> I still don't understand why you are saying this. Can you give =20
> precise examples ?

Consider a combining character such as diaeresis (U+0308).  It can =20
only be combined with certain base characters.  It cannot be combined =20=

with arbitrary characters.   If one has an attribute intended to hold =20=

a single Unicode code point, then stores a diaresis as a value of that =20=

attribute, in the UTF-8 LDIF, one would end up diaeresis following a =20
colon or a space, neither of which is valid text.

This is just the tip of the iceberg.  There are even issues where the =20=

LDAP values, being well-formed text by themselves, when composed into =20=

a file will result in the file not being well-formed text, such as in =20=

values which utilize bi-directional text.

IDNA when through all of this.  They found that they had to place =20
significant restrictions on Unicode domain components to ensure that a =20=

domain name was well-formed Unicode text.

You have not suggested placing similar restrictions on LDIF but simply =20=

removing the ASCII restriction.

>
>> A simple diff tool might show two DIFFERENT values the same way, =20
>> leading the human to believe there is no difference when there is a =20=

>> significant difference.
>
> So, for example, one file contains U+2026 (ellipsis, "=85") while the =20=

> other contains three U_002E (three times the full stop character, =20
> "..."), and the issue of duplication in Unicode =
(http://en.wikipedia.org/wiki/Duplicate_characters_in_Unicode=20
> ) ?

Well, that's one case.  I had a few cases in mind: character =20
equivalences and look likes (but non-equivalences).

>
> Well, what do we do today when diff shows us that there is a =20
> difference in two ascii files, but our eyes can't see it ? We hex =20
> dump of the offending line(s), and go "ahah, I've got <CR> here, but =20=

> <CR><LF> there).

But there are more subtles involved.  There are cases where the diff =20
will show a difference, your eye will think it sees the difference, =20
but the actual difference will be hidden.

This is why ldif difference tools are needed and why they have been =20
written.  Removing the UTF-8 restriction won't make their job any =20
easier.  It will just add another encoding option.

> On the other hand what's the percentage of time you diff files and =20
> run into this problem ?

Well, where such problems are less likely, that makes the hidden =20
difference problem even worse (as won't be an expected issue).   In =20
some situations, the problem will be quite likely.

It was likely enough in IDNA for restrictions (*) to be placed on code =20=

point sequences used in domain names.  (* many of the INDA Unicode =20
restrictions were to address other issues, but there are IDNA =20
restrictions in place to ensure that domain components and domain name =20=

sequences of Unicode code points are well formed text.)

>
> Having those base64 encoded will make the fact that they are =20
> different more obvious, but won't help us when we're trying to =20
> understand why they are different, we will still have to go base64 --=20=

> > UTF-8 --> hex values.

While I don't disagree that to understand the differences, one has to =20=

consider the value octet by octet, this in no way implies that we need =20=

another way to encode those octets.

> Data can be different because it comes from different sources, in =20
> which case it is likely that people will use different duplicate of =20=

> Unicode point to express the same thing, but data can also be =20
> different because it was changed, which is the case I am more =20
> concerned with (backup of LDAP servers etc...), and in this case, I =20=

> doubt somebody will change the mu symbol to the micro symbol or =20
> something similar.

Ah, but someone might apply normalization to the data.  This may well =20=

be worthy of detection.

> It might happen for malicious reasons, but I expect that to be a =20
> small percentage.

Could be.  But could happen for non-malicious reasons.  Things could =20
be "broken" because some re-normalized the data using the wrong =20
normalization function.   Visual differencing won't detect this sort =20
of change, as normalization doesn't change the display of the text (it =20=

just changes which code points are used to represent the text).

>> The key phrase here is "Unicode text".  And most such display tools =20=

>> not only require "well-formed" text, but often cannot display all =20
>> "well-formed" text.  But removing the ASCII restriction does not =20
>> make a LDIF file "Unicode text".  It makes it a series of Unicode =20
>> code points and hence display of it as text will be quite =20
>> problematic.
>
> I don't understand what you are saying here. Could you give an =20
> example of a problematic case, or a link to an explanation ?

See the bare combining character illustration above.

> Why is a series of Unicode point not text ?

For a series of Unicode code point to be text, certain restrictions =20
must be maintained.  For instance, U+0308 by itself is not text.  It =20
has to be combined with an appropriate base character to be text.

> What do you mean by "well-formed text", or actually by text that is =20=

> not well-formed ?

Well, even where a sequence of code points represent a sequence of =20
characters and control information, that sequence itself may not be =20
well-formed text.

It is also important to note that the sequence composed of two =20
sequences of code points each representing well formed text may not be =20=

well-formed text.  That is, WF(A + B) may be false even where WF(A) =20
and WF(B) are true.   BIDI is case where this often happens.

>
>> And even it's displayable, you have the problem that two values =20
>> might display in the same way, making visual diff'ing problematic.
>
> This is the issue above (Unicode duplication) ? Right ?

It's a problem of equivalences, look-a-likes, etc.

> >> Other case: People have mentioned scripts that build LDIF file from
> >> other source, and have mentioned that encoding the values in =20
> base64 is
> >> an overhead they could do without.
> >
> > While base64 data is an additional step, it's an additional step =20
> that
> > well supported today.  If we lift the ASCII restriction now, we'll =20=

> have
> > some implementations that do support it and some that don't, and =20
> that
> > will cause interop problems.   I cannot support inducing such =20
> interop
> > problems without a strong justification.
>
> Adding a new version of the standard, does not remove the old one. =20
> You can tell people that you only work with version 1 LDIF files. =20
> The advantage of having a standard, is that the tools will slowly =20
> adopt it and will be able to deal with it.  If we don't create a =20
> standard, we run into the situation where everybody creates their =20
> own format, and create their own tools to transform their format to =20=

> the current LDIF one.

When LDIFv1 was introduced, it brought with it interop problems with U-=20=

Mich LDIF (v0).  The standards community however well justified the =20
standardization of LDIF, the changes from U-Mich LDIF.  But to this =20
day, we still suffer from some interop problems tied specifically to =20
differences between U-Mich LDIF and v1.

> If I remember right, Jim mentioned that he already uses his version =20=

> of extended LDIF.

There are actually a number of extended versions of LDIF in use =20
today... and a number of interop problems exist because of this.  =20
Standardization won't necessary remove the desire to have extended =20
versions of LDIF, as the standard LDIF is never going to met =20
everyone's desires for additional capabilities.

In my opinion, in revising Standard Track LDIF, we should focus on =20
changes that improve data interchange functionality and =20
interoperability.  We should dismiss changes that our outside of the =20
primary goal of having an interoperable interchange format, includes =20
dismissing changes intended to improve the "human readability" of LDIF.

-- Kurt


From Kurt.Zeilenga@Isode.com  Tue Jun 16 12:01:43 2009
Return-Path: <Kurt.Zeilenga@Isode.com>
X-Original-To: ldapext@core3.amsl.com
Delivered-To: ldapext@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 132C428C1AF for <ldapext@core3.amsl.com>; Tue, 16 Jun 2009 12:01:43 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.371
X-Spam-Level: 
X-Spam-Status: No, score=-2.371 tagged_above=-999 required=5 tests=[AWL=0.076,  BAYES_00=-2.599, SARE_SUB_ENC_UTF8=0.152]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id oUnFqJkYJfZH for <ldapext@core3.amsl.com>; Tue, 16 Jun 2009 12:01:42 -0700 (PDT)
Received: from boole.openldap.org (boole.openldap.org [IPv6:2001:4f8:3:ba:219:21ff:fe2f:68d5]) by core3.amsl.com (Postfix) with ESMTP id E45CC3A6D6C for <ldapext@ietf.org>; Tue, 16 Jun 2009 12:01:38 -0700 (PDT)
Received: from [192.168.1.102] (75-141-233-128.dhcp.nv.charter.com [75.141.233.128] (may be forged)) (authenticated bits=0) by boole.openldap.org (8.14.3/8.14.3) with ESMTP id n5GJ1gas090977; Tue, 16 Jun 2009 19:01:42 GMT (envelope-from Kurt.Zeilenga@Isode.com)
Message-Id: <C6C69FB4-E767-4A18-89F6-B070E88DA82F@Isode.com>
From: Kurt Zeilenga <Kurt.Zeilenga@Isode.com>
To: Yves Dorfsman <yves@zioup.com>
In-Reply-To: <68380E97-521C-4A80-A569-D09F8F626F6F@Isode.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v935.3)
Date: Tue, 16 Jun 2009 12:01:41 -0700
References: <49C497F9.7010200@zioup.com>	<CD3905D4-2A25-4C56-8187-3CE10D46C929@isode.com>	<49C870C6.4010803@zioup.com>	<E94B7389-9A6D-4CB6-BB2C-649CCD3FD15B@Isode.com>	<49CB192E.5050105@zioup.com>	<49CB211C.6070108@eb2bcom.com>	<49CB87FE.1050809@zioup.com>	<49CC01DE.6040506@eb2bcom.com>	<4A24557D.7030006@zioup.com> <4A26A05D.8040105@zioup.com>	<245BF18B-2066-4E36-9502-16F4A3140D9E@Isode.com> <4A309775.3080406@zioup.com> <4A311ED1.1030202@stroeder.com> <4A31D27B.3090208@zioup.com> <35B2A165-CE5D-4650-AADE-CC233F71470E@Isode.com> <4A35D23D.5040307@zioup.com> <D437E784-4198-4037-A4EA-0300439C3D2C@Isode.com> <4A37BCEB.5040103@zioup.com> <68380E97-521C-4A80-A569-D09F8F626F6F@Isode.com>
X-Mailer: Apple Mail (2.935.3)
Cc: ldapext@ietf.org
Subject: Re: [ldapext] UTF-8 full support in LDIF / LDIF v2
X-BeenThere: ldapext@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: LDAP Extension Working Group <ldapext.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/ldapext>, <mailto:ldapext-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/ldapext>
List-Post: <mailto:ldapext@ietf.org>
List-Help: <mailto:ldapext-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/ldapext>, <mailto:ldapext-request@ietf.org?subject=subscribe>
X-List-Received-Date: Tue, 16 Jun 2009 19:01:43 -0000

Another issue...

Though LDIFv1 doesn't actually allow multi-byte UTF-8 anywhere (except  
in base64 encoded strings), RFC 2849 did say:
   Implementations SHOULD NOT fold lines in the middle of a multi-byte  
UTF-8 character.
(this was a left over, I think, when non-base64 encodings of UTF-8 was  
being considered for LDIFv1.  That is, the IETF seems to have already  
considered this issue when it was developing LDIFv1 but choose to  
require base64 of multi-byte UTF-8 characters.)
Anyways, if we remove the ASCII restriction and leave the above  
SHOULD, that would mean that implementations would have to deal with  
folding in the middle of multi-byte UTF-8 sequences, and that means  
the file as whole might not be valid UTF-8, and that would be  
problematic.
If we do remove the ASCII restriction (which I don't support), I think  
we install a couple of restrictions.
1) The resulting file MUST only be valid UTF-8 (minimally length  
encodings, a start octet must be followed by continuation octets,  
etc.) AND MUST be a valid Unicode code point (U+0000 through U+D7FF, U 
+E000 through U+10FFFF, inclusively) excluding the NUL (U+0000) code  
point.
2) Lines longer that 76 octets MUST be folded.  Folding of lines MUST  
NOT occur in the middle of a multi-byte UTF-8 character.   
Implementations need to take care that both of these requirements are  
met.
Both of these requirements will significantly increase the complexity  
of LDIF encoders.  Though the first is similar to current requirement  
to ensure that values encodes as SAFE-STRING met all of SAFE-STRING  
requirements, these checks are octet by octet.  Checking for valid  
UTF-8 and Unicode requires both an expansion of each UTF-8 sequence  
but the code point represented by that UTF-8 sequence.  And the second  
requires a bit more math, something experience has show implementors  
tend to get wrong.
We'd also need text explaining that format does not ensure the UTF-8  
represents well-formed text, or even text, and hence may not be  
displayable or editable.  And even displayable, users must be careful  
not to rely on visual inspection.
Also, users must be careful to avoid passing LDIF data through systems  
that might apply Unicode normalization or other transformations.  Such  
systems will likely lead to unintended changes to the LDAP data  
represented by the original LDIF (and/or cause the data to be non- 
conforming LDIF).  For instance, when emailing an LDIF file, both the  
sending and receiving user must be careful that their MUAs (and MTAs)  
are not altering the LDIF data.  Unfortunately, many MUAs and MTAs do  
alter text attachments (to conform to various conventions).  While  
such alterations may be desirable for text attachments, it is not  
desirable for LDIF data (which are commonly sent as sent as text  
attachments).
By raising these points, I hope to show that simply removing the ASCII  
restrictions will lead to problems, and even though some of them (such  
as invalid UTF-8) can be address by installing various restrictions,  
the simple fact that LDIF represents LDAP data not text will lead to  
various problems (such as unintended text file conversions, inability  
to use text processing programs, etc.).
-- Kurt

From hyc@highlandsun.com  Tue Jun 16 12:33:10 2009
Return-Path: <hyc@highlandsun.com>
X-Original-To: ldapext@core3.amsl.com
Delivered-To: ldapext@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id EC9713A6D8B for <ldapext@core3.amsl.com>; Tue, 16 Jun 2009 12:33:10 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.685
X-Spam-Level: 
X-Spam-Status: No, score=-1.685 tagged_above=-999 required=5 tests=[AWL=0.762,  BAYES_00=-2.599, SARE_SUB_ENC_UTF8=0.152]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ZIxJa-biYYAE for <ldapext@core3.amsl.com>; Tue, 16 Jun 2009 12:33:10 -0700 (PDT)
Received: from mail.highlandsun.com (mail.highlandsun.com [70.87.222.79]) by core3.amsl.com (Postfix) with ESMTP id 02AD93A6B92 for <ldapext@ietf.org>; Tue, 16 Jun 2009 12:33:10 -0700 (PDT)
Received: from [127.0.0.1] (localhost [127.0.0.1]) by mail.highlandsun.com (Postfix) with ESMTP id DDF4E10F96; Tue, 16 Jun 2009 15:32:37 -0400 (EDT)
Message-ID: <4A37F352.1000600@highlandsun.com>
Date: Tue, 16 Jun 2009 12:32:34 -0700
From: Howard Chu <hyc@highlandsun.com>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; rv:1.9.1b5pre) Gecko/20090611 SeaMonkey/2.0a1pre Firefox/3.0.3
MIME-Version: 1.0
To: Kurt Zeilenga <Kurt.Zeilenga@Isode.com>
References: <49C497F9.7010200@zioup.com>	<CD3905D4-2A25-4C56-8187-3CE10D46C929@isode.com>	<49C870C6.4010803@zioup.com>	<E94B7389-9A6D-4CB6-BB2C-649CCD3FD15B@Isode.com>	<49CB192E.5050105@zioup.com>	<49CB211C.6070108@eb2bcom.com>	<49CB87FE.1050809@zioup.com>	<49CC01DE.6040506@eb2bcom.com>	<4A24557D.7030006@zioup.com>	<4A26A05D.8040105@zioup.com>	<245BF18B-2066-4E36-9502-16F4A3140D9E@Isode.com>	<4A309775.3080406@zioup.com> <4A311ED1.1030202@stroeder.com>	<4A31D27B.3090208@zioup.com>	<35B2A165-CE5D-4650-AADE-CC233F71470E@Isode.com>	<4A35D23D.5040307@zioup.com>	<D437E784-4198-4037-A4EA-0300439C3D2C@Isode.com>	<4A37BCEB.5040103@zioup.com>	<68380E97-521C-4A80-A569-D09F8F626F6F@Isode.com> <C6C69FB4-E767-4A18-89F6-B070E88DA82F@Isode.com>
In-Reply-To: <C6C69FB4-E767-4A18-89F6-B070E88DA82F@Isode.com>
Content-Type: text/plain; charset=us-ascii; format=flowed
Content-Transfer-Encoding: 7bit
Cc: ldapext@ietf.org
Subject: Re: [ldapext] UTF-8 full support in LDIF / LDIF v2
X-BeenThere: ldapext@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: LDAP Extension Working Group <ldapext.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/ldapext>, <mailto:ldapext-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/ldapext>
List-Post: <mailto:ldapext@ietf.org>
List-Help: <mailto:ldapext-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/ldapext>, <mailto:ldapext-request@ietf.org?subject=subscribe>
X-List-Received-Date: Tue, 16 Jun 2009 19:33:11 -0000

Kurt Zeilenga wrote:
> By raising these points, I hope to show that simply removing the ASCII
> restrictions will lead to problems, and even though some of them (such
> as invalid UTF-8) can be address by installing various restrictions,
> the simple fact that LDIF represents LDAP data not text will lead to
> various problems (such as unintended text file conversions, inability
> to use text processing programs, etc.).
> -- Kurt

Agreed. Not to mention that to implement this properly will require complete 
schema knowledge at the time the LDIF is generated. (Otherwise, how do you 
distinguish a genuine octetString value, which cannot be safely represented in 
UTF-8, from a directoryString value...)

-- 
   -- Howard Chu
   CTO, Symas Corp.           http://www.symas.com
   Director, Highland Sun     http://highlandsun.com/hyc/
   Chief Architect, OpenLDAP  http://www.openldap.org/project/

From Kurt.Zeilenga@Isode.com  Tue Jun 16 13:25:53 2009
Return-Path: <Kurt.Zeilenga@Isode.com>
X-Original-To: ldapext@core3.amsl.com
Delivered-To: ldapext@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 0DFF13A6C30 for <ldapext@core3.amsl.com>; Tue, 16 Jun 2009 13:25:53 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.076
X-Spam-Level: 
X-Spam-Status: No, score=-2.076 tagged_above=-999 required=5 tests=[AWL=-0.229, BAYES_00=-2.599, J_CHICKENPOX_25=0.6, SARE_SUB_ENC_UTF8=0.152]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id pPOfzrJo7aUa for <ldapext@core3.amsl.com>; Tue, 16 Jun 2009 13:25:52 -0700 (PDT)
Received: from boole.openldap.org (boole.openldap.org [IPv6:2001:4f8:3:ba:219:21ff:fe2f:68d5]) by core3.amsl.com (Postfix) with ESMTP id B93EB3A6C01 for <ldapext@ietf.org>; Tue, 16 Jun 2009 13:25:51 -0700 (PDT)
Received: from [192.168.1.102] (75-141-233-128.dhcp.nv.charter.com [75.141.233.128] (may be forged)) (authenticated bits=0) by boole.openldap.org (8.14.3/8.14.3) with ESMTP id n5GKPomP094337; Tue, 16 Jun 2009 20:25:51 GMT (envelope-from Kurt.Zeilenga@Isode.com)
Message-Id: <C8753ADB-6E4A-4841-8E60-FA4B5B51D473@Isode.com>
From: Kurt Zeilenga <Kurt.Zeilenga@Isode.com>
To: Howard Chu <hyc@highlandsun.com>
In-Reply-To: <4A37F352.1000600@highlandsun.com>
Content-Type: multipart/mixed; boundary=Apple-Mail-24-588778161
Mime-Version: 1.0 (Apple Message framework v935.3)
Date: Tue, 16 Jun 2009 13:25:50 -0700
References: <49C497F9.7010200@zioup.com>	<CD3905D4-2A25-4C56-8187-3CE10D46C929@isode.com>	<49C870C6.4010803@zioup.com>	<E94B7389-9A6D-4CB6-BB2C-649CCD3FD15B@Isode.com>	<49CB192E.5050105@zioup.com>	<49CB211C.6070108@eb2bcom.com>	<49CB87FE.1050809@zioup.com>	<49CC01DE.6040506@eb2bcom.com>	<4A24557D.7030006@zioup.com>	<4A26A05D.8040105@zioup.com>	<245BF18B-2066-4E36-9502-16F4A3140D9E@Isode.com>	<4A309775.3080406@zioup.com> <4A311ED1.1030202@stroeder.com>	<4A31D27B.3090208@zioup.com>	<35B2A165-CE5D-4650-AADE-CC233F71470E@Isode.com>	<4A35D23D.5040307@zioup.com>	<D437E784-4198-4037-A4EA-0300439C3D2C@Isode.com>	<4A37BCEB.5040103@zioup.com>	<68380E97-521C-4A80-A569-D09F8F626F6F@Isode.com> <C6C69FB4-E767-4A18-89F6-B070E88DA82F@Isode.com> <4A37F352.1000600@highlandsun.com>
X-Mailer: Apple Mail (2.935.3)
Cc: ldapext@ietf.org
Subject: Re: [ldapext] UTF-8 full support in LDIF / LDIF v2
X-BeenThere: ldapext@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: LDAP Extension Working Group <ldapext.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/ldapext>, <mailto:ldapext-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/ldapext>
List-Post: <mailto:ldapext@ietf.org>
List-Help: <mailto:ldapext-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/ldapext>, <mailto:ldapext-request@ietf.org?subject=subscribe>
X-List-Received-Date: Tue, 16 Jun 2009 20:25:53 -0000

--Apple-Mail-24-588778161
Content-Type: text/plain;
	charset=US-ASCII;
	format=flowed;
	delsp=yes
Content-Transfer-Encoding: 7bit


On Jun 16, 2009, at 12:32 PM, Howard Chu wrote:

> Not to mention that to implement this properly will require complete  
> schema knowledge at the time the LDIF is generated. (Otherwise, how  
> do you distinguish a genuine octetString value, which cannot be  
> safely represented in UTF-8, from a directoryString value...)

Well, one could scan the value to see the octets are a valid UTF-8  
sequence of valid Unicode code points just as today most  
implementations scan data for octets within of SAFE-STRING.  The  
significant difference is the check is straight forward in LDIFv1, as  
it's san octet-by-octet check.  But if we allow UTF-8 sequences of  
valid Unicode points, each octet of the value must be checked to see  
that it's part of a valid UTF-8 sequence, and each UTF-8 sequence  
checked to see if encodes a valid Unicode code point.  And then  
wrapping becomes more complicated, etc..

And even with all of that, LDIF would still not be well-formed Unicode  
text.  And even if we solved that (by even more complex restrictions  
on what Unicode code point sequences can be represented as UTF-8  
instead of base64-encoded UTF-8), we'd have the problem of unintended  
Unicode transformations in transporting LDIF.  (We have this problem  
with LDIFv1, but it's generally limited to end-of-line characters.   
With UTF-8, data will be impacted.  For instance, consider MUAs (or  
the like) that might convert (on send or receive) text to Net-Unicode.)

I've expanded my UTF-8 LDIF with some more goofiness.


--Apple-Mail-24-588778161
Content-Disposition: attachment;
	filename=utf8-ldif.txt
Content-Type: text/plain;
	x-unix-mode=0664;
	name="utf8-ldif.txt"
Content-Transfer-Encoding: quoted-printable

version: 2

dn: cn=3Dfunky
bom:=EF=BB=BF
smiley-face:=E2=98=BA
# only SPACE is special
no-break-space: =20
zero-width-space:=E2=80=8B
word-joiner:=E2=81=A0
ideographic-space:=E3=80=80
zero-width-no-break-space:=EF=BB=BF
# line separators and other such things
nel:=C2=85
ls:=E2=80=A8
ps:=E2=80=A9
ff:=0C
# these hyphen differ but may look the same
hyphen-minus:-
hyphen:=E2=80=90
non-breaking-hyphen:=E2=80=91
figure-dash:=E2=80=92
en-dash:=E2=80=93
minus-sign:=E2=88=92
roman-uncia-sign:=F0=90=86=91
# these differ but may look the same
o-diaeresis:=C3=B6
o-diaeresis-decomposed:o=CC=88
# ignorables
ignore:=E2=81=A0
ignore:=E2=81=A1
ignore:=C2=AD
# inside-out rule
inside-out:a=CC=88=CC=83=CC=A3=CC=AD
inside-out:=E0=B8=82=E0=B8=B6=E0=B9=89
# combining character
diaeresis:=CC=88
# bidi
bidi:=DA=83
bidi:=D9=B1AB=D9=B9=DA=83
bidi:=D9=B137=D9=B9=DA=83
bidi:=E2=80=AEABC=E2=80=AC
bidi:=E2=80=AD=D9=B1=D9=B9=DA=83=E2=80=AC
# bidi wrapped
bidi:
 =DA=83
bidi:=D9=B1A
 B=D9=B9=DA=83
bidi:=D9=B13
 7=D9=B9=DA=83
bidi:=E2=80=AEA
 BC=E2=80=AC
bidi:=E2=80=AD=D9=B1
 =D9=B9=DA=83=E2=80=AC
# private use
pu:=EE=80=80=EE=80=81=EE=80=82

--Apple-Mail-24-588778161
Content-Type: text/plain;
	charset=US-ASCII;
	format=flowed
Content-Transfer-Encoding: 7bit




-- Kurt
--Apple-Mail-24-588778161--

From steven.legg@eb2bcom.com  Tue Jun 16 16:54:08 2009
Return-Path: <steven.legg@eb2bcom.com>
X-Original-To: ldapext@core3.amsl.com
Delivered-To: ldapext@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id B07703A6DBD for <ldapext@core3.amsl.com>; Tue, 16 Jun 2009 16:54:08 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: 0.378
X-Spam-Level: 
X-Spam-Status: No, score=0.378 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, FH_HOST_EQ_D_D_D_D=0.765, FH_HOST_EQ_D_D_D_DB=0.888, HOST_EQ_STATIC=1.172, SARE_SUB_ENC_UTF8=0.152]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id M+qBaBVClzND for <ldapext@core3.amsl.com>; Tue, 16 Jun 2009 16:54:08 -0700 (PDT)
Received: from eb2bcom.com (165.203.232.72.static.reverse.ltdomains.com [72.232.203.165]) by core3.amsl.com (Postfix) with ESMTP id E425B3A6AA8 for <ldapext@ietf.org>; Tue, 16 Jun 2009 16:54:07 -0700 (PDT)
Received: from [202.164.192.219] (helo=[10.1.2.237]) by host.eb2bcom.com with esmtpa (Exim 4.69) (envelope-from <steven.legg@eb2bcom.com>) id 1MGiTr-00083A-3R; Wed, 17 Jun 2009 09:54:19 +1000
Message-ID: <4A3830A6.4030407@eb2bcom.com>
Date: Wed, 17 Jun 2009 09:54:14 +1000
From: Steven Legg <steven.legg@eb2bcom.com>
User-Agent: Thunderbird 2.0.0.21 (Windows/20090302)
MIME-Version: 1.0
To: Yves Dorfsman <yves@zioup.com>
References: <49C497F9.7010200@zioup.com>	<CD3905D4-2A25-4C56-8187-3CE10D46C929@isode.com>	<49C870C6.4010803@zioup.com>	<E94B7389-9A6D-4CB6-BB2C-649CCD3FD15B@Isode.com>	<49CB192E.5050105@zioup.com>	<49CB211C.6070108@eb2bcom.com>	<49CB87FE.1050809@zioup.com>	<49CC01DE.6040506@eb2bcom.com>	<4A24557D.7030006@zioup.com>	<4A26A05D.8040105@zioup.com>	<245BF18B-2066-4E36-9502-16F4A3140D9E@Isode.com>	<4A309775.3080406@zioup.com>	<4A311ED1.1030202@stroeder.com>	<4A31D27B.3090208@zioup.com>	<4A325A40.2050802@stroeder.com>	<4A35CDDE.8000604@zioup.com> <4A37719E.3010006@stroeder.com> <4A37A5D9.4040901@zioup.com>
In-Reply-To: <4A37A5D9.4040901@zioup.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
X-AntiAbuse: This header was added to track abuse, please include it with any abuse report
X-AntiAbuse: Primary Hostname - host.eb2bcom.com
X-AntiAbuse: Original Domain - ietf.org
X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12]
X-AntiAbuse: Sender Address Domain - eb2bcom.com
X-Source: 
X-Source-Args: 
X-Source-Dir: 
Cc: ldapext@ietf.org
Subject: Re: [ldapext] UTF-8 full support in LDIF / LDIF v2
X-BeenThere: ldapext@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: LDAP Extension Working Group <ldapext.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/ldapext>, <mailto:ldapext-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/ldapext>
List-Post: <mailto:ldapext@ietf.org>
List-Help: <mailto:ldapext-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/ldapext>, <mailto:ldapext-request@ietf.org?subject=subscribe>
X-List-Received-Date: Tue, 16 Jun 2009 23:54:08 -0000

Yves,

Yves Dorfsman wrote:
[snip]
> Listing the case we have so far (please argument against if you think a 
> case is not valid):
> 
> -comparing at the file level by humans, to fix an issue for example ; 
> this imply that the user is using a display properly setup for UTF-8 
> (Yves Dorfsman).
> 
> -simplifying scripts to import/export data form/to other format to/from 
> LDIF (Steven Legg. Steven).
> 
> -simplifying online editors (Michael Stöder).
> 
> 
> 
> 
> Steven:
> Can you confirm this is what you were saying, otherwise reword ?

That is my main use case, including where some of the composition is done
by hand.

For the record, although XML documents can be more or less arbitrary UTF-8
character strings, most of the ones I deal with are pure ASCII. It is just
the line separators that require me to base64 encode such documents when
storing them as directory attribute values through LDIFv1.

Also, X.500 directories already lose something in the translation when outputting
as LDIF. For example, the choice in a DirectoryString is lost and if that
choice is teletexString then transcoding wipes out the exact octet encoding.
Most of the ad-hoc LDAP string encodings are lossy in some respect.
Such changes are tolerable because the resulting value in LDIF is the same
as far as the matching rules are concerned. Unicode normalization of the
extended LDIF output is a similar situation.

Regards,
Steven

From Kurt.Zeilenga@Isode.com  Tue Jun 16 20:12:39 2009
Return-Path: <Kurt.Zeilenga@Isode.com>
X-Original-To: ldapext@core3.amsl.com
Delivered-To: ldapext@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 2783828C10C for <ldapext@core3.amsl.com>; Tue, 16 Jun 2009 20:12:39 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.361
X-Spam-Level: 
X-Spam-Status: No, score=-2.361 tagged_above=-999 required=5 tests=[AWL=0.086,  BAYES_00=-2.599, SARE_SUB_ENC_UTF8=0.152]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Jhkx49S4Y9uz for <ldapext@core3.amsl.com>; Tue, 16 Jun 2009 20:12:38 -0700 (PDT)
Received: from boole.openldap.org (boole.openldap.org [IPv6:2001:4f8:3:ba:219:21ff:fe2f:68d5]) by core3.amsl.com (Postfix) with ESMTP id 631743A6D8D for <ldapext@ietf.org>; Tue, 16 Jun 2009 20:12:38 -0700 (PDT)
Received: from [192.168.1.102] (75-141-233-128.dhcp.nv.charter.com [75.141.233.128] (may be forged)) (authenticated bits=0) by boole.openldap.org (8.14.3/8.14.3) with ESMTP id n5H3CZrX024384; Wed, 17 Jun 2009 03:12:36 GMT (envelope-from Kurt.Zeilenga@Isode.com)
Message-Id: <93053DE7-C324-4124-BF8F-B3C7088D66EB@Isode.com>
From: Kurt Zeilenga <Kurt.Zeilenga@Isode.com>
To: Steven Legg <steven.legg@eb2bcom.com>
In-Reply-To: <4A3830A6.4030407@eb2bcom.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v935.3)
Date: Tue, 16 Jun 2009 20:12:35 -0700
References: <49C497F9.7010200@zioup.com>	<CD3905D4-2A25-4C56-8187-3CE10D46C929@isode.com>	<49C870C6.4010803@zioup.com>	<E94B7389-9A6D-4CB6-BB2C-649CCD3FD15B@Isode.com>	<49CB192E.5050105@zioup.com>	<49CB211C.6070108@eb2bcom.com>	<49CB87FE.1050809@zioup.com>	<49CC01DE.6040506@eb2bcom.com>	<4A24557D.7030006@zioup.com>	<4A26A05D.8040105@zioup.com>	<245BF18B-2066-4E36-9502-16F4A3140D9E@Isode.com>	<4A309775.3080406@zioup.com>	<4A311ED1.1030202@stroeder.com>	<4A31D27B.3090208@zioup.com>	<4A325A40.2050802@stroeder.com>	<4A35CDDE.8000604@zioup.com> <4A37719E.3010006@stroeder.com> <4A37A5D9.4040901@zioup.com> <4A3830A6.4030407@eb2bcom.com>
X-Mailer: Apple Mail (2.935.3)
Cc: ldapext@ietf.org
Subject: Re: [ldapext] UTF-8 full support in LDIF / LDIF v2
X-BeenThere: ldapext@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: LDAP Extension Working Group <ldapext.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/ldapext>, <mailto:ldapext-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/ldapext>
List-Post: <mailto:ldapext@ietf.org>
List-Help: <mailto:ldapext-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/ldapext>, <mailto:ldapext-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 17 Jun 2009 03:12:39 -0000

On Jun 16, 2009, at 4:54 PM, Steven Legg wrote:

> Also, X.500 directories already lose something in the translation  
> when outputting
> as LDIF.  For example, the choice in a DirectoryString is lost and  
> if that
> choice is teletexString then transcoding wipes out the exact octet  
> encoding.
> Most of the ad-hoc LDAP string encodings are lossy in some respect.
> Such changes are tolerable because the resulting value in LDIF is  
> the same
> as far as the matching rules are concerned. Unicode normalization of  
> the
> extended LDIF output is a similar situation.


The loss you are talking about is inherent in LDAP not LDIF.  That is,  
LDIF does not lose anything (for the LDAP requests it's design to  
represent) in translation to/from LDAP.  I don't it is tolerable for  
an LDAP intermediate format to "lose" LDAP information.

Unicode normalization of the LDIF will be problematic where the  
normalization required by the directory is different.   For instance,  
if the value is expected to be NFD but NFC was applied breakage will  
occur.

-- Kurt


From steven.legg@eb2bcom.com  Wed Jun 17 23:57:32 2009
Return-Path: <steven.legg@eb2bcom.com>
X-Original-To: ldapext@core3.amsl.com
Delivered-To: ldapext@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 6510E28C0D6 for <ldapext@core3.amsl.com>; Wed, 17 Jun 2009 23:57:32 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: 0.828
X-Spam-Level: 
X-Spam-Status: No, score=0.828 tagged_above=-999 required=5 tests=[AWL=0.450,  BAYES_00=-2.599, FH_HOST_EQ_D_D_D_D=0.765, FH_HOST_EQ_D_D_D_DB=0.888,  HOST_EQ_STATIC=1.172, SARE_SUB_ENC_UTF8=0.152]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Hxq6kv8ULWY0 for <ldapext@core3.amsl.com>; Wed, 17 Jun 2009 23:57:31 -0700 (PDT)
Received: from eb2bcom.com (165.203.232.72.static.reverse.ltdomains.com [72.232.203.165]) by core3.amsl.com (Postfix) with ESMTP id 982803A67E9 for <ldapext@ietf.org>; Wed, 17 Jun 2009 23:57:31 -0700 (PDT)
Received: from eth3065.vic.adsl.internode.on.net ([150.101.156.248] helo=[192.168.1.182]) by host.eb2bcom.com with esmtpa (Exim 4.69) (envelope-from <steven.legg@eb2bcom.com>) id 1MHBZA-0008DG-Bb; Thu, 18 Jun 2009 16:57:44 +1000
Message-ID: <4A39E563.1040107@eb2bcom.com>
Date: Thu, 18 Jun 2009 16:57:39 +1000
From: Steven Legg <steven.legg@eb2bcom.com>
User-Agent: Thunderbird 2.0.0.21 (Windows/20090302)
MIME-Version: 1.0
To: Kurt Zeilenga <Kurt.Zeilenga@Isode.com>
References: <49C497F9.7010200@zioup.com>	<CD3905D4-2A25-4C56-8187-3CE10D46C929@isode.com>	<49C870C6.4010803@zioup.com>	<E94B7389-9A6D-4CB6-BB2C-649CCD3FD15B@Isode.com>	<49CB192E.5050105@zioup.com>	<49CB211C.6070108@eb2bcom.com>	<49CB87FE.1050809@zioup.com>	<49CC01DE.6040506@eb2bcom.com>	<4A24557D.7030006@zioup.com>	<4A26A05D.8040105@zioup.com>	<245BF18B-2066-4E36-9502-16F4A3140D9E@Isode.com>	<4A309775.3080406@zioup.com>	<4A311ED1.1030202@stroeder.com>	<4A31D27B.3090208@zioup.com>	<4A325A40.2050802@stroeder.com>	<4A35CDDE.8000604@zioup.com>	<4A37719E.3010006@stroeder.com> <4A37A5D9.4040901@zioup.com>	<4A3830A6.4030407@eb2bcom.com> <93053DE7-C324-4124-BF8F-B3C7088D66EB@Isode.com>
In-Reply-To: <93053DE7-C324-4124-BF8F-B3C7088D66EB@Isode.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-AntiAbuse: This header was added to track abuse, please include it with any abuse report
X-AntiAbuse: Primary Hostname - host.eb2bcom.com
X-AntiAbuse: Original Domain - ietf.org
X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12]
X-AntiAbuse: Sender Address Domain - eb2bcom.com
X-Source: 
X-Source-Args: 
X-Source-Dir: 
Cc: ldapext@ietf.org
Subject: Re: [ldapext] UTF-8 full support in LDIF / LDIF v2
X-BeenThere: ldapext@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: LDAP Extension Working Group <ldapext.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/ldapext>, <mailto:ldapext-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/ldapext>
List-Post: <mailto:ldapext@ietf.org>
List-Help: <mailto:ldapext-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/ldapext>, <mailto:ldapext-request@ietf.org?subject=subscribe>
X-List-Received-Date: Thu, 18 Jun 2009 06:57:32 -0000

Kurt,

Kurt Zeilenga wrote:
> 
> On Jun 16, 2009, at 4:54 PM, Steven Legg wrote:
> 
>> Also, X.500 directories already lose something in the translation when 
>> outputting
>> as LDIF.  For example, the choice in a DirectoryString is lost and if 
>> that
>> choice is teletexString then transcoding wipes out the exact octet 
>> encoding.
>> Most of the ad-hoc LDAP string encodings are lossy in some respect.
>> Such changes are tolerable because the resulting value in LDIF is the 
>> same
>> as far as the matching rules are concerned. Unicode normalization of the
>> extended LDIF output is a similar situation.
> 
> 
> The loss you are talking about is inherent in LDAP not LDIF.  That is, 
> LDIF does not lose anything (for the LDAP requests it's design to 
> represent) in translation to/from LDAP.  I don't it is tolerable for an 
> LDAP intermediate format to "lose" LDAP information.

A "here" document mechanism where the UTF-8 character sequence between
the end of the introducer and the beginning of the terminator is the
literal directory attribute value without any modification would satisfy
the no loss requirement. We could always recommend base64 encoding the
entire file to transfer it across the network, or at least treating it as
a binary file, or suggest using LDIFv1 instead for such purposes.

For me, the convenience of being able to easily view, edit or compose the
content in files I'm uploading outweighs the need to be careful about
which text handling tools I use. If others have similar or overlapping
requirements it would be better for us to use the same specification even
if it never becomes a proposed standard RFC.

Would you feel better if we called it the LDAP Data Composition Format(LDCF)
instead of LDIFv2 or Extended LDIF ?

Regards,
Steven

> 
> Unicode normalization of the LDIF will be problematic where the 
> normalization required by the directory is different.   For instance, if 
> the value is expected to be NFD but NFC was applied breakage will occur.
> 
> -- Kurt
> 
> _______________________________________________
> Ldapext mailing list
> Ldapext@ietf.org
> https://www.ietf.org/mailman/listinfo/ldapext

From michael@stroeder.com  Thu Jun 18 03:15:10 2009
Return-Path: <michael@stroeder.com>
X-Original-To: ldapext@core3.amsl.com
Delivered-To: ldapext@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 45E9A3A68EC for <ldapext@core3.amsl.com>; Thu, 18 Jun 2009 03:15:10 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.147
X-Spam-Level: 
X-Spam-Status: No, score=-2.147 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, MIME_8BIT_HEADER=0.3, SARE_SUB_ENC_UTF8=0.152]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id dJYEGrjnZXcE for <ldapext@core3.amsl.com>; Thu, 18 Jun 2009 03:15:09 -0700 (PDT)
Received: from srv1.stroeder.com (srv1.stroeder.com [213.240.180.113]) by core3.amsl.com (Postfix) with ESMTP id 3F9E03A6896 for <ldapext@ietf.org>; Thu, 18 Jun 2009 03:15:09 -0700 (PDT)
Received: from localhost (localhost [127.0.0.1]) by srv1.stroeder.com (Postfix) with ESMTP id 32F144E074; Thu, 18 Jun 2009 12:14:58 +0200 (CEST)
X-Virus-Scanned: amavisd-new at stroeder.com
Received: from srv1.stroeder.com ([127.0.0.1]) by localhost (srv1.stroeder.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 0m3VFhsVMRGC; Thu, 18 Jun 2009 12:14:55 +0200 (CEST)
Received: from [10.1.0.2] (unknown [10.1.0.2]) by srv1.stroeder.com (Postfix) with ESMTP id B33614E06D; Thu, 18 Jun 2009 12:14:54 +0200 (CEST)
Message-ID: <4A3A13B2.6060307@stroeder.com>
Date: Thu, 18 Jun 2009 12:15:14 +0200
From: =?ISO-8859-1?Q?Michael_Str=F6der?= <michael@stroeder.com>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.22) Gecko/20090606 SeaMonkey/1.1.17
MIME-Version: 1.0
To: Steven Legg <steven.legg@eb2bcom.com>
References: <49C497F9.7010200@zioup.com>	<CD3905D4-2A25-4C56-8187-3CE10D46C929@isode.com>	<49C870C6.4010803@zioup.com>	<E94B7389-9A6D-4CB6-BB2C-649CCD3FD15B@Isode.com>	<49CB192E.5050105@zioup.com>	<49CB211C.6070108@eb2bcom.com>	<49CB87FE.1050809@zioup.com>	<49CC01DE.6040506@eb2bcom.com>	<4A24557D.7030006@zioup.com>	<4A26A05D.8040105@zioup.com>	<245BF18B-2066-4E36-9502-16F4A3140D9E@Isode.com>	<4A309775.3080406@zioup.com>	<4A311ED1.1030202@stroeder.com>	<4A31D27B.3090208@zioup.com>	<4A325A40.2050802@stroeder.com>	<4A35CDDE.8000604@zioup.com>	<4A37719E.3010006@stroeder.com>	<4A37A5D9.4040901@zioup.com>	<4A3830A6.4030407@eb2bcom.com>	<93053DE7-C324-4124-BF8F-B3C7088D66EB@Isode.com> <4A39E563.1040107@eb2bcom.com>
In-Reply-To: <4A39E563.1040107@eb2bcom.com>
X-Enigmail-Version: 0.95.7
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Cc: ldapext@ietf.org
Subject: Re: [ldapext] UTF-8 full support in LDIF / LDIF v2
X-BeenThere: ldapext@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: LDAP Extension Working Group <ldapext.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/ldapext>, <mailto:ldapext-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/ldapext>
List-Post: <mailto:ldapext@ietf.org>
List-Help: <mailto:ldapext-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/ldapext>, <mailto:ldapext-request@ietf.org?subject=subscribe>
X-List-Received-Date: Thu, 18 Jun 2009 10:15:10 -0000

Steven Legg wrote:
> Kurt Zeilenga wrote:
>> On Jun 16, 2009, at 4:54 PM, Steven Legg wrote:
>>> Also, X.500 directories already lose something in the translation
>>> when outputting as LDIF.  For example, the choice in a
>>> DirectoryString is lost and if that choice is teletexString then
>>> transcoding wipes out the exact octet encoding. Most of the
>>> ad-hoc LDAP string encodings are lossy in some respect. Such
>>> changes are tolerable because the resulting value in LDIF is the 
>>> same as far as the matching rules are concerned. Unicode
>>> normalization of the extended LDIF output is a similar situation.
>>
>> The loss you are talking about is inherent in LDAP not LDIF.  That is,
>> LDIF does not lose anything (for the LDAP requests it's design to
>> represent) in translation to/from LDAP.  I don't it is tolerable for
>> an LDAP intermediate format to "lose" LDAP information.
> 
> A "here" document mechanism where the UTF-8 character sequence between
> the end of the introducer and the beginning of the terminator is the
> literal directory attribute value without any modification would satisfy
> the no loss requirement.

Steven, like Kurt I cannot see how the loss of the choice in a
DirectoryString stored in a X.500 directory is prevented by the "here"
document mechanism since it's simply already lost when retrieving the
DirectoryString attribute via LDAP. (Well, one might think about
inventing a scheme with attribute sub-types to keep the choice...)

Ciao, Michael.

From michael@stroeder.com  Thu Jun 18 03:31:41 2009
Return-Path: <michael@stroeder.com>
X-Original-To: ldapext@core3.amsl.com
Delivered-To: ldapext@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 9A8C83A6AE1 for <ldapext@core3.amsl.com>; Thu, 18 Jun 2009 03:31:41 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.147
X-Spam-Level: 
X-Spam-Status: No, score=-2.147 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, MIME_8BIT_HEADER=0.3, SARE_SUB_ENC_UTF8=0.152]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id oW06QKv242PS for <ldapext@core3.amsl.com>; Thu, 18 Jun 2009 03:31:40 -0700 (PDT)
Received: from srv1.stroeder.com (srv1.stroeder.com [213.240.180.113]) by core3.amsl.com (Postfix) with ESMTP id 925A93A6A67 for <ldapext@ietf.org>; Thu, 18 Jun 2009 03:31:40 -0700 (PDT)
Received: from localhost (localhost [127.0.0.1]) by srv1.stroeder.com (Postfix) with ESMTP id E44934E07A; Thu, 18 Jun 2009 12:31:29 +0200 (CEST)
X-Virus-Scanned: amavisd-new at stroeder.com
Received: from srv1.stroeder.com ([127.0.0.1]) by localhost (srv1.stroeder.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id z6IQeIL+zmld; Thu, 18 Jun 2009 12:31:27 +0200 (CEST)
Received: from [10.1.0.2] (unknown [10.1.0.2]) by srv1.stroeder.com (Postfix) with ESMTP id 087534E06D; Thu, 18 Jun 2009 12:31:27 +0200 (CEST)
Message-ID: <4A3A1792.60109@stroeder.com>
Date: Thu, 18 Jun 2009 12:31:46 +0200
From: =?ISO-8859-1?Q?Michael_Str=F6der?= <michael@stroeder.com>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.22) Gecko/20090606 SeaMonkey/1.1.17
MIME-Version: 1.0
To: Howard Chu <hyc@highlandsun.com>
References: <49C497F9.7010200@zioup.com>	<CD3905D4-2A25-4C56-8187-3CE10D46C929@isode.com>	<49C870C6.4010803@zioup.com>	<E94B7389-9A6D-4CB6-BB2C-649CCD3FD15B@Isode.com>	<49CB192E.5050105@zioup.com>	<49CB211C.6070108@eb2bcom.com>	<49CB87FE.1050809@zioup.com>	<49CC01DE.6040506@eb2bcom.com>	<4A24557D.7030006@zioup.com>	<4A26A05D.8040105@zioup.com>	<245BF18B-2066-4E36-9502-16F4A3140D9E@Isode.com>	<4A309775.3080406@zioup.com>	<4A311ED1.1030202@stroeder.com>	<4A31D27B.3090208@zioup.com>	<35B2A165-CE5D-4650-AADE-CC233F71470E@Isode.com>	<4A35D23D.5040307@zioup.com>	<D437E784-4198-4037-A4EA-0300439C3D2C@Isode.com>	<4A37BCEB.5040103@zioup.com>	<68380E97-521C-4A80-A569-D09F8F626F6F@Isode.com>	<C6C69FB4-E767-4A18-89F6-B070E88DA82F@Isode.com> <4A37F352.1000600@highlandsun.com>
In-Reply-To: <4A37F352.1000600@highlandsun.com>
X-Enigmail-Version: 0.95.7
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Cc: ldapext@ietf.org
Subject: Re: [ldapext] UTF-8 full support in LDIF / LDIF v2
X-BeenThere: ldapext@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: LDAP Extension Working Group <ldapext.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/ldapext>, <mailto:ldapext-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/ldapext>
List-Post: <mailto:ldapext@ietf.org>
List-Help: <mailto:ldapext-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/ldapext>, <mailto:ldapext-request@ietf.org?subject=subscribe>
X-List-Received-Date: Thu, 18 Jun 2009 10:31:41 -0000

Howard Chu wrote:
> Kurt Zeilenga wrote:
>> By raising these points, I hope to show that simply removing the ASCII
>> restrictions will lead to problems, and even though some of them (such
>> as invalid UTF-8) can be address by installing various restrictions,
>> the simple fact that LDIF represents LDAP data not text will lead to
>> various problems (such as unintended text file conversions, inability
>> to use text processing programs, etc.).
>> -- Kurt
> 
> Agreed. Not to mention that to implement this properly will require
> complete schema knowledge at the time the LDIF is generated. (Otherwise,
> how do you distinguish a genuine octetString value, which cannot be
> safely represented in UTF-8, from a directoryString value...)

Well, in the real world applications are already applying schema
knowledge to the LDAP/LDIF data produced or parsed. But not in the
formal way of a DSA or DUA using information from the subschema
subentry. Rather it's more hard-coded a-priori schema knowledge.

A LDIF writer/reader module could be called with data passed in as
Unicode strings vs. binary buffers to let the application decide which
attribute values to treat as textual string (to be encoded as UTF-8) and
which to treat as blob which needs base64-encoding.

Ciao, Michael.

From michael@stroeder.com  Thu Jun 18 03:57:03 2009
Return-Path: <michael@stroeder.com>
X-Original-To: ldapext@core3.amsl.com
Delivered-To: ldapext@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id C3AC228C120 for <ldapext@core3.amsl.com>; Thu, 18 Jun 2009 03:57:03 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.147
X-Spam-Level: 
X-Spam-Status: No, score=-2.147 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, MIME_8BIT_HEADER=0.3, SARE_SUB_ENC_UTF8=0.152]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id lxIdYSony8ER for <ldapext@core3.amsl.com>; Thu, 18 Jun 2009 03:57:02 -0700 (PDT)
Received: from srv1.stroeder.com (srv1.stroeder.com [213.240.180.113]) by core3.amsl.com (Postfix) with ESMTP id 619A728C122 for <ldapext@ietf.org>; Thu, 18 Jun 2009 03:57:02 -0700 (PDT)
Received: from localhost (localhost [127.0.0.1]) by srv1.stroeder.com (Postfix) with ESMTP id 33E274E080; Thu, 18 Jun 2009 12:56:49 +0200 (CEST)
X-Virus-Scanned: amavisd-new at stroeder.com
Received: from srv1.stroeder.com ([127.0.0.1]) by localhost (srv1.stroeder.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id f-mczaFjS8MQ; Thu, 18 Jun 2009 12:56:46 +0200 (CEST)
Received: from [10.1.0.2] (unknown [10.1.0.2]) by srv1.stroeder.com (Postfix) with ESMTP id 765D64E07F; Thu, 18 Jun 2009 12:56:45 +0200 (CEST)
Message-ID: <4A3A1D81.1000407@stroeder.com>
Date: Thu, 18 Jun 2009 12:57:05 +0200
From: =?windows-1252?Q?Michael_Str=F6der?= <michael@stroeder.com>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.22) Gecko/20090606 SeaMonkey/1.1.17
MIME-Version: 1.0
To: Kurt Zeilenga <Kurt.Zeilenga@Isode.com>
References: <49C497F9.7010200@zioup.com>	<CD3905D4-2A25-4C56-8187-3CE10D46C929@isode.com>	<49C870C6.4010803@zioup.com>	<E94B7389-9A6D-4CB6-BB2C-649CCD3FD15B@Isode.com>	<49CB192E.5050105@zioup.com>	<49CB211C.6070108@eb2bcom.com>	<49CB87FE.1050809@zioup.com>	<49CC01DE.6040506@eb2bcom.com>	<4A24557D.7030006@zioup.com>	<4A26A05D.8040105@zioup.com>	<245BF18B-2066-4E36-9502-16F4A3140D9E@Isode.com>	<4A309775.3080406@zioup.com> <4A311ED1.1030202@stroeder.com>	<4A31D27B.3090208@zioup.com>	<35B2A165-CE5D-4650-AADE-CC233F71470E@Isode.com>	<4A35D23D.5040307@zioup.com>	<D437E784-4198-4037-A4EA-0300439C3D2C@Isode.com>	<4A37BCEB.5040103@zioup.com> <68380E97-521C-4A80-A569-D09F8F626F6F@Isode.com>
In-Reply-To: <68380E97-521C-4A80-A569-D09F8F626F6F@Isode.com>
X-Enigmail-Version: 0.95.7
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: 7bit
Cc: ldapext@ietf.org
Subject: Re: [ldapext] UTF-8 full support in LDIF / LDIF v2
X-BeenThere: ldapext@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: LDAP Extension Working Group <ldapext.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/ldapext>, <mailto:ldapext-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/ldapext>
List-Post: <mailto:ldapext@ietf.org>
List-Help: <mailto:ldapext-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/ldapext>, <mailto:ldapext-request@ietf.org?subject=subscribe>
X-List-Received-Date: Thu, 18 Jun 2009 10:57:03 -0000

Kurt Zeilenga wrote:
> 
> IDNA when through all of this.  They found that they had to place
> significant restrictions on Unicode domain components to ensure that a
> domain name was well-formed Unicode text.

I'd like to learn more about the term "well-formed Unicode text". Do you
have a reference at hand? [NAMEPREP] and/or [STRINGPREP]?

I found

http://www.unicode.org/versions/Unicode5.1.0/#Conformance_Changes

which contains a replacement for the text in Unicode5.0 standard.
(Strange that one cannot simply download the recent version.)

> You have not suggested placing similar restrictions on LDIF but simply
> removing the ASCII restriction.

Would it help to define similar restrictions?

Stepping back a bit from the details of the interesting Unicode issues
posted here I wonder what the general strategy of the IETF regarding
these issues is? I remember discussions on the ietf-pkix mailing list
mentioning problems like these (e.g. when displaying subject names of
X.509 certs) without any real solution.

I think any system which takes (user) input, decodes it to a Unicode
code point sequence and display it to the user is affected by issues
with BIDI, combining characters and duplicate Unicode points.

Ciao, Michael.

From michael@stroeder.com  Thu Jun 18 03:59:25 2009
Return-Path: <michael@stroeder.com>
X-Original-To: ldapext@core3.amsl.com
Delivered-To: ldapext@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 4A1193A6CB1 for <ldapext@core3.amsl.com>; Thu, 18 Jun 2009 03:59:25 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.147
X-Spam-Level: 
X-Spam-Status: No, score=-2.147 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, MIME_8BIT_HEADER=0.3, SARE_SUB_ENC_UTF8=0.152]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 6Ca0RPGySJ79 for <ldapext@core3.amsl.com>; Thu, 18 Jun 2009 03:59:24 -0700 (PDT)
Received: from srv1.stroeder.com (srv1.stroeder.com [213.240.180.113]) by core3.amsl.com (Postfix) with ESMTP id 624E23A6C9E for <ldapext@ietf.org>; Thu, 18 Jun 2009 03:59:24 -0700 (PDT)
Received: from localhost (localhost [127.0.0.1]) by srv1.stroeder.com (Postfix) with ESMTP id 09C8B4E080; Thu, 18 Jun 2009 12:59:07 +0200 (CEST)
X-Virus-Scanned: amavisd-new at stroeder.com
Received: from srv1.stroeder.com ([127.0.0.1]) by localhost (srv1.stroeder.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id xvEX5MWzUOMV; Thu, 18 Jun 2009 12:59:04 +0200 (CEST)
Received: from [10.1.0.2] (unknown [10.1.0.2]) by srv1.stroeder.com (Postfix) with ESMTP id 096FD4E07F; Thu, 18 Jun 2009 12:59:03 +0200 (CEST)
Message-ID: <4A3A1E0B.4090408@stroeder.com>
Date: Thu, 18 Jun 2009 12:59:23 +0200
From: =?UTF-8?B?TWljaGFlbCBTdHLDtmRlcg==?= <michael@stroeder.com>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.22) Gecko/20090606 SeaMonkey/1.1.17
MIME-Version: 1.0
To: Kurt Zeilenga <Kurt.Zeilenga@Isode.com>
References: <49C497F9.7010200@zioup.com>	<CD3905D4-2A25-4C56-8187-3CE10D46C929@isode.com>	<49C870C6.4010803@zioup.com>	<E94B7389-9A6D-4CB6-BB2C-649CCD3FD15B@Isode.com>	<49CB192E.5050105@zioup.com>	<49CB211C.6070108@eb2bcom.com>	<49CB87FE.1050809@zioup.com>	<49CC01DE.6040506@eb2bcom.com>	<4A24557D.7030006@zioup.com>	<4A26A05D.8040105@zioup.com>	<245BF18B-2066-4E36-9502-16F4A3140D9E@Isode.com>	<4A309775.3080406@zioup.com>	<4A311ED1.1030202@stroeder.com>	<4A31D27B.3090208@zioup.com>	<35B2A165-CE5D-4650-AADE-CC233F71470E@Isode.com>	<4A35D23D.5040307@zioup.com>	<D437E784-4198-4037-A4EA-0300439C3D2C@Isode.com>	<4A37BCEB.5040103@zioup.com>	<68380E97-521C-4A80-A569-D09F8F626F6F@Isode.com>	<C6C69FB4-E767-4A18-89F6-B070E88DA82F@Isode.com>	<4A37F352.1000600@highlandsun.com> <C8753ADB-6E4A-4841-8E60-FA4B5B51D473@Isode.com>
In-Reply-To: <C8753ADB-6E4A-4841-8E60-FA4B5B51D473@Isode.com>
X-Enigmail-Version: 0.95.7
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Cc: ldapext@ietf.org
Subject: Re: [ldapext] UTF-8 full support in LDIF / LDIF v2
X-BeenThere: ldapext@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: LDAP Extension Working Group <ldapext.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/ldapext>, <mailto:ldapext-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/ldapext>
List-Post: <mailto:ldapext@ietf.org>
List-Help: <mailto:ldapext-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/ldapext>, <mailto:ldapext-request@ietf.org?subject=subscribe>
X-List-Received-Date: Thu, 18 Jun 2009 10:59:25 -0000

Kurt Zeilenga wrote:
> I've expanded my UTF-8 LDIF with some more goofiness.

I like it. ;-)

Someone else once posted this look-alike Unicode sequence to a mailing
list (ietf-pkix) during a similar discussion:

Міϲrοѕоft Іnс

(see example LDIF below)

But personally I wonder whether this is a general problem which needs to
be solved within UIs.

Ciao, Michael.

------------------------- snip -------------------------
dn:: z3QnNGWz7Jyzr/RldC+ZnQg0IZu0YEsb3U9VGVzdGluZyxkYz1zdHJvZWRlcixkYz1k
 ZQ==
description: Weird Unicode sample demonstrating potential of name faking
o:: 0JzRls+ycs6/0ZXQvmZ0INCGbtGB
objectClass: organization

------------------------- snip -------------------------

From Kurt.Zeilenga@Isode.com  Thu Jun 18 06:34:42 2009
Return-Path: <Kurt.Zeilenga@Isode.com>
X-Original-To: ldapext@core3.amsl.com
Delivered-To: ldapext@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id CE1EF3A691A for <ldapext@core3.amsl.com>; Thu, 18 Jun 2009 06:34:42 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.366
X-Spam-Level: 
X-Spam-Status: No, score=-2.366 tagged_above=-999 required=5 tests=[AWL=0.081,  BAYES_00=-2.599, SARE_SUB_ENC_UTF8=0.152]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id XRfCRBymnxTs for <ldapext@core3.amsl.com>; Thu, 18 Jun 2009 06:34:41 -0700 (PDT)
Received: from rufus.isode.com (rufus.isode.com [62.3.217.251]) by core3.amsl.com (Postfix) with ESMTP id 89BC93A67A3 for <ldapext@ietf.org>; Thu, 18 Jun 2009 06:34:41 -0700 (PDT)
Received: from [192.168.1.101] ((unknown) [75.141.233.128])  by rufus.isode.com (submission channel) via TCP with ESMTPSA  id <SjpCfAAh5CBi@rufus.isode.com>; Thu, 18 Jun 2009 14:34:52 +0100
X-SMTP-Protocol-Errors: NORDNS
Message-Id: <00B0D25E-1707-4A6E-8570-968D3D9AF5F0@Isode.com>
From: Kurt Zeilenga <Kurt.Zeilenga@Isode.com>
To: Steven Legg <steven.legg@eb2bcom.com>
In-Reply-To: <4A39E563.1040107@eb2bcom.com>
Date: Thu, 18 Jun 2009 06:34:50 -0700
References: <49C497F9.7010200@zioup.com> <CD3905D4-2A25-4C56-8187-3CE10D46C929@isode.com> <49C870C6.4010803@zioup.com> <E94B7389-9A6D-4CB6-BB2C-649CCD3FD15B@Isode.com> <49CB192E.5050105@zioup.com> <49CB211C.6070108@eb2bcom.com> <49CB87FE.1050809@zioup.com> <49CC01DE.6040506@eb2bcom.com> <4A24557D.7030006@zioup.com> <4A26A05D.8040105@zioup.com> <245BF18B-2066-4E36-9502-16F4A3140D9E@Isode.com> <4A309775.3080406@zioup.com> <4A311ED1.1030202@stroeder.com> <4A31D27B.3090208@zioup.com> <4A325A40.2050802@stroeder.com> <4A35CDDE.8000604@zioup.com> <4A37719E.3010006@stroeder.com> <4A37A5D9.4040901@zioup.com> <4A3830A6.4030407@eb2bcom.com> <93053DE7-C324-4124-BF8F-B3C7088D66EB@Isode.com> <4A39E563.1040107@eb2bcom.com>
X-Mailer: Apple Mail (2.935.3)
MIME-Version: 1.0
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Cc: ldapext@ietf.org
Subject: Re: [ldapext] UTF-8 full support in LDIF / LDIF v2
X-BeenThere: ldapext@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: LDAP Extension Working Group <ldapext.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/ldapext>, <mailto:ldapext-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/ldapext>
List-Post: <mailto:ldapext@ietf.org>
List-Help: <mailto:ldapext-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/ldapext>, <mailto:ldapext-request@ietf.org?subject=subscribe>
X-List-Received-Date: Thu, 18 Jun 2009 13:34:42 -0000

On Jun 17, 2009, at 11:57 PM, Steven Legg wrote:

>
> Kurt,
>
> Kurt Zeilenga wrote:
>> On Jun 16, 2009, at 4:54 PM, Steven Legg wrote:
>>> Also, X.500 directories already lose something in the translation  
>>> when outputting
>>> as LDIF.  For example, the choice in a DirectoryString is lost and  
>>> if that
>>> choice is teletexString then transcoding wipes out the exact octet  
>>> encoding.
>>> Most of the ad-hoc LDAP string encodings are lossy in some respect.
>>> Such changes are tolerable because the resulting value in LDIF is  
>>> the same
>>> as far as the matching rules are concerned. Unicode normalization  
>>> of the
>>> extended LDIF output is a similar situation.
>> The loss you are talking about is inherent in LDAP not LDIF.  That  
>> is, LDIF does not lose anything (for the LDAP requests it's design  
>> to represent) in translation to/from LDAP.  I don't it is tolerable  
>> for an LDAP intermediate format to "lose" LDAP information.

First, for loss, let's only talk about loss between the data format  
and LDAP requests.   Loss between the data format and DAP or some  
other protocol ought to be beyond the scope of an LDAP data format.   
Loss of the DirectoryString CHOICE is an LDAP issue, not a LDAP data  
format issue.

So when I respond to your reply, I'm only going to consider loss of  
information between the file and protocol representations of the LDAP  
request.

> A "here" document mechanism where the UTF-8 character sequence between
> the end of the introducer and the beginning of the terminator is the
> literal directory attribute value without any modification would  
> satisfy
> the no loss requirement.

Depending on particulars of value and "here" mechanism.   For  
instance, line separators are problematic in the mechanism suggested  
by Yves.

> We could always recommend base64 encoding the
> entire file to transfer it across the network, or at least treating  
> it as
> a binary file, or suggest using LDIFv1 instead for such purposes.
>
> For me, the convenience of being able to easily view, edit or  
> compose the
> content in files

Have you tried to view/edit the UTF-8 file I sent out?

> I'm uploading outweighs the need to be careful about
> which text handling tools I use.

You will have to be quite careful in the tools you use.  If they are  
"text" tools, you'll have problems, because the LDIF file is not  
restricted to text.  And the tools encoding of values as text might be  
problematic (wrong normalization on a per value basis), etc..

For instance, any "text" tool which normalizes its output is generally  
unsuitable because that normalization will be incorrect for some set  
of values.

> If others have similar or overlapping
> requirements it would be better for us to use the same specification  
> even
> if it never becomes a proposed standard RFC.

I wouldn't object to experimenting here, just so long as it not  
confused with standard LDIF.

But if one is going to try to solve a particular problem within the  
standard community, they should try to solve it generally.  That is,  
we should dismiss solutions which only work well for limited subsets  
of Unicode.

>
> Would you feel better if we called it the LDAP Data Composition  
> Format(LDCF)
> instead of LDIFv2 or Extended LDIF ?

Well, if one was to have an experiment to develop a LDAP data format,  
which the whole content is say Net-Unicode, calling it something other  
than LDIF would be good to avoid confusing it with LDIF.  Things to  
tackle would include

- how to represent text values needing to be in different Unicode  
normalization (in the directory) than the required normalization,
- how to represent non-text Unicode values (arbitrary sequences of  
Unicode points),
- if Net-Unicode line separators are to be considered part of the  
value, how to represent values requiring other line separators, and
- bidirectional values.

Note: not requiring a particular normalization in the file is not a  
solution to first item, as that merely leaves it up to tools as to  
which normalization (if any) to apply.  So the first item just  
becomes, how to convey the normalization algorithm required for the  
value when represented in the LDAP request.

These are not simple problems and are unlikely to be solved well on  
the first publication, hence experimental track does seem appropriate  
for this feature.

I would like LDIF standard extensions work to be limited to improving  
data interchange capabilities (such as representation of LDAP  
responses), which is a much more straight forward problem, something  
we are far more likely to get right on the first publication.

-- Kurt

From Kurt.Zeilenga@Isode.com  Thu Jun 18 07:07:23 2009
Return-Path: <Kurt.Zeilenga@Isode.com>
X-Original-To: ldapext@core3.amsl.com
Delivered-To: ldapext@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id BD6B628C32C for <ldapext@core3.amsl.com>; Thu, 18 Jun 2009 07:07:23 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.221
X-Spam-Level: 
X-Spam-Status: No, score=-2.221 tagged_above=-999 required=5 tests=[AWL=-0.074, BAYES_00=-2.599, MIME_8BIT_HEADER=0.3, SARE_SUB_ENC_UTF8=0.152]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ya3i-wNpVWr5 for <ldapext@core3.amsl.com>; Thu, 18 Jun 2009 07:07:22 -0700 (PDT)
Received: from rufus.isode.com (rufus.isode.com [62.3.217.251]) by core3.amsl.com (Postfix) with ESMTP id 6457A28C319 for <ldapext@ietf.org>; Thu, 18 Jun 2009 07:07:22 -0700 (PDT)
Received: from [192.168.1.101] ((unknown) [75.141.233.128])  by rufus.isode.com (submission channel) via TCP with ESMTPSA  id <SjpKJAAh5K43@rufus.isode.com>; Thu, 18 Jun 2009 15:07:33 +0100
X-SMTP-Protocol-Errors: NORDNS
Message-Id: <32A3DDC8-FC80-490B-89A2-71ADC9582190@Isode.com>
From: Kurt Zeilenga <Kurt.Zeilenga@Isode.com>
To: =?ISO-8859-1?Q?Michael_Str=F6der?= <michael@stroeder.com>
In-Reply-To: <4A3A1D81.1000407@stroeder.com>
Date: Thu, 18 Jun 2009 07:07:30 -0700
References: <49C497F9.7010200@zioup.com> <CD3905D4-2A25-4C56-8187-3CE10D46C929@isode.com> <49C870C6.4010803@zioup.com> <E94B7389-9A6D-4CB6-BB2C-649CCD3FD15B@Isode.com> <49CB192E.5050105@zioup.com> <49CB211C.6070108@eb2bcom.com> <49CB87FE.1050809@zioup.com> <49CC01DE.6040506@eb2bcom.com> <4A24557D.7030006@zioup.com> <4A26A05D.8040105@zioup.com> <245BF18B-2066-4E36-9502-16F4A3140D9E@Isode.com> <4A309775.3080406@zioup.com> <4A311ED1.1030202@stroeder.com> <4A31D27B.3090208@zioup.com> <35B2A165-CE5D-4650-AADE-CC233F71470E@Isode.com> <4A35D23D.5040307@zioup.com> <D437E784-4198-4037-A4EA-0300439C3D2C@Isode.com> <4A37BCEB.5040103@zioup.com> <68380E97-521C-4A80-A569-D09F8F626F6F@Isode.com> <4A3A1D81.1000407@stroeder.com>
X-Mailer: Apple Mail (2.935.3)
MIME-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1; format=flowed; delsp=yes
Content-Transfer-Encoding: quoted-printable
Cc: ldapext@ietf.org
Subject: Re: [ldapext] UTF-8 full support in LDIF / LDIF v2
X-BeenThere: ldapext@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: LDAP Extension Working Group <ldapext.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/ldapext>, <mailto:ldapext-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/ldapext>
List-Post: <mailto:ldapext@ietf.org>
List-Help: <mailto:ldapext-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/ldapext>, <mailto:ldapext-request@ietf.org?subject=subscribe>
X-List-Received-Date: Thu, 18 Jun 2009 14:07:23 -0000

On Jun 18, 2009, at 3:57 AM, Michael Str=F6der wrote:

> Kurt Zeilenga wrote:
>>
>> IDNA when through all of this.  They found that they had to place
>> significant restrictions on Unicode domain components to ensure =20
>> that a
>> domain name was well-formed Unicode text.
>
> I'd like to learn more about the term "well-formed Unicode text". Do =20=

> you
> have a reference at hand? [NAMEPREP] and/or [STRINGPREP]?

Unfortunately, I don't have a good reference handy.  The Unicode =20
community might actually use different terms.  I tried to loosely =20
define the terms in a prior email.  (I'm not a Unicode expert, just =20
someone who's been involved Unicode issues (such as with IDNA) for a =20
number of years.)

To rephrase MY definitions:

"text" merely implies that the sequence of Unicode code points =20
represents a character.  In my ldif example, there is a colon followed =20=

by a combining code point.  This is an example of a sequence which =20
doesn't represent "text".

"well-formed text" implies that not only is the sequence is "text" but =20=

that various other rules are met.  For instance, the sequence will =20
result in proper directional display of bidirectional text.  There are =20=

some examples in the LDIF which show that introduction of line =20
wrapping can break the directional display of the value.


>
> I found
>
> http://www.unicode.org/versions/Unicode5.1.0/#Conformance_Changes
>
> which contains a replacement for the text in Unicode5.0 standard.
> (Strange that one cannot simply download the recent version.)

You can download each chapter of the current version (each has a front =20=

page detailing copying restrictions, etc.).

>
>> You have not suggested placing similar restrictions on LDIF but =20
>> simply
>> removing the ASCII restriction.
>
> Would it help to define similar restrictions?

First, I don't see how any of this helps in LDAP data interchange, the =20=

primary purpose of LDIF.

Second, if one were to say that the resulting file has to be Net-=20
Unicode (which I think at least means the file is "text"), you run =20
into "data loss" problems due to unintended transformations.

> Stepping back a bit from the details of the interesting Unicode issues
> posted here I wonder what the general strategy of the IETF regarding
> these issues is?

Punt.

> I remember discussions on the ietf-pkix mailing list
> mentioning problems like these (e.g. when displaying subject names of
> X.509 certs) without any real solution.
>
> I think any system which takes (user) input, decodes it to a Unicode
> code point sequence and display it to the user is affected by issues
> with BIDI, combining characters and duplicate Unicode points.

Yes.  The IETF tends to punt such issues to the user interface =20
development community.  The IETF tends to restrict itself to design of =20=

protocols not design user interface (though the IETF does try to =20
document user interface issues, especially those with security impact).

I think of LDIF as an alternative encoding of protocol data units, =20
used for out-of-band transfer data between protocol peers.  That is, I =20=

punt the "user" as far as I can.

Others see LDIF as a user display format and user input format for =20
LDAP data.  I argue that LDIFv1 didn't handle this well for ASCII and =20=

that handling this for ASCII (without data loss) is hard.  Solving it =20=

for Unicode, well, that's very, very hard.

-- Kurt=

From steven.legg@eb2bcom.com  Thu Jun 18 18:19:08 2009
Return-Path: <steven.legg@eb2bcom.com>
X-Original-To: ldapext@core3.amsl.com
Delivered-To: ldapext@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id B16323A69BC for <ldapext@core3.amsl.com>; Thu, 18 Jun 2009 18:19:08 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: 0.528
X-Spam-Level: 
X-Spam-Status: No, score=0.528 tagged_above=-999 required=5 tests=[AWL=-0.150,  BAYES_00=-2.599, FH_HOST_EQ_D_D_D_D=0.765, FH_HOST_EQ_D_D_D_DB=0.888,  HOST_EQ_STATIC=1.172, MIME_8BIT_HEADER=0.3, SARE_SUB_ENC_UTF8=0.152]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id rR86Rj952rlG for <ldapext@core3.amsl.com>; Thu, 18 Jun 2009 18:19:08 -0700 (PDT)
Received: from eb2bcom.com (165.203.232.72.static.reverse.ltdomains.com [72.232.203.165]) by core3.amsl.com (Postfix) with ESMTP id 0D81A3A681E for <ldapext@ietf.org>; Thu, 18 Jun 2009 18:18:55 -0700 (PDT)
Received: from [202.164.192.219] (helo=[10.1.2.237]) by host.eb2bcom.com with esmtpa (Exim 4.69) (envelope-from <steven.legg@eb2bcom.com>) id 1MHSl0-0005aA-RK; Fri, 19 Jun 2009 11:19:07 +1000
Message-ID: <4A3AE785.5010509@eb2bcom.com>
Date: Fri, 19 Jun 2009 11:19:01 +1000
From: Steven Legg <steven.legg@eb2bcom.com>
User-Agent: Thunderbird 2.0.0.21 (Windows/20090302)
MIME-Version: 1.0
To: =?ISO-8859-1?Q?Michael_Str=F6der?= <michael@stroeder.com>
References: <49C497F9.7010200@zioup.com>	<CD3905D4-2A25-4C56-8187-3CE10D46C929@isode.com>	<49C870C6.4010803@zioup.com>	<E94B7389-9A6D-4CB6-BB2C-649CCD3FD15B@Isode.com>	<49CB192E.5050105@zioup.com>	<49CB211C.6070108@eb2bcom.com>	<49CB87FE.1050809@zioup.com>	<49CC01DE.6040506@eb2bcom.com>	<4A24557D.7030006@zioup.com>	<4A26A05D.8040105@zioup.com>	<245BF18B-2066-4E36-9502-16F4A3140D9E@Isode.com>	<4A309775.3080406@zioup.com>	<4A311ED1.1030202@stroeder.com>	<4A31D27B.3090208@zioup.com>	<4A325A40.2050802@stroeder.com>	<4A35CDDE.8000604@zioup.com>	<4A37719E.3010006@stroeder.com>	<4A37A5D9.4040901@zioup.com>	<4A3830A6.4030407@eb2bcom.com>	<93053DE7-C324-4124-BF8F-B3C7088D66EB@Isode.com> <4A39E563.1040107@eb2bcom.com> <4A3A13B2.6060307@stroeder.com>
In-Reply-To: <4A3A13B2.6060307@stroeder.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 8bit
X-AntiAbuse: This header was added to track abuse, please include it with any abuse report
X-AntiAbuse: Primary Hostname - host.eb2bcom.com
X-AntiAbuse: Original Domain - ietf.org
X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12]
X-AntiAbuse: Sender Address Domain - eb2bcom.com
X-Source: 
X-Source-Args: 
X-Source-Dir: 
Cc: ldapext@ietf.org
Subject: Re: [ldapext] UTF-8 full support in LDIF / LDIF v2
X-BeenThere: ldapext@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: LDAP Extension Working Group <ldapext.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/ldapext>, <mailto:ldapext-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/ldapext>
List-Post: <mailto:ldapext@ietf.org>
List-Help: <mailto:ldapext-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/ldapext>, <mailto:ldapext-request@ietf.org?subject=subscribe>
X-List-Received-Date: Fri, 19 Jun 2009 01:19:08 -0000

Michael,

Michael Str�der wrote:
> Steven Legg wrote:
>> Kurt Zeilenga wrote:
>>> On Jun 16, 2009, at 4:54 PM, Steven Legg wrote:
>>>> Also, X.500 directories already lose something in the translation
>>>> when outputting as LDIF.  For example, the choice in a
>>>> DirectoryString is lost and if that choice is teletexString then
>>>> transcoding wipes out the exact octet encoding. Most of the
>>>> ad-hoc LDAP string encodings are lossy in some respect. Such
>>>> changes are tolerable because the resulting value in LDIF is the 
>>>> same as far as the matching rules are concerned. Unicode
>>>> normalization of the extended LDIF output is a similar situation.
>>> The loss you are talking about is inherent in LDAP not LDIF.  That is,
>>> LDIF does not lose anything (for the LDAP requests it's design to
>>> represent) in translation to/from LDAP.  I don't it is tolerable for
>>> an LDAP intermediate format to "lose" LDAP information.
>> A "here" document mechanism where the UTF-8 character sequence between
>> the end of the introducer and the beginning of the terminator is the
>> literal directory attribute value without any modification would satisfy
>> the no loss requirement.
> 
> Steven, like Kurt I cannot see how the loss of the choice in a
> DirectoryString stored in a X.500 directory is prevented by the "here"
> document mechanism

It isn't. My comment was in the context of LDAP to LDAP interchange.

 > since it's simply already lost when retrieving the
> DirectoryString attribute via LDAP. (Well, one might think about
> inventing a scheme with attribute sub-types to keep the choice...)

There is a scheme already. I could create LDIF dumps where every attribute
of DirectoryString syntax is output with ";binary", which would necessarily
cause the values to be base64 encoded. However, I don't think there are
many directory implementations that could cope with that so I accept that
X.500 to LDAP to LDIF is essentially lossy (but not in any way that has
proven to be critical). Consequently I'm not too concerned if LDAP to LDIFv2
isn't perfect either, though the "here" mechanism described above does at
least provide for lossless LDAP to LDIFv2 conversion.

The potential for an inadvertent change of normalization in the LDIFv2 if
it is edited doesn't overly concern me. Stringprep takes care of it for
matching purposes and any client that expects attribute values to be in,
or remain in, a particular normalization form is asking for trouble. The
values could be modified by some other client that changes the normalization
during editing and I wouldn't count on every directory implementation
preserving the exact character sequence it is given (though mine does).
If a client needs the values to be in a particular normalization form it
should do the conversion itself.

Regards,
Steven

> 
> Ciao, Michael.

From steven.legg@eb2bcom.com  Thu Jun 18 18:20:15 2009
Return-Path: <steven.legg@eb2bcom.com>
X-Original-To: ldapext@core3.amsl.com
Delivered-To: ldapext@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id BC8593A69A2 for <ldapext@core3.amsl.com>; Thu, 18 Jun 2009 18:20:15 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: 0.453
X-Spam-Level: 
X-Spam-Status: No, score=0.453 tagged_above=-999 required=5 tests=[AWL=0.075,  BAYES_00=-2.599, FH_HOST_EQ_D_D_D_D=0.765, FH_HOST_EQ_D_D_D_DB=0.888,  HOST_EQ_STATIC=1.172, SARE_SUB_ENC_UTF8=0.152]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id MYULSBcPfMKz for <ldapext@core3.amsl.com>; Thu, 18 Jun 2009 18:20:14 -0700 (PDT)
Received: from eb2bcom.com (165.203.232.72.static.reverse.ltdomains.com [72.232.203.165]) by core3.amsl.com (Postfix) with ESMTP id 8936D3A67F1 for <ldapext@ietf.org>; Thu, 18 Jun 2009 18:20:14 -0700 (PDT)
Received: from [202.164.192.219] (helo=[10.1.2.237]) by host.eb2bcom.com with esmtpa (Exim 4.69) (envelope-from <steven.legg@eb2bcom.com>) id 1MHSmJ-0006Kj-8l; Fri, 19 Jun 2009 11:20:28 +1000
Message-ID: <4A3AE7D6.2090004@eb2bcom.com>
Date: Fri, 19 Jun 2009 11:20:22 +1000
From: Steven Legg <steven.legg@eb2bcom.com>
User-Agent: Thunderbird 2.0.0.21 (Windows/20090302)
MIME-Version: 1.0
To: Kurt Zeilenga <Kurt.Zeilenga@Isode.com>
References: <49C497F9.7010200@zioup.com> <CD3905D4-2A25-4C56-8187-3CE10D46C929@isode.com> <49C870C6.4010803@zioup.com> <E94B7389-9A6D-4CB6-BB2C-649CCD3FD15B@Isode.com> <49CB192E.5050105@zioup.com> <49CB211C.6070108@eb2bcom.com> <49CB87FE.1050809@zioup.com> <49CC01DE.6040506@eb2bcom.com> <4A24557D.7030006@zioup.com> <4A26A05D.8040105@zioup.com> <245BF18B-2066-4E36-9502-16F4A3140D9E@Isode.com> <4A309775.3080406@zioup.com> <4A311ED1.1030202@stroeder.com> <4A31D27B.3090208@zioup.com> <4A325A40.2050802@stroeder.com> <4A35CDDE.8000604@zioup.com> <4A37719E.3010006@stroeder.com> <4A37A5D9.4040901@zioup.com> <4A3830A6.4030407@eb2bcom.com> <93053DE7-C324-4124-BF8F-B3C7088D66EB@Isode.com> <4A39E563.1040107@eb2bcom.com> <00B0D25E-1707-4A6E-8570-968D3D9AF5F0@Isode.com>
In-Reply-To: <00B0D25E-1707-4A6E-8570-968D3D9AF5F0@Isode.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-AntiAbuse: This header was added to track abuse, please include it with any abuse report
X-AntiAbuse: Primary Hostname - host.eb2bcom.com
X-AntiAbuse: Original Domain - ietf.org
X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12]
X-AntiAbuse: Sender Address Domain - eb2bcom.com
X-Source: 
X-Source-Args: 
X-Source-Dir: 
Cc: ldapext@ietf.org
Subject: Re: [ldapext] UTF-8 full support in LDIF / LDIF v2
X-BeenThere: ldapext@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: LDAP Extension Working Group <ldapext.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/ldapext>, <mailto:ldapext-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/ldapext>
List-Post: <mailto:ldapext@ietf.org>
List-Help: <mailto:ldapext-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/ldapext>, <mailto:ldapext-request@ietf.org?subject=subscribe>
X-List-Received-Date: Fri, 19 Jun 2009 01:20:15 -0000

Kurt,

Kurt Zeilenga wrote:
> 
> On Jun 17, 2009, at 11:57 PM, Steven Legg wrote:
> 
>>
>> Kurt,
>>
>> Kurt Zeilenga wrote:
>>> On Jun 16, 2009, at 4:54 PM, Steven Legg wrote:
>>>> Also, X.500 directories already lose something in the translation 
>>>> when outputting
>>>> as LDIF.  For example, the choice in a DirectoryString is lost and 
>>>> if that
>>>> choice is teletexString then transcoding wipes out the exact octet 
>>>> encoding.
>>>> Most of the ad-hoc LDAP string encodings are lossy in some respect.
>>>> Such changes are tolerable because the resulting value in LDIF is 
>>>> the same
>>>> as far as the matching rules are concerned. Unicode normalization of 
>>>> the
>>>> extended LDIF output is a similar situation.
>>> The loss you are talking about is inherent in LDAP not LDIF.  That 
>>> is, LDIF does not lose anything (for the LDAP requests it's design to 
>>> represent) in translation to/from LDAP.  I don't it is tolerable for 
>>> an LDAP intermediate format to "lose" LDAP information.
> 
> First, for loss, let's only talk about loss between the data format and 
> LDAP requests.   Loss between the data format and DAP or some other 
> protocol ought to be beyond the scope of an LDAP data format.  Loss of 
> the DirectoryString CHOICE is an LDAP issue, not a LDAP data format issue.
> 
> So when I respond to your reply, I'm only going to consider loss of 
> information between the file and protocol representations of the LDAP 
> request.
> 
>> A "here" document mechanism where the UTF-8 character sequence between
>> the end of the introducer and the beginning of the terminator is the
>> literal directory attribute value without any modification would satisfy
>> the no loss requirement.
> 
> Depending on particulars of value and "here" mechanism.   For instance, 
> line separators are problematic in the mechanism suggested by Yves.

Which is why I don't like it and why I say above "the literal directory
attribute value". The directory attribute value may have no LDIF line
separators at all which means either ELDIF/LDIFv2/LDCF would have to allow
arbitrarily long "lines" in the "here" document mechanism (my preference)
or fallback to base64 if the value has any "lines" (in the LDIF sense,
which might not correspond to the natural lines in the value) that are
above some specific length.

> 
>> We could always recommend base64 encoding the
>> entire file to transfer it across the network, or at least treating it as
>> a binary file, or suggest using LDIFv1 instead for such purposes.
>>
>> For me, the convenience of being able to easily view, edit or compose the
>> content in files
> 
> Have you tried to view/edit the UTF-8 file I sent out?

Not until now. My LDIF editing tool of choice is vim which doesn't display the
non-ASCII characters sensibly, but that doesn't really matter to me. I'm more
interested in the structure of structured values in GSER or XML format. The
element and attribute names in an XML document are almost certainly in ASCII
and come from an international standard. The syntactically significant
punctuation is also in ASCII. I can see and manipulate the overall structure
of a complex value even if the contained character data is unreadable. If it's
not in English it doesn't mean much to me anyway so I have no particular
purpose in editing it. If it isn't a Latin script it's totally opaque to me
regardless of how vim displays it. What's important is that I can faithfully
cut and paste such unreadable values and write out the file without any
conversions taking place. If I did want to fiddle the character data, then I
would either do that through a directory client after uploading the data or
cut out the text, put it in a separate file, edit that with a proper text
editor then paste it back into the LDIFv2 using vim.

> 
>> I'm uploading outweighs the need to be careful about
>> which text handling tools I use.
> 
> You will have to be quite careful in the tools you use.  If they are 
> "text" tools, you'll have problems, because the LDIF file is not 
> restricted to text.  And the tools encoding of values as text might be 
> problematic (wrong normalization on a per value basis), etc..

Which is a problem with the tool rather than a problem with the file
representation.

> 
> For instance, any "text" tool which normalizes its output is generally 
> unsuitable because that normalization will be incorrect for some set of 
> values.

That problem will exist with LDAP clients doing updates as well. We use
stringprep to mitigate such problems, but the clients have to take some
responsibility for their expectations.

> 
>> If others have similar or overlapping
>> requirements it would be better for us to use the same specification even
>> if it never becomes a proposed standard RFC.
> 
> I wouldn't object to experimenting here, just so long as it not confused 
> with standard LDIF.
> 
> But if one is going to try to solve a particular problem within the 
> standard community, they should try to solve it generally.  That is, we 
> should dismiss solutions which only work well for limited subsets of 
> Unicode.
> 
>>
>> Would you feel better if we called it the LDAP Data Composition 
>> Format(LDCF)
>> instead of LDIFv2 or Extended LDIF ?
> 
> Well, if one was to have an experiment to develop a LDAP data format, 
> which the whole content is say Net-Unicode, calling it something other 
> than LDIF would be good to avoid confusing it with LDIF.  Things to 
> tackle would include
> 
> - how to represent text values needing to be in different Unicode 
> normalization (in the directory) than the required normalization,

That's also a problem for LDAP clients that we don't need to solve
for ELDIF/LDIFv2/LDCF.

> - how to represent non-text Unicode values (arbitrary sequences of 
> Unicode points),

Literally.

> - if Net-Unicode line separators are to be considered part of the value, 
> how to represent values requiring other line separators, and
> - bidirectional values.

I don't have a need for anything other than the literal UTF-8 attribute
value. It's simple, preserves the LDAP value, and doesn't face problems
we don't already have with normal directory access using LDAP clients.

Regards,
Steven

> 
> Note: not requiring a particular normalization in the file is not a 
> solution to first item, as that merely leaves it up to tools as to which 
> normalization (if any) to apply.  So the first item just becomes, how to 
> convey the normalization algorithm required for the value when 
> represented in the LDAP request.
> 
> These are not simple problems and are unlikely to be solved well on the 
> first publication, hence experimental track does seem appropriate for 
> this feature.
> 
> I would like LDIF standard extensions work to be limited to improving 
> data interchange capabilities (such as representation of LDAP 
> responses), which is a much more straight forward problem, something we 
> are far more likely to get right on the first publication.
> 
> -- Kurt

From yves@zioup.com  Thu Jun 18 21:00:20 2009
Return-Path: <yves@zioup.com>
X-Original-To: ldapext@core3.amsl.com
Delivered-To: ldapext@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 47DDE3A67E6 for <ldapext@core3.amsl.com>; Thu, 18 Jun 2009 21:00:20 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.164
X-Spam-Level: 
X-Spam-Status: No, score=-2.164 tagged_above=-999 required=5 tests=[AWL=0.283,  BAYES_00=-2.599, SARE_SUB_ENC_UTF8=0.152]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id MxTPXqCXDkkA for <ldapext@core3.amsl.com>; Thu, 18 Jun 2009 21:00:19 -0700 (PDT)
Received: from idcmail-mo2no.shaw.ca (idcmail-mo2no.shaw.ca [64.59.134.9]) by core3.amsl.com (Postfix) with ESMTP id 663333A677C for <ldapext@ietf.org>; Thu, 18 Jun 2009 21:00:19 -0700 (PDT)
Received: from pd6ml1no-ssvc.prod.shaw.ca ([10.0.153.160]) by pd5mo1no-svcs.prod.shaw.ca with ESMTP; 18 Jun 2009 21:59:54 -0600
X-Cloudmark-SP-Filtered: true
X-Cloudmark-SP-Result: v=1.0 c=0 a=J6b0bBEkAAAA:8 a=iZxe15jGPwG3X2mD_ugA:9 a=2Pun_oGEvZYe-HvHWO8A:7 a=gOYiypcaOD9sezF6xZlJKRl3E-sA:4 a=WPV3hXSGBlIA:10
Received: from unknown (HELO home.zioup.com) ([68.147.37.46]) by pd6ml1no-dmz.prod.shaw.ca with ESMTP; 18 Jun 2009 21:59:54 -0600
Received: from [192.168.2.202] (aristotle.zioup.net [192.168.2.202]) by home.zioup.com (Postfix) with ESMTP id 7FBE2C2F0 for <ldapext@ietf.org>; Thu, 18 Jun 2009 21:59:54 -0600 (MDT)
Message-ID: <4A3B0D3A.6080406@zioup.com>
Date: Thu, 18 Jun 2009 21:59:54 -0600
From: Yves Dorfsman <yves@zioup.com>
User-Agent: Thunderbird 2.0.0.21 (X11/20090409)
MIME-Version: 1.0
To: ldapext@ietf.org
References: <49C497F9.7010200@zioup.com>	<CD3905D4-2A25-4C56-8187-3CE10D46C929@isode.com>	<49C870C6.4010803@zioup.com>	<E94B7389-9A6D-4CB6-BB2C-649CCD3FD15B@Isode.com>	<49CB192E.5050105@zioup.com> <49CB211C.6070108@eb2bcom.com>	<49CB87FE.1050809@zioup.com> <49CC01DE.6040506@eb2bcom.com>	<4A24557D.7030006@zioup.com> <4A26A05D.8040105@zioup.com>	<245BF18B-2066-4E36-9502-16F4A3140D9E@Isode.com>	<4A309775.3080406@zioup.com> <4A311ED1.1030202@stroeder.com>	<4A31D27B.3090208@zioup.com> <4A325A40.2050802@stroeder.com>	<4A35CDDE.8000604@zioup.com> <4A37719E.3010006@stroeder.com>	<4A37A5D9.4040901@zioup.com> <4A3830A6.4030407@eb2bcom.com>	<93053DE7-C324-4124-BF8F-B3C7088D66EB@Isode.com>	<4A39E563.1040107@eb2bcom.com> <00B0D25E-1707-4A6E-8570-968D3D9AF5F0@Isode.com>
In-Reply-To: <00B0D25E-1707-4A6E-8570-968D3D9AF5F0@Isode.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Subject: Re: [ldapext] UTF-8 full support in LDIF / LDIF v2
X-BeenThere: ldapext@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: LDAP Extension Working Group <ldapext.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/ldapext>, <mailto:ldapext-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/ldapext>
List-Post: <mailto:ldapext@ietf.org>
List-Help: <mailto:ldapext-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/ldapext>, <mailto:ldapext-request@ietf.org?subject=subscribe>
X-List-Received-Date: Fri, 19 Jun 2009 04:00:20 -0000

Kurt Zeilenga wrote:
> 
>> If others have similar or overlapping
>> requirements it would be better for us to use the same specification even
>> if it never becomes a proposed standard RFC.
> 
> I wouldn't object to experimenting here, just so long as it not confused 
> with standard LDIF.
> 
> But if one is going to try to solve a particular problem within the 
> standard community, they should try to solve it generally.  That is, we 
> should dismiss solutions which only work well for limited subsets of 
> Unicode.
> 
>>
>> Would you feel better if we called it the LDAP Data Composition 
>> Format(LDCF)
>> instead of LDIFv2 or Extended LDIF ?
> 
> Well, if one was to have an experiment to develop a LDAP data format, 
> which the whole content is say Net-Unicode, calling it something other 
> than LDIF would be good to avoid confusing it with LDIF.  Things to 
> tackle would include
> 
> - how to represent text values needing to be in different Unicode 
> normalization (in the directory) than the required normalization,
> - how to represent non-text Unicode values (arbitrary sequences of 
> Unicode points),
> - if Net-Unicode line separators are to be considered part of the value, 
> how to represent values requiring other line separators, and
> - bidirectional values.
> 
> Note: not requiring a particular normalization in the file is not a 
> solution to first item, as that merely leaves it up to tools as to which 
> normalization (if any) to apply.  So the first item just becomes, how to 
> convey the normalization algorithm required for the value when 
> represented in the LDAP request.
> 
> These are not simple problems and are unlikely to be solved well on the 
> first publication, hence experimental track does seem appropriate for 
> this feature.
> 
> I would like LDIF standard extensions work to be limited to improving 
> data interchange capabilities (such as representation of LDAP 
> responses), which is a much more straight forward problem, something we 
> are far more likely to get right on the first publication.

Just when I was about to give up.... Anyway, if we go ahead with this, I am 
assuming we need to move the discussion to a different mailing list ?

Should it be an ietf mailing list, or should it happen somewhere else and 
come back here when it starts to take form ?

-- 
Yves.
http://www.sollers.ca/


From yves@zioup.com  Thu Jun 18 21:04:25 2009
Return-Path: <yves@zioup.com>
X-Original-To: ldapext@core3.amsl.com
Delivered-To: ldapext@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 5DBD23A68DB for <ldapext@core3.amsl.com>; Thu, 18 Jun 2009 21:04:25 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.193
X-Spam-Level: 
X-Spam-Status: No, score=-2.193 tagged_above=-999 required=5 tests=[AWL=0.254,  BAYES_00=-2.599, SARE_SUB_ENC_UTF8=0.152]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id NgSIxacZohO7 for <ldapext@core3.amsl.com>; Thu, 18 Jun 2009 21:04:24 -0700 (PDT)
Received: from idcmail-mo2no.shaw.ca (idcmail-mo2no.shaw.ca [64.59.134.9]) by core3.amsl.com (Postfix) with ESMTP id 6EF1C3A6869 for <ldapext@ietf.org>; Thu, 18 Jun 2009 21:04:24 -0700 (PDT)
Received: from pd6ml2no-ssvc.prod.shaw.ca ([10.0.153.163]) by pd5mo1no-svcs.prod.shaw.ca with ESMTP; 18 Jun 2009 22:04:37 -0600
X-Cloudmark-SP-Filtered: true
X-Cloudmark-SP-Result: v=1.0 c=0 a=J6b0bBEkAAAA:8 a=Q_JOdeXFmLuMA8WOGv8A:9 a=QCeCIzDgAEweb6yA0ZIA:7 a=vkeM06qa-uFLXdjh94TFmAykUusA:4 a=WPV3hXSGBlIA:10
Received: from unknown (HELO home.zioup.com) ([68.147.37.46]) by pd6ml2no-dmz.prod.shaw.ca with ESMTP; 18 Jun 2009 22:04:37 -0600
Received: from [192.168.2.202] (aristotle.zioup.net [192.168.2.202]) by home.zioup.com (Postfix) with ESMTP id 03C64C2F0 for <ldapext@ietf.org>; Thu, 18 Jun 2009 22:04:36 -0600 (MDT)
Message-ID: <4A3B0E54.6090808@zioup.com>
Date: Thu, 18 Jun 2009 22:04:36 -0600
From: Yves Dorfsman <yves@zioup.com>
User-Agent: Thunderbird 2.0.0.21 (X11/20090409)
MIME-Version: 1.0
To: ldapext@ietf.org
References: <49C497F9.7010200@zioup.com>	<CD3905D4-2A25-4C56-8187-3CE10D46C929@isode.com>	<49C870C6.4010803@zioup.com>	<E94B7389-9A6D-4CB6-BB2C-649CCD3FD15B@Isode.com>	<49CB192E.5050105@zioup.com> <49CB211C.6070108@eb2bcom.com>	<49CB87FE.1050809@zioup.com> <49CC01DE.6040506@eb2bcom.com>	<4A24557D.7030006@zioup.com> <4A26A05D.8040105@zioup.com>	<245BF18B-2066-4E36-9502-16F4A3140D9E@Isode.com>	<4A309775.3080406@zioup.com> <4A311ED1.1030202@stroeder.com>	<4A31D27B.3090208@zioup.com> <4A325A40.2050802@stroeder.com>	<4A35CDDE.8000604@zioup.com> <4A37719E.3010006@stroeder.com>	<4A37A5D9.4040901@zioup.com> <4A3830A6.4030407@eb2bcom.com>	<93053DE7-C324-4124-BF8F-B3C7088D66EB@Isode.com>	<4A39E563.1040107@eb2bcom.com>	<00B0D25E-1707-4A6E-8570-968D3D9AF5F0@Isode.com> <4A3AE7D6.2090004@eb2bcom.com>
In-Reply-To: <4A3AE7D6.2090004@eb2bcom.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Subject: Re: [ldapext] UTF-8 full support in LDIF / LDIF v2
X-BeenThere: ldapext@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: LDAP Extension Working Group <ldapext.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/ldapext>, <mailto:ldapext-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/ldapext>
List-Post: <mailto:ldapext@ietf.org>
List-Help: <mailto:ldapext-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/ldapext>, <mailto:ldapext-request@ietf.org?subject=subscribe>
X-List-Received-Date: Fri, 19 Jun 2009 04:04:25 -0000

Steven Legg wrote:
> 
> Not until now. My LDIF editing tool of choice is vim which doesn't 
> display the non-ASCII characters sensibly, but that doesn't really matter

For whatever it's worth, vim (and vim, not vi, not nvi) is able to display 
non-latin character, but, you have to:
-have the right font installed on your system
-your environment needs to be setup correctly (LANG needs to be set for 
en_CA.UTF8 for example

-enough pixel. On most setup, you get that by switching xterm to huge font 
(<ctrl>, right click).

On the other hand, vim does not know how to display Right To Left text. 
Funnily enough, gedit handle all of it nicely (see the Ali Baba example I 
had sent a few months ago).


If anything this highlights what Kurt was saying, that in order to display / 
edit Unicode proper tools, editors that know how to, but it also highlights 
the fact that the people who need them tend to have the tools, and vice 
versa, if 7 bit ascii covers all your needs, then chances are you won't make 
a priority of moving to an environment that supports Unicode well, using new 
tools etc...

-- 
Yves.
http://www.sollers.ca/


From michael@stroeder.com  Fri Jun 19 00:40:09 2009
Return-Path: <michael@stroeder.com>
X-Original-To: ldapext@core3.amsl.com
Delivered-To: ldapext@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id E16493A6938 for <ldapext@core3.amsl.com>; Fri, 19 Jun 2009 00:40:09 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.147
X-Spam-Level: 
X-Spam-Status: No, score=-2.147 tagged_above=-999 required=5 tests=[AWL=-0.000, BAYES_00=-2.599, MIME_8BIT_HEADER=0.3, SARE_SUB_ENC_UTF8=0.152]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id V1hXGuo3FmyZ for <ldapext@core3.amsl.com>; Fri, 19 Jun 2009 00:40:09 -0700 (PDT)
Received: from srv1.stroeder.com (srv1.stroeder.com [213.240.180.113]) by core3.amsl.com (Postfix) with ESMTP id 051F43A679C for <ldapext@ietf.org>; Fri, 19 Jun 2009 00:40:08 -0700 (PDT)
Received: from localhost (localhost [127.0.0.1]) by srv1.stroeder.com (Postfix) with ESMTP id 8C15B4E076; Fri, 19 Jun 2009 09:39:42 +0200 (CEST)
X-Virus-Scanned: amavisd-new at stroeder.com
Received: from srv1.stroeder.com ([127.0.0.1]) by localhost (srv1.stroeder.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id JxXAOu0ladk6; Fri, 19 Jun 2009 09:39:39 +0200 (CEST)
Received: from [10.1.0.2] (unknown [10.1.0.2]) by srv1.stroeder.com (Postfix) with ESMTP id ACA1A4E06F; Fri, 19 Jun 2009 09:39:39 +0200 (CEST)
Message-ID: <4A3B40D1.7090106@stroeder.com>
Date: Fri, 19 Jun 2009 09:40:01 +0200
From: =?UTF-8?B?TWljaGFlbCBTdHLDtmRlcg==?= <michael@stroeder.com>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.22) Gecko/20090606 SeaMonkey/1.1.17
MIME-Version: 1.0
To: Yves Dorfsman <yves@zioup.com>
References: <49C497F9.7010200@zioup.com>	<CD3905D4-2A25-4C56-8187-3CE10D46C929@isode.com>	<49C870C6.4010803@zioup.com>	<E94B7389-9A6D-4CB6-BB2C-649CCD3FD15B@Isode.com>	<49CB192E.5050105@zioup.com>	<49CB211C.6070108@eb2bcom.com>	<49CB87FE.1050809@zioup.com>	<49CC01DE.6040506@eb2bcom.com>	<4A24557D.7030006@zioup.com>	<4A26A05D.8040105@zioup.com>	<245BF18B-2066-4E36-9502-16F4A3140D9E@Isode.com>	<4A309775.3080406@zioup.com>	<4A311ED1.1030202@stroeder.com>	<4A31D27B.3090208@zioup.com>	<35B2A165-CE5D-4650-AADE-CC233F71470E@Isode.com>	<4A35D23D.5040307@zioup.com>	<D437E784-4198-4037-A4EA-0300439C3D2C@Isode.com>	<4A37BCEB.5040103@zioup.com>	<68380E97-521C-4A80-A569-D09F8F626F6F@Isode.com>	<C6C69FB4-E767-4A18-89F6-B070E88DA82F@Isode.com>	<4A37F352.1000600@highlandsun.com>	<C8753ADB-6E4A-4841-8E60-FA4B5B51D473@Isode.com> <4A3A1E0B.4090408@stroeder.com> <4A3B0AB6.6070808@zioup.com>
In-Reply-To: <4A3B0AB6.6070808@zioup.com>
X-Enigmail-Version: 0.95.7
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Cc: ldapext@ietf.org
Subject: Re: [ldapext] UTF-8 full support in LDIF / LDIF v2
X-BeenThere: ldapext@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: LDAP Extension Working Group <ldapext.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/ldapext>, <mailto:ldapext-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/ldapext>
List-Post: <mailto:ldapext@ietf.org>
List-Help: <mailto:ldapext-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/ldapext>, <mailto:ldapext-request@ietf.org?subject=subscribe>
X-List-Received-Date: Fri, 19 Jun 2009 07:40:10 -0000

Yves Dorfsman wrote:
> Michael Ströder wrote:
>> Someone else once posted this look-alike Unicode sequence to a mailing
>> list (ietf-pkix) during a similar discussion:
>>
>> Міϲrοѕоft Іnс
>>
>> (see example LDIF below)
>>
>> But personally I wonder whether this is a general problem which needs to
>> be solved within UIs.
>>
>> ------------------------- snip -------------------------
>> dn:: z3QnNGWz7Jyzr/RldC+ZnQg0IZu0YEsb3U9VGVzdGluZyxkYz1zdHJvZWRlcixkYz1k
>>  ZQ==
>> description: Weird Unicode sample demonstrating potential of name faking
>> o:: 0JzRls+ycs6/0ZXQvmZ0INCGbtGB
>> objectClass: organization
>>
>> ------------------------- snip -------------------------
> 
> Michael, are you sure about your dn ?
> It does not seem to decode properly...

Sorry, messed it up in my MUA when manually editing line-breaks. ;-)

------------------------- snip -------------------------
dn:: bz3QnNGWz7Jyzr/RldC+ZnQg0IZu0YEsb3U9VGVzdGluZyxkYz1zdHJvZWRlcixkYz
 1kZQ==
description: Weird Unicode sample demonstrating potential of name faking
o:: 0JzRls+ycs6/0ZXQvmZ0INCGbtGB
objectClass: organization

------------------------- snip -------------------------

In Python:

>>> s="""bz3QnNGWz7Jyzr/RldC+ZnQg0IZu0YEsb3U9VGVzdGluZyxkYz1zdHJvZWRlcixkYz
...  1kZQ=="""
>>> s.decode('base64').decode('utf-8')
u'o=\u041c\u0456\u03f2r\u03bf\u0455\u043eft
\u0406n\u0441,ou=Testing,dc=stroeder
,dc=de'


Ciao, Michael.


From Kurt.Zeilenga@Isode.com  Mon Jun 22 18:23:17 2009
Return-Path: <Kurt.Zeilenga@Isode.com>
X-Original-To: ldapext@core3.amsl.com
Delivered-To: ldapext@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 14C1A3A6B20 for <ldapext@core3.amsl.com>; Mon, 22 Jun 2009 18:23:17 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.447
X-Spam-Level: 
X-Spam-Status: No, score=-2.447 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, SARE_SUB_ENC_UTF8=0.152]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id oR6TBchv8KqY for <ldapext@core3.amsl.com>; Mon, 22 Jun 2009 18:23:16 -0700 (PDT)
Received: from rufus.isode.com (rufus.isode.com [62.3.217.251]) by core3.amsl.com (Postfix) with ESMTP id 0BD283A68AD for <ldapext@ietf.org>; Mon, 22 Jun 2009 18:23:16 -0700 (PDT)
Received: from [172.16.2.150] (shiny.isode.com [62.3.217.250])  by rufus.isode.com (submission channel) via TCP with ESMTPSA  id <SkAujQBV9KDM@rufus.isode.com>; Tue, 23 Jun 2009 02:23:30 +0100
Message-Id: <0B741CDA-8A31-4A69-B272-BD58C43F097C@Isode.com>
From: Kurt Zeilenga <Kurt.Zeilenga@Isode.com>
To: Yves Dorfsman <yves@zioup.com>
In-Reply-To: <4A3B0D3A.6080406@zioup.com>
Date: Mon, 22 Jun 2009 18:23:23 -0700
References: <49C497F9.7010200@zioup.com> <CD3905D4-2A25-4C56-8187-3CE10D46C929@isode.com> <49C870C6.4010803@zioup.com> <E94B7389-9A6D-4CB6-BB2C-649CCD3FD15B@Isode.com> <49CB192E.5050105@zioup.com> <49CB211C.6070108@eb2bcom.com> <49CB87FE.1050809@zioup.com> <49CC01DE.6040506@eb2bcom.com> <4A24557D.7030006@zioup.com> <4A26A05D.8040105@zioup.com> <245BF18B-2066-4E36-9502-16F4A3140D9E@Isode.com> <4A309775.3080406@zioup.com> <4A311ED1.1030202@stroeder.com> <4A31D27B.3090208@zioup.com> <4A325A40.2050802@stroeder.com> <4A35CDDE.8000604@zioup.com> <4A37719E.3010006@stroeder.com> <4A37A5D9.4040901@zioup.com> <4A3830A6.4030407@eb2bcom.com> <93053DE7-C324-4124-BF8F-B3C7088D66EB@Isode.com> <4A39E563.1040107@eb2bcom.com> <00B0D25E-1707-4A6E-8570-968D3D9AF5F0@Isode.com> <4A3B0D3A.6080406@zioup.com>
X-Mailer: Apple Mail (2.935.3)
MIME-Version: 1.0
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Cc: ldapext@ietf.org
Subject: Re: [ldapext] UTF-8 full support in LDIF / LDIF v2
X-BeenThere: ldapext@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: LDAP Extension Working Group <ldapext.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/ldapext>, <mailto:ldapext-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/ldapext>
List-Post: <mailto:ldapext@ietf.org>
List-Help: <mailto:ldapext-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/ldapext>, <mailto:ldapext-request@ietf.org?subject=subscribe>
X-List-Received-Date: Tue, 23 Jun 2009 01:23:17 -0000

On Jun 18, 2009, at 8:59 PM, Yves Dorfsman wrote:

> Just when I was about to give up.... Anyway, if we go ahead with  
> this, I am assuming we need to move the discussion to a different  
> mailing list ?

We actually need less LDAP lists.  It is quite fine to continue to use  
this list for LDAP extension engineering discussions, whether standard  
track or not.

-- Kurt

From Kurt.Zeilenga@isode.com  Mon Jun 22 19:06:23 2009
Return-Path: <Kurt.Zeilenga@isode.com>
X-Original-To: ldapext@core3.amsl.com
Delivered-To: ldapext@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 115373A6B05 for <ldapext@core3.amsl.com>; Mon, 22 Jun 2009 19:06:23 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.367
X-Spam-Level: 
X-Spam-Status: No, score=-2.367 tagged_above=-999 required=5 tests=[AWL=0.080,  BAYES_00=-2.599, SARE_SUB_ENC_UTF8=0.152]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id c4xeKCXWtS+A for <ldapext@core3.amsl.com>; Mon, 22 Jun 2009 19:06:22 -0700 (PDT)
Received: from rufus.isode.com (rufus.isode.com [62.3.217.251]) by core3.amsl.com (Postfix) with ESMTP id EF6943A67F8 for <ldapext@ietf.org>; Mon, 22 Jun 2009 19:06:21 -0700 (PDT)
Received: from [192.168.1.102] ((unknown) [75.141.233.128])  by rufus.isode.com (submission channel) via TCP with ESMTPSA  id <SkA4pwBV9GJ9@rufus.isode.com>; Tue, 23 Jun 2009 03:06:36 +0100
X-SMTP-Protocol-Errors: NORDNS
Message-Id: <53BFEFC2-BA68-45EE-B77B-F91FDAAD8BE1@isode.com>
From: Kurt Zeilenga <Kurt.Zeilenga@isode.com>
To: Steven Legg <steven.legg@eb2bcom.com>
In-Reply-To: <4A3AE785.5010509@eb2bcom.com>
Date: Mon, 22 Jun 2009 19:06:29 -0700
References: <49C497F9.7010200@zioup.com> <CD3905D4-2A25-4C56-8187-3CE10D46C929@isode.com> <49C870C6.4010803@zioup.com> <E94B7389-9A6D-4CB6-BB2C-649CCD3FD15B@Isode.com> <49CB192E.5050105@zioup.com> <49CB211C.6070108@eb2bcom.com> <49CB87FE.1050809@zioup.com> <49CC01DE.6040506@eb2bcom.com> <4A24557D.7030006@zioup.com> <4A26A05D.8040105@zioup.com> <245BF18B-2066-4E36-9502-16F4A3140D9E@Isode.com> <4A309775.3080406@zioup.com> <4A311ED1.1030202@stroeder.com> <4A31D27B.3090208@zioup.com> <4A325A40.2050802@stroeder.com> <4A35CDDE.8000604@zioup.com> <4A37719E.3010006@stroeder.com> <4A37A5D9.4040901@zioup.com> <4A3830A6.4030407@eb2bcom.com> <93053DE7-C324-4124-BF8F-B3C7088D66EB@Isode.com> <4A39E563.1040107@eb2bcom.com> <4A3A13B2.6060307@stroeder.com> <4A3AE785.5010509@eb2bcom.com>
X-Mailer: Apple Mail (2.935.3)
MIME-Version: 1.0
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Cc: ldapext@ietf.org, =?ISO-8859-1?Q?Michael_Str=F6der?= <michael@stroeder.com>
Subject: Re: [ldapext] UTF-8 full support in LDIF / LDIF v2
X-BeenThere: ldapext@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: LDAP Extension Working Group <ldapext.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/ldapext>, <mailto:ldapext-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/ldapext>
List-Post: <mailto:ldapext@ietf.org>
List-Help: <mailto:ldapext-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/ldapext>, <mailto:ldapext-request@ietf.org?subject=subscribe>
X-List-Received-Date: Tue, 23 Jun 2009 02:06:23 -0000

On Jun 18, 2009, at 6:19 PM, Steven Legg wrote:

> The potential for an inadvertent change of normalization in the  
> LDIFv2 if
> it is edited doesn't overly concern me. Stringprep takes care of it  
> for
> matching purposes

Not for userPassword and the like.

Not for value syntaxes which require a specific normalization to be  
applied else result in a syntax error.

And, end-of-line characters appearing in values are not required to be  
base64'ed or otherwise escaped, there will inadvertent change of end- 
of-line characters to deal with.

LDIFv1 avoided such problems by limiting the characters in values that  
could appear without being base64'ed to a subset of the ASCII subset  
of characters.  These issues haven't gone away since the introduction  
of LDIFv1.

> and any client that expects attribute values to be in,
> or remain in, a particular normalization form is asking for trouble.

If a technical specification says an attribute value is to be in a  
particular Unicode normalization form, then all clients supporting  
that technical specification need to be provide the values of that  
attribute in a particular Unicode normalization form.

> The
> values could be modified by some other client that changes the  
> normalization
> during editing and I wouldn't count on every directory implementation
> preserving the exact character sequence it is given (though mine  
> does).

If the normalization is specified as part of the LDAP syntax for the  
attribute value syntax, it follows that there would be a requirement  
for directory servers to preserve that normalization.  Or the value  
might be stored in an octet string (like userPassword) and the server  
required to preserve the octets and hence the normalization.

> If a client needs the values to be in a particular normalization  
> form it
> should do the conversion itself.

We already have one standard attribute, userPassword, where values  
(when text) SHOULD to be provided in a particular Unicode normalization.

-- Kurt

From yves@zioup.com  Tue Jun 23 08:01:13 2009
Return-Path: <yves@zioup.com>
X-Original-To: ldapext@core3.amsl.com
Delivered-To: ldapext@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id D1D743A6EC1 for <ldapext@core3.amsl.com>; Tue, 23 Jun 2009 08:01:13 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.216
X-Spam-Level: 
X-Spam-Status: No, score=-2.216 tagged_above=-999 required=5 tests=[AWL=0.231,  BAYES_00=-2.599, SARE_SUB_ENC_UTF8=0.152]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id mWE-044qz30I for <ldapext@core3.amsl.com>; Tue, 23 Jun 2009 08:01:12 -0700 (PDT)
Received: from idcmail-mo1so.shaw.ca (idcmail-mo1so.shaw.ca [24.71.223.10]) by core3.amsl.com (Postfix) with ESMTP id 33B393A6ECE for <ldapext@ietf.org>; Tue, 23 Jun 2009 08:01:12 -0700 (PDT)
Received: from pd3ml1so-ssvc.prod.shaw.ca ([10.0.141.140]) by pd4mo1so-svcs.prod.shaw.ca with ESMTP; 23 Jun 2009 09:00:53 -0600
X-Cloudmark-SP-Filtered: true
X-Cloudmark-SP-Result: v=1.0 c=0 a=J6b0bBEkAAAA:8 a=-Gj4E45Jd3tXcTHM7hcA:9 a=QC_GsbTiMNW2j9W0EkMA:7 a=2hUEGeJJZUfrvWPqmlOmcj0QuCAA:4 a=WPV3hXSGBlIA:10
Received: from unknown (HELO home.zioup.com) ([68.147.37.46]) by pd3ml1so-dmz.prod.shaw.ca with ESMTP; 23 Jun 2009 09:00:53 -0600
Received: from [192.168.2.202] (aristotle.zioup.net [192.168.2.202]) by home.zioup.com (Postfix) with ESMTP id DFD252C0C5 for <ldapext@ietf.org>; Tue, 23 Jun 2009 09:00:52 -0600 (MDT)
Message-ID: <4A40EE24.6050003@zioup.com>
Date: Tue, 23 Jun 2009 09:00:52 -0600
From: Yves Dorfsman <yves@zioup.com>
User-Agent: Thunderbird 2.0.0.21 (X11/20090409)
MIME-Version: 1.0
To: ldapext@ietf.org
References: <49C497F9.7010200@zioup.com> <49C870C6.4010803@zioup.com> <E94B7389-9A6D-4CB6-BB2C-649CCD3FD15B@Isode.com> <49CB192E.5050105@zioup.com> <49CB211C.6070108@eb2bcom.com> <49CB87FE.1050809@zioup.com> <49CC01DE.6040506@eb2bcom.com> <4A24557D.7030006@zioup.com> <4A26A05D.8040105@zioup.com> <245BF18B-2066-4E36-9502-16F4A3140D9E@Isode.com> <4A309775.3080406@zioup.com> <4A311ED1.1030202@stroeder.com> <4A31D27B.3090208@zioup.com> <4A325A40.2050802@stroeder.com> <4A35CDDE.8000604@zioup.com> <4A37719E.3010006@stroeder.com> <4A37A5D9.4040901@zioup.com> <4A3830A6.4030407@eb2bcom.com> <93053DE7-C324-4124-BF8F-B3C7088D66EB@Isode.com> <4A39E563.1040107@eb2bcom.com> <00B0D25E-1707-4A6E-8570-968D3D9AF5F0@Isode.com> <4A3B0D3A.6080406@zioup.com> <0B741CDA-8A31-4A69-B272-BD58C43F097C@Isode.com>
In-Reply-To: <0B741CDA-8A31-4A69-B272-BD58C43F097C@Isode.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Subject: Re: [ldapext] UTF-8 full support in LDIF / LDIF v2
X-BeenThere: ldapext@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: LDAP Extension Working Group <ldapext.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/ldapext>, <mailto:ldapext-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/ldapext>
List-Post: <mailto:ldapext@ietf.org>
List-Help: <mailto:ldapext-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/ldapext>, <mailto:ldapext-request@ietf.org?subject=subscribe>
X-List-Received-Date: Tue, 23 Jun 2009 15:01:14 -0000

Kurt Zeilenga wrote:
> 
>> Just when I was about to give up.... Anyway, if we go ahead with this, 
>> I am assuming we need to move the discussion to a different mailing 
>> list ?
> 
> We actually need less LDAP lists.  It is quite fine to continue to use 
> this list for LDAP extension engineering discussions, whether standard 
> track or not.

Ok. I will have intermittent access to email throughout July, if nothing 
happens here by early august, I will re-start this discussion, focusing on a 
Data Composition Format (to re-use Jim's idea) then.

-- 
Yves.
http://www.sollers.ca/


From steven.legg@eb2bcom.com  Wed Jun 24 02:04:08 2009
Return-Path: <steven.legg@eb2bcom.com>
X-Original-To: ldapext@core3.amsl.com
Delivered-To: ldapext@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 1EDC33A67BD for <ldapext@core3.amsl.com>; Wed, 24 Jun 2009 02:04:08 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: 0.428
X-Spam-Level: 
X-Spam-Status: No, score=0.428 tagged_above=-999 required=5 tests=[AWL=0.050,  BAYES_00=-2.599, FH_HOST_EQ_D_D_D_D=0.765, FH_HOST_EQ_D_D_D_DB=0.888,  HOST_EQ_STATIC=1.172, SARE_SUB_ENC_UTF8=0.152]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id OaIEMv-zGLpc for <ldapext@core3.amsl.com>; Wed, 24 Jun 2009 02:04:07 -0700 (PDT)
Received: from eb2bcom.com (165.203.232.72.static.reverse.ltdomains.com [72.232.203.165]) by core3.amsl.com (Postfix) with ESMTP id 027853A6A38 for <ldapext@ietf.org>; Wed, 24 Jun 2009 02:03:40 -0700 (PDT)
Received: from [202.164.192.219] (helo=[192.168.1.9]) by host.eb2bcom.com with esmtpa (Exim 4.69) (envelope-from <steven.legg@eb2bcom.com>) id 1MJNJE-0003CE-MP; Wed, 24 Jun 2009 17:54:21 +1000
Message-ID: <4A41DBA8.8050009@eb2bcom.com>
Date: Wed, 24 Jun 2009 17:54:16 +1000
From: Steven Legg <steven.legg@eb2bcom.com>
User-Agent: Thunderbird 2.0.0.22 (Windows/20090605)
MIME-Version: 1.0
To: Kurt Zeilenga <Kurt.Zeilenga@isode.com>
References: <49C497F9.7010200@zioup.com> <CD3905D4-2A25-4C56-8187-3CE10D46C929@isode.com> <49C870C6.4010803@zioup.com> <E94B7389-9A6D-4CB6-BB2C-649CCD3FD15B@Isode.com> <49CB192E.5050105@zioup.com> <49CB211C.6070108@eb2bcom.com> <49CB87FE.1050809@zioup.com> <49CC01DE.6040506@eb2bcom.com> <4A24557D.7030006@zioup.com> <4A26A05D.8040105@zioup.com> <245BF18B-2066-4E36-9502-16F4A3140D9E@Isode.com> <4A309775.3080406@zioup.com> <4A311ED1.1030202@stroeder.com> <4A31D27B.3090208@zioup.com> <4A325A40.2050802@stroeder.com> <4A35CDDE.8000604@zioup.com> <4A37719E.3010006@stroeder.com> <4A37A5D9.4040901@zioup.com> <4A3830A6.4030407@eb2bcom.com> <93053DE7-C324-4124-BF8F-B3C7088D66EB@Isode.com> <4A39E563.1040107@eb2bcom.com> <4A3A13B2.6060307@stroeder.com> <4A3AE785.5010509@eb2bcom.com> <53BFEFC2-BA68-45EE-B77B-F91FDAAD8BE1@isode.com>
In-Reply-To: <53BFEFC2-BA68-45EE-B77B-F91FDAAD8BE1@isode.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-AntiAbuse: This header was added to track abuse, please include it with any abuse report
X-AntiAbuse: Primary Hostname - host.eb2bcom.com
X-AntiAbuse: Original Domain - ietf.org
X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12]
X-AntiAbuse: Sender Address Domain - eb2bcom.com
X-Source: 
X-Source-Args: 
X-Source-Dir: 
Cc: ldapext@ietf.org
Subject: Re: [ldapext] UTF-8 full support in LDIF / LDIF v2
X-BeenThere: ldapext@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: LDAP Extension Working Group <ldapext.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/ldapext>, <mailto:ldapext-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/ldapext>
List-Post: <mailto:ldapext@ietf.org>
List-Help: <mailto:ldapext-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/ldapext>, <mailto:ldapext-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 24 Jun 2009 09:04:08 -0000

Kurt,

Kurt Zeilenga wrote:
> On Jun 18, 2009, at 6:19 PM, Steven Legg wrote:
> 
>> The potential for an inadvertent change of normalization in the LDIFv2 if
>> it is edited doesn't overly concern me. Stringprep takes care of it for
>> matching purposes
> 
> Not for userPassword and the like.

The extended format in the ELDIF specification in its current form can alter
end-of-line characters so I only use it for syntaxes where I know this is
harmless, which basically means XED syntaxes that are known to contain only XML
documents. Since the Octet String syntax doesn't fit this criterion, the
userPassword attribute never uses the extended format. If we generalize the
extended format to allow "here" documents with the unmodified literal LDAP
attribute value, then I would expect the extended format to be limited to
syntaxes that are known to produce exclusively UTF-8 character strings,
which would continue to exclude userPassword.

When I dump userPassword values they are encrypted, so even if the contents
of the octet string were UTF-8 to start with they probably aren't after
the encryption is done with it.

> 
> Not for value syntaxes which require a specific normalization to be 
> applied else result in a syntax error.

I'm not aware of any such syntax.

> 
> And, end-of-line characters appearing in values are not required to be 
> base64'ed or otherwise escaped, there will inadvertent change of 
> end-of-line characters to deal with.

It's best to be tolerant of such variations anyway since editing by
ordinary LDAP clients could create such inadvertent changes.

> 
> LDIFv1 avoided such problems by limiting the characters in values that 
> could appear without being base64'ed to a subset of the ASCII subset of 
> characters.  These issues haven't gone away since the introduction of 
> LDIFv1.
> 
>> and any client that expects attribute values to be in,
>> or remain in, a particular normalization form is asking for trouble.
> 
> If a technical specification says an attribute value is to be in a 
> particular Unicode normalization form, then all clients supporting that 
> technical specification need to be provide the values of that attribute 
> in a particular Unicode normalization form.

I don't know of any such specification for an existing syntax that produces
exclusively UTF-8 encodings. It would be most unwise for such a requirement
to be placed on the use of an existing syntax (e.g., Directory String)
because of the installed base of software that just wouldn't honour the
requirement. If it's a new syntax, then it wouldn't be known to existing
ELDIF implementations so attributes of the syntax wouldn't use the
extended format. Depending on the details, when I got around to implementing
it (i.e., making it known) I might explicitly exclude it from using the
extended format and/or have my ELDIF parser renormalize the values it is
importing.

> 
>> The
>> values could be modified by some other client that changes the 
>> normalization
>> during editing and I wouldn't count on every directory implementation
>> preserving the exact character sequence it is given (though mine does).
> 
> If the normalization is specified as part of the LDAP syntax for the 
> attribute value syntax, it follows that there would be a requirement for 
> directory servers to preserve that normalization.  Or the value might be
> stored in an octet string (like userPassword) and the server required to 
> preserve the octets and hence the normalization.

Putting it in an octet string would protect it from using the extended format,
as I envisage its applicability.

Regards,
Steven

> 
>> If a client needs the values to be in a particular normalization form it
>> should do the conversion itself.
> 
> We already have one standard attribute, userPassword, where values (when 
> text) SHOULD to be provided in a particular Unicode normalization.
> 
> -- Kurt

From michael@stroeder.com  Tue Jun 30 15:13:28 2009
Return-Path: <michael@stroeder.com>
X-Original-To: ldapext@core3.amsl.com
Delivered-To: ldapext@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id DDD1E3A6D03 for <ldapext@core3.amsl.com>; Tue, 30 Jun 2009 15:13:28 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.154
X-Spam-Level: 
X-Spam-Status: No, score=-2.154 tagged_above=-999 required=5 tests=[AWL=-0.007, BAYES_00=-2.599, MIME_8BIT_HEADER=0.3, SARE_SUB_ENC_UTF8=0.152]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id wmOXwvkKDU4j for <ldapext@core3.amsl.com>; Tue, 30 Jun 2009 15:13:28 -0700 (PDT)
Received: from srv1.stroeder.com (srv1.stroeder.com [213.240.180.113]) by core3.amsl.com (Postfix) with ESMTP id 948FC3A6D00 for <ldapext@ietf.org>; Tue, 30 Jun 2009 15:13:27 -0700 (PDT)
Received: from localhost (localhost [127.0.0.1]) by srv1.stroeder.com (Postfix) with ESMTP id 7B0AE4E074; Wed,  1 Jul 2009 00:12:54 +0200 (CEST)
X-Virus-Scanned: amavisd-new at stroeder.com
Received: from srv1.stroeder.com ([127.0.0.1]) by localhost (srv1.stroeder.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id rhqj2IiD90ng; Wed,  1 Jul 2009 00:12:50 +0200 (CEST)
Received: from [10.1.0.2] (unknown [10.1.0.2]) by srv1.stroeder.com (Postfix) with ESMTP id 2CF3E4E06F; Wed,  1 Jul 2009 00:12:49 +0200 (CEST)
Message-ID: <4A4A8E14.5020909@stroeder.com>
Date: Wed, 01 Jul 2009 00:13:40 +0200
From: =?ISO-8859-1?Q?Michael_Str=F6der?= <michael@stroeder.com>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.22) Gecko/20090606 SeaMonkey/1.1.17
MIME-Version: 1.0
To: Kurt Zeilenga <Kurt.Zeilenga@Isode.com>
References: <49C497F9.7010200@zioup.com> <CD3905D4-2A25-4C56-8187-3CE10D46C929@isode.com> <49C870C6.4010803@zioup.com> <E94B7389-9A6D-4CB6-BB2C-649CCD3FD15B@Isode.com> <49CB192E.5050105@zioup.com> <49CB211C.6070108@eb2bcom.com> <49CB87FE.1050809@zioup.com> <49CC01DE.6040506@eb2bcom.com> <4A24557D.7030006@zioup.com> <4A26A05D.8040105@zioup.com> <245BF18B-2066-4E36-9502-16F4A3140D9E@Isode.com> <4A309775.3080406@zioup.com> <4A311ED1.1030202@stroeder.com> <4A31D27B.3090208@zioup.com> <35B2A165-CE5D-4650-AADE-CC233F71470E@Isode.com> <4A35D23D.5040307@zioup.com> <D437E784-4198-4037-A4EA-0300439C3D2C@Isode.com> <4A37BCEB.5040103@zioup.com> <68380E97-521C-4A80-A569-D09F8F626F6F@Isode.com> <4A3A1D81.1000407@stroeder.com> <32A3DDC8-FC80-490B-89A2-71ADC9582190@Isode.com>
In-Reply-To: <32A3DDC8-FC80-490B-89A2-71ADC9582190@Isode.com>
X-Enigmail-Version: 0.95.7
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 8bit
Cc: ldapext@ietf.org
Subject: Re: [ldapext] UTF-8 full support in LDIF / LDIF v2
X-BeenThere: ldapext@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: LDAP Extension Working Group <ldapext.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/ldapext>, <mailto:ldapext-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/ldapext>
List-Post: <mailto:ldapext@ietf.org>
List-Help: <mailto:ldapext-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/ldapext>, <mailto:ldapext-request@ietf.org?subject=subscribe>
X-List-Received-Date: Tue, 30 Jun 2009 22:13:29 -0000

Kurt Zeilenga wrote:
> 
> On Jun 18, 2009, at 3:57 AM, Michael Str�der wrote:
> 
>> Stepping back a bit from the details of the interesting Unicode issues
>> posted here I wonder what the general strategy of the IETF regarding
>> these issues is?
> 
> Punt.
> 
>> I remember discussions on the ietf-pkix mailing list
>> mentioning problems like these (e.g. when displaying subject names of
>> X.509 certs) without any real solution.
>>
>> I think any system which takes (user) input, decodes it to a Unicode
>> code point sequence and display it to the user is affected by issues
>> with BIDI, combining characters and duplicate Unicode points.
> 
> Yes.  The IETF tends to punt such issues to the user interface
> development community.  The IETF tends to restrict itself to design of
> protocols not design user interface (though the IETF does try to
> document user interface issues, especially those with security impact).
> 
> I think of LDIF as an alternative encoding of protocol data units, used
> for out-of-band transfer data between protocol peers.  That is, I punt
> the "user" as far as I can.

I'm not sure whether I fully understand the usage of the word "punt".
(Bear in mind I'm not a native English speaker.)

I agree that LDIF is just an alternative encoding of protocol data
units. So lifting the ASCII limitation in LDIF would IMO not introduce
any other problem a LDAP client with user interface does not already
have today (despite the new-line issue).

> Others see LDIF as a user display format

Strictly speaking nobody said that. ;-}

Seriously there's another layer between the LDIF data and the user - the
user interface. So if the IETF does not care about the user interface we
also don't have to care about that regarding LDIF.

Whether a particular LDIF is displayable by any means in some localized
context is not something we have to solve at this LDIF-formatted PDU
level because we also don't do it for normal LDAP PDUs.

> and user input format for LDAP data.

If the user enters data into an input field most times he uses his
normal keyboard and the display shows the input using fonts installed on
his system which matches his language. I don't think that there is
really a new problem introduced except a BIDI problem because the
attribute type names are written left-to-right. Maybe even this could be
solved in a decent user interface.

So I'd lift the ASCII limitation but I'm not keen on having to deal with
the "here" document scheme.

Ciao, Michael.

From Kurt.Zeilenga@Isode.com  Tue Jun 30 15:48:30 2009
Return-Path: <Kurt.Zeilenga@Isode.com>
X-Original-To: ldapext@core3.amsl.com
Delivered-To: ldapext@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 8E2D128C2EF for <ldapext@core3.amsl.com>; Tue, 30 Jun 2009 15:48:30 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.229
X-Spam-Level: 
X-Spam-Status: No, score=-2.229 tagged_above=-999 required=5 tests=[AWL=-0.082, BAYES_00=-2.599, MIME_8BIT_HEADER=0.3, SARE_SUB_ENC_UTF8=0.152]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id wRC968J-gJpq for <ldapext@core3.amsl.com>; Tue, 30 Jun 2009 15:48:29 -0700 (PDT)
Received: from rufus.isode.com (rufus.isode.com [62.3.217.251]) by core3.amsl.com (Postfix) with ESMTP id 501A83A69DC for <ldapext@ietf.org>; Tue, 30 Jun 2009 15:48:29 -0700 (PDT)
Received: from [192.168.1.102] ((unknown) [75.141.233.128])  by rufus.isode.com (submission channel) via TCP with ESMTPSA  id <SkqWJwBV9CzM@rufus.isode.com>; Tue, 30 Jun 2009 23:48:11 +0100
X-SMTP-Protocol-Errors: NORDNS
Message-Id: <5EF85B07-379E-4549-8112-17B9D0428589@Isode.com>
From: Kurt Zeilenga <Kurt.Zeilenga@Isode.com>
To: =?ISO-8859-1?Q?Michael_Str=F6der?= <michael@stroeder.com>
In-Reply-To: <4A4A8E14.5020909@stroeder.com>
Date: Tue, 30 Jun 2009 15:48:05 -0700
References: <49C497F9.7010200@zioup.com> <CD3905D4-2A25-4C56-8187-3CE10D46C929@isode.com> <49C870C6.4010803@zioup.com> <E94B7389-9A6D-4CB6-BB2C-649CCD3FD15B@Isode.com> <49CB192E.5050105@zioup.com> <49CB211C.6070108@eb2bcom.com> <49CB87FE.1050809@zioup.com> <49CC01DE.6040506@eb2bcom.com> <4A24557D.7030006@zioup.com> <4A26A05D.8040105@zioup.com> <245BF18B-2066-4E36-9502-16F4A3140D9E@Isode.com> <4A309775.3080406@zioup.com> <4A311ED1.1030202@stroeder.com> <4A31D27B.3090208@zioup.com> <35B2A165-CE5D-4650-AADE-CC233F71470E@Isode.com> <4A35D23D.5040307@zioup.com> <D437E784-4198-4037-A4EA-0300439C3D2C@Isode.com> <4A37BCEB.5040103@zioup.com> <68380E97-521C-4A80-A569-D09F8F626F6F@Isode.com> <4A3A1D81.1000407@stroeder.com> <32A3DDC8-FC80-490B-89A2-71ADC9582190@Isode.com> <4A4A8E14.5020909@stroeder.com>
X-Mailer: Apple Mail (2.935.3)
MIME-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1; format=flowed; delsp=yes
Content-Transfer-Encoding: quoted-printable
Cc: ldapext@ietf.org
Subject: Re: [ldapext] UTF-8 full support in LDIF / LDIF v2
X-BeenThere: ldapext@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: LDAP Extension Working Group <ldapext.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/ldapext>, <mailto:ldapext-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/ldapext>
List-Post: <mailto:ldapext@ietf.org>
List-Help: <mailto:ldapext-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/ldapext>, <mailto:ldapext-request@ietf.org?subject=subscribe>
X-List-Received-Date: Tue, 30 Jun 2009 22:48:30 -0000

On Jun 30, 2009, at 3:13 PM, Michael Str=F6der wrote:

> Kurt Zeilenga wrote:
>>
>> On Jun 18, 2009, at 3:57 AM, Michael Str=F6der wrote:
>>
>>> Stepping back a bit from the details of the interesting Unicode =20
>>> issues
>>> posted here I wonder what the general strategy of the IETF regarding
>>> these issues is?
>>
>> Punt.
>>
>>> I remember discussions on the ietf-pkix mailing list
>>> mentioning problems like these (e.g. when displaying subject names =20=

>>> of
>>> X.509 certs) without any real solution.
>>>
>>> I think any system which takes (user) input, decodes it to a Unicode
>>> code point sequence and display it to the user is affected by issues
>>> with BIDI, combining characters and duplicate Unicode points.
>>
>> Yes.  The IETF tends to punt such issues to the user interface
>> development community.  The IETF tends to restrict itself to design =20=

>> of
>> protocols not design user interface (though the IETF does try to
>> document user interface issues, especially those with security =20
>> impact).
>>
>> I think of LDIF as an alternative encoding of protocol data units, =20=

>> used
>> for out-of-band transfer data between protocol peers.  That is, I =20
>> punt
>> the "user" as far as I can.
>
> I'm not sure whether I fully understand the usage of the word "punt".
> (Bear in mind I'm not a native English speaker.)
>
> I agree that LDIF is just an alternative encoding of protocol data
> units. So lifting the ASCII limitation in LDIF would IMO not introduce
> any other problem a LDAP client with user interface does not already
> have today (despite the new-line issue).

Removing the ASCII restriction will mean that users and systems will =20
need to be far more careful in how they transfer LDIF data.   Today, =20
one can email LDIF files about, FTP them without putting clients in =20
"binary" mode, etc, because of the ASCII restriction.  Without this =20
restriction, users and systems will have to be quite careful to ensure =20=

transfers preserve the LDIF data octet-for-octet.

>
>> Others see LDIF as a user display format
>
> Strictly speaking nobody said that. ;-}
>
> Seriously there's another layer between the LDIF data and the user - =20=

> the
> user interface. So if the IETF does not care about the user =20
> interface we
> also don't have to care about that regarding LDIF.

It's not that the IETF doesn't care about UIs, it's just that UIs are =20=

typically outside the scope of expertise and areas of work the IETF =20
normally undertakes.

>
> Whether a particular LDIF is displayable by any means in some =20
> localized
> context is not something we have to solve at this LDIF-formatted PDU
> level because we also don't do it for normal LDAP PDUs.

Yes.  But folks here seem to be calling for LDIF to be displayable/=20
editable.

>
>> and user input format for LDAP data.
>
> If the user enters data into an input field most times he uses his
> normal keyboard and the display shows the input using fonts =20
> installed on
> his system which matches his language.

It's generally presumed that the user is using a client which is aware =20=

of the application specific semantics and syntaxes of the data he is =20
entering, and if not, the user is obligated to do the work on his own =20=

to ensure application-specific semantics and syntaxes are met.

> I don't think that there is
> really a new problem introduced except a BIDI problem because the
> attribute type names are written left-to-right. Maybe even this =20
> could be
> solved in a decent user interface.

A decent UI would not display attribute type names, but localized =20
strings convening the appropriate semantics.  For instance, a decent =20
UI ought not display "SN" (or the OID identifying the SN attribute =20
type) but the local equivalent to "Surname", and then ensure valued =20
entered met the syntactical requirements of SN attribute type.

> So I'd lift the ASCII limitation but I'm not keen on having to deal =20=

> with
> the "here" document scheme.

I would have less problem replacing the current "safe character" =20
restriction with a broader set of "safe characters".  However, I =20
rather not spend my time arguing over which particular characters are =20=

"safe enough" (for interoperability in data exchange).

-- Kurt=