Subject: [core] fixing YANG to CBOR

From: Andy Bierman <[email-masked]>
Date: Fri, 22 Jul 2016 00:02:25 -0700

Hi,

It seems to me that the only tagging actually needed
would be to identify the ordered list of member types within a union type.

typedef A {
  type union {
    type int32 ;   // 1
    type string; // 2
    type B;   // 3, 4, 5
    type identityref { base bar; }   // 6
  }
}

typedef B {
  type union {
    type uint8;
    type uint16;
    type uint32;
  }
}

leaf foo { type A; }

YANG allows the member types to be reordered if it does not
change the value set accepted by the type.  It is not clear
if new member types can be added in new module revisions.

If new member types are allowed in typedef B, then the relative
position of (6) can change, even if typedef A does not change.
YANG does not explicitly state if this is allowed or not.

bits and enumerations can be expanded in new revisions,
which changes the value set, but existing value and position numbers
are not allowed to change.

The problematic text is in 6020bis, sec 11

  o  A "type" statement may be replaced with another "type" statement
      that does not change the syntax or semantics of the type.  For
      example, an inline type definition may be replaced with a typedef,
      but an int8 type cannot be replaced by an int16, since the syntax
      would change.

In practice reordering member types would be very rare, but
the SID numbering would need to remember the member type order for each leaf
or leaf-list that resolves to a union type.

It doesn't help to tag any other type besides union.

Andy

---

From: Alexander Pelov <[email-masked]>
Date: Fri, 22 Jul 2016 10:15:48 +0200

Hi Andy,

That sounds great! We’ll have to see how to actually number the elements with Tags. Not sure we can have parameters to Tags. So, it should be something like:
(Tag)[order, value]

Do you think we can use the SIDs to achieve the same? 
E.g. a leaf of type A would result in 6 SIDs from your example. The question here would be if we’ll be losing to much SIDs by doing this. 

Best,
Alexander

> Le 22 juil. 2016 à 09:02, Andy Bierman <[email-masked]> a écrit :
> 
> Hi,
> 
> _______________________________________________
> core mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/core

---

From: Andy Bierman <[email-masked]>
Date: Fri, 22 Jul 2016 01:33:49 -0700

On Fri, Jul 22, 2016 at 1:15 AM, Alexander Pelov <[email-masked]> wrote:

> Hi Andy,
>
> That sounds great! We’ll have to see how to actually number the elements
> with Tags. Not sure we can have parameters to Tags. So, it should be
> something like:
> (Tag)[order, value]
>
> E.g. a leaf of type A would result in 6 SIDs from your example. The
> question here would be if we’ll be losing to much SIDs by doing this.
>
>
I don't know the best way to encode this in CBOR.
I think you mean  (Tag)[sid, value].
This should work, but the client and server might implement this by
actually testing a value against the union type and determining the order
value.
It could be hard-coded with the SID as well.

Andy

---

From: Ladislav Lhotka <[email-masked]>
Date: Mon, 25 Jul 2016 15:00:02 +0200

> change the value set accepted by the type.  It is not clear

I don't think this is true. The only statement in 6020bis that is
related to this issue is the bullet that you cite below, but IMO
reordering member types in a union type definition may change the type
semantics because the receiving side resolves the type according to the
order of member types.

Lada

-- 
Ladislav Lhotka, CZ.NIC Labs
PGP Key ID: E74E8C0C

---

From: Andy Bierman <[email-masked]>
Date: Mon, 25 Jul 2016 07:21:59 -0700

> semantics because the receiving side resolves the type according to the

This depends on the format, but non-overlapping member types
can change position without changing syntax or semantics.
The exact member type that matches is not important.

   typedef foo1 {
      type union {
         type int32 { range "1..10"; }
         type int32 { range "11..20"; }
      }
    }

  typedef foo2 {
      type union {
         type int32 { range "11..20"; }
         type int32 { range "1..10"; }
      }
    }

Lada

Andy

---

From: Ladislav Lhotka <[email-masked]>
Date: Mon, 25 Jul 2016 16:41:26 +0200

> On 25 Jul 2016, at 16:21, Andy Bierman <[email-masked]> wrote:
> 
> 
> 

Yes, but, for instance, the "inet:host" type in RFC 6991 has overlapping members: "1.2.3.4" is valid as both "inet:ip-address" and "inet:domain-name".

Lada

---

From: Michel Veillette <[email-masked]>
Date: Mon, 25 Jul 2016 16:05:38 +0000

Hi Lada, Hi Andy

The important questions we need answer to validate the current solution are the following.

- In a union, is it valid to define multiple enumerations which reuse the same value(s) with a different semantic meaning?
- In a union, is it valid to define multiple bits which reuse the same position(s) with a different semantic meaning?

In the following examples, value or position 0 means both white and red which I believe it's invalid.
If this is the case, the currently proposed solution to tag some specific datatype (bits,  enumeration, identityref, instance-identifier) when used in a union resolve any potential ambiguities.
Can we move ahead with the current solution?

ENUMERATION EXAMPLE

typedef A {
  type union {
    type enumeration {
      enum white {
        value 0;
      }
      enum black {
        value 1;
      }
    }  
    type enumeration {
      enum red {
        value 0;
      }
      enum green {
        value 1;
      }
      enum blue {
        value 2;
      }
    }    
  }
}

BITS EXAMPLE

typedef B {
  type union {
    type bits {
      bit white {
        position 0;
      }
      bit black {
        position 1;
      }
    }
    type bits {
      bit red {
        position 0;
      }
      bit green {
        position 1;
      }
      bit blue {
        position 2;
      }
    }
  }
}

Regards,
Michel

-----Original Message-----
From: core [mailto:[email-masked]] On Behalf Of Ladislav Lhotka
Sent: Monday, July 25, 2016 10:41 AM
To: Andy Bierman <[email-masked]>
Cc: Core <[email-masked]>
Subject: Re: [core] fixing YANG to CBOR

> > It seems to me that the only tagging actually needed would be to 
> > identify the ordered list of member types within a union type.
> >
> >
> > typedef A {
> >   type union {
> >     type int32 ;   // 1
> >     type string; // 2
> >     type B;   // 3, 4, 5
> > YANG allows the member types to be reordered if it does not change 
> > the value set accepted by the type.  It is not clear
> 
> semantics because the receiving side resolves the type according to 
> the order of member types.
> 
> 
> 
> This depends on the format, but non-overlapping member types can 
> change position without changing syntax or semantics.

Lada

> > bits and enumerations can be expanded in new revisions, which 
> > changes the value set, but existing value and position numbers are 
> > not allowed to change.
> >
> > In practice reordering member types would be very rare, but the SID 
> > numbering would need to remember the member type order for each leaf

---

From: Andy Bierman <[email-masked]>
Date: Mon, 25 Jul 2016 09:17:26 -0700

On Mon, Jul 25, 2016 at 9:05 AM, Michel Veillette <
[email-masked]> wrote:

> - In a union, is it valid to define multiple enumerations which reuse the
> same value(s) with a different semantic meaning?
>

Remember that the enum "value" and bit "position" are irrelevant.
They are not defined in YANG to be used on the wire.
So Carsten's examples (2 different enumeration types) is valid
and not a problem for XML and JSON.  They are only a problem
because CBOR is attempting to use something defined NOT to be sent on the
wire.

Lada, There are examples that can be constructed that break,
and others that don't break. So what? Doesn't help solve anything.

Andy

- In a union, is it valid to define multiple bits which reuse the same
> position(s) with a different semantic meaning?
>
> If this is the case, the currently proposed solution to tag some specific
> datatype (bits,  enumeration, identityref, instance-identifier) when used
> in a union resolve any potential ambiguities.
> Yes, but, for instance, the "inet:host" type in RFC 6991 has overlapping
> members: "1.2.3.4" is valid as both "inet:ip-address" and
> "inet:domain-name".
>
> Lada
>

---

From: Michel Veillette <[email-masked]>
Date: Mon, 25 Jul 2016 16:33:50 +0000

Hi Andy

I understand that the enumeration values and bit positions have not been used over the wire for NETconf and RESTconf. However, this doesn't means they are irrelevant and can't be used in the context of a new protocol. Unless you point me on a paragraph in draft-ietf-netmod-rfc6020bis which say that these numbers have been defined for no reasons and can't be used.

Regards,
Michel

From: Andy Bierman [mailto:[email-masked]]
Sent: Monday, July 25, 2016 12:17 PM
To: Michel Veillette <[email-masked]>
Cc: Ladislav Lhotka <[email-masked]>; Core <[email-masked]>
Subject: Re: [core] fixing YANG to CBOR

On Mon, Jul 25, 2016 at 9:05 AM, Michel Veillette <[email-masked]<mailto:[email-masked]>> wrote:
Hi Lada, Hi Andy

because CBOR is attempting to use something defined NOT to be sent on the wire.

Andy

ENUMERATION EXAMPLE

typedef A {
  type union {
    type enumeration {
      enum white {
        value 0;
      }
      enum black {
        value 1;
      }
    }
    type enumeration {
      enum red {
        value 0;
      }
      enum green {
        value 1;
      }
      enum blue {
        value 2;
      }
    }
  }
}

BITS EXAMPLE

typedef B {
  type union {
    type bits {
      bit white {
        position 0;
      }
      bit black {
        position 1;
      }
    }
    type bits {
      bit red {
        position 0;
      }
      bit green {
        position 1;
      }
      bit blue {
        position 2;
      }
    }
  }
}

Regards,
Michel

-----Original Message-----
From: core [mailto:[email-masked]<mailto:[email-masked]>] On Behalf Of Ladislav Lhotka
Sent: Monday, July 25, 2016 10:41 AM
To: Andy Bierman <[email-masked]<mailto:[email-masked]>>
Cc: Core <[email-masked]<mailto:[email-masked]>>
Subject: Re: [core] fixing YANG to CBOR

> On 25 Jul 2016, at 16:21, Andy Bierman <[email-masked]<mailto:[email-masked]>> wrote:
>
>
>
> On Mon, Jul 25, 2016 at 6:00 AM, Ladislav Lhotka <[email-masked]<mailto:[email-masked]>> wrote:
> Andy Bierman <[email-masked]<mailto:[email-masked]>> writes:
>
> > Hi,
> >

Lada

> > [email-masked]<mailto:[email-masked]>

_______________________________________________
core mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/core

---

From: Andy Bierman <[email-masked]>
Date: Mon, 25 Jul 2016 10:09:00 -0700

On Mon, Jul 25, 2016 at 9:33 AM, Michel Veillette <
[email-masked]> wrote:

> Hi Andy
>
>
>
> I understand that the enumeration values and bit positions have not been
> used over the wire for NETconf and RESTconf. However, this doesn't means
> they are irrelevant and can't be used in the context of a new protocol.
> Unless you point me on a paragraph in draft-ietf-netmod-rfc6020bis which
> say that these numbers have been defined for no reasons and can't be used.
>
>
>

There is no text in YANG, NETCONF, or RESTCONF that says these values are
sent on the wire.
There is no defined purpose for them at all in YANG.
They were completely ignored when the union type was designed.
Perhaps this is a bug or not.  The text I cited is very vague.
It does say if "syntax and semantics" is limited to external API behavior or
includes internal implementation details.

Andy

>
>
> *From:* Andy Bierman [mailto:[email-masked]]
> *Sent:* Monday, July 25, 2016 12:17 PM
> *To:* Michel Veillette <[email-masked]>
> *Cc:* Ladislav Lhotka <[email-masked]>; Core <[email-masked]>
> *Subject:* Re: [core] fixing YANG to CBOR
>
>
>
>
>
>
>

---

From: Michel Veillette <[email-masked]>
Date: Mon, 25 Jul 2016 18:00:32 +0000

Hi Any

YANG is a modeling language and shall not care about what is sent on the wire.
YANG just defines the semantic (assign a text and a number for each enumeration and bit position).

Both NETCONF and RESTCONF are non constrained and text based.
In this context, the use of the text to transfer the value of an enumeration or a flag within a bits was logical.
In the context of a constrained protocol, it should now be possible to use the number.

It is very possible that some individuals focused NETCONF or RESTCONF didn't care about those numbers, but I'll be surprised if YANG design team have introduced these numbers just for fun.

Regards,
Michel

Sent: Monday, July 25, 2016 1:09 PM

On Mon, Jul 25, 2016 at 9:33 AM, Michel Veillette <[email-masked]<mailto:[email-masked]>> wrote:
Hi Andy

There is no text in YANG, NETCONF, or RESTCONF that says these values are sent on the wire.

Regards,
Michel

Andy

From: Andy Bierman [mailto:[email-masked]<mailto:[email-masked]>]
Sent: Monday, July 25, 2016 12:17 PM
To: Michel Veillette <[email-masked]<mailto:[email-masked]>>
Cc: Ladislav Lhotka <[email-masked]<mailto:[email-masked]>>; Core <[email-masked]<mailto:[email-masked]>>
Subject: Re: [core] fixing YANG to CBOR

Andy

ENUMERATION EXAMPLE

typedef A {
  type union {
    type enumeration {
      enum white {
        value 0;
      }
      enum black {
        value 1;
      }
    }
    type enumeration {
      enum red {
        value 0;
      }
      enum green {
        value 1;
      }
      enum blue {
        value 2;
      }
    }
  }
}

BITS EXAMPLE

typedef B {
  type union {
    type bits {
      bit white {
        position 0;
      }
      bit black {
        position 1;
      }
    }
    type bits {
      bit red {
        position 0;
      }
      bit green {
        position 1;
      }
      bit blue {
        position 2;
      }
    }
  }
}

Regards,
Michel

-----Original Message-----
From: core [mailto:[email-masked]<mailto:[email-masked]>] On Behalf Of Ladislav Lhotka
Sent: Monday, July 25, 2016 10:41 AM
To: Andy Bierman <[email-masked]<mailto:[email-masked]>>
Cc: Core <[email-masked]<mailto:[email-masked]>>
Subject: Re: [core] fixing YANG to CBOR

Lada

---

From: Ladislav Lhotka <[email-masked]>
Date: Mon, 25 Jul 2016 20:51:36 +0200

> On 25 Jul 2016, at 18:05, Michel Veillette <[email-masked]> wrote:
> 
> Hi Lada, Hi Andy
> 

IMO, yes, in both cases. I am not saying that it's a good design though. 

> In the following examples, value or position 0 means both white and red which I believe it's invalid.

I don't think so. The member types are checked in the same order as they appear in the union's definition, and the first type for which a given value is valid is used.

> If this is the case, the currently proposed solution to tag some specific datatype (bits,  enumeration, identityref, instance-identifier) when used in a union resolve any potential ambiguities.

Yes, I think this is a good idea because otherwise resolving the member types may be difficult. Moreover, sometimes the sending side might want to override the resolution algorithm - e.g., if a client wants to set a leaf with "inet:host" type to "1.2.3.4" and interpret it as a domain name rather than IP address.  

> Can we move ahead with the current solution?

Values 0 and 1 are interpreted as "white" and "black", value 2 is "blue".

>        position 2;

Ditto.

Lada

---

From: Andy Bierman <[email-masked]>
Date: Mon, 25 Jul 2016 12:00:39 -0700

On Mon, Jul 25, 2016 at 11:51 AM, Ladislav Lhotka <[email-masked]> wrote:

>
> > On 25 Jul 2016, at 18:05, Michel Veillette <
> > - In a union, is it valid to define multiple enumerations which reuse
> the same value(s) with a different semantic meaning?
> I don't think so. The member types are checked in the same order as they
> appear in the union's definition, and the first type for which a given
> value is valid is used.
>
>

That's the issue here.
The member type value spaces are allowed to overlap each other.
The "first wins" depends on the value being sent.
The same problem exists for JSON for boolean and numeric types.
Only XML passes everything as a string.  YANG union was designed only to
work
robustly with XML.

> Yes, I think this is a good idea because otherwise resolving the member
> types may be difficult. Moreover, sometimes the sending side might want to
> override the resolution algorithm - e.g., if a client wants to set a leaf
> with "inet:host" type to "1.2.3.4" and interpret it as a domain name rather
> than IP address.
>

The current solution is just as good and just as broken as the JSON
solution,
and we already moved ahead with that in NETMOD WG. (So I guess that means
yes?)

Andy

---

From: Michel Veillette <[email-masked]>
Date: Mon, 25 Jul 2016 19:12:24 +0000

Hi Lada

In your answer " Yes, I think this is a good idea...", you seem to assume that union member are tagged, not datatype.
See below for an example of both approaches.
We basically need to select one of these.

# UNION MEMBER TAGGING

typedef A {

  type union {
    type identityref {     <====== TAG member 1 (e..g. tag 40)
      base "X:Y";
    }

    type enumeration {     <====== TAG member 2 (e..g. tag 41)
      enum white {
        value 0;
      }      enum black {
        value 1;
      }
    } 

    type enumeration {     <====== TAG member 3 (e..g. tag 42)
      enum red {
        value 0;
      }
      enum blue {
        value 1;
      }
    }    
  }
}

PROS, completely remove any ambiguities.
CONS, not backward compatible between module revisions if the order change or new members are inserted.

# DATATYPE TAGGING

typedef A {

  type union {
    type identityref {     <====== TAG identityref (e..g. tag 40)
      base "X:Y";
    }

    type enumeration {     <====== TAG enumeration (e..g. tag 41)
      enum white {
        value 0;
      }      enum black {
        value 1;
      }
    } 

    type enumeration {     <====== TAG enumeration (e..g. tag 41)
      enum red {
        value 0;
      }
      enum blue {
        value 1;
      }
    }    
  }
}

PROS, backward compatible between module versions
CONS, ambiguities may still exists for multiple use of enumeration or bits in the same union.

Regards,
Michel

-----Original Message-----
From: Ladislav Lhotka [mailto:[email-masked]] 
Sent: Monday, July 25, 2016 2:52 PM
To: Michel Veillette <[email-masked]>
Cc: Andy Bierman <[email-masked]>; Core <[email-masked]>
Subject: Re: [core] fixing YANG to CBOR

Ditto.

Lada

>> reordering member types in a union type definition may change the 
>> type semantics because the receiving side resolves the type according 
>> to the order of member types.
>> 
>> 
>>

---

From: Ladislav Lhotka <[email-masked]>
Date: Mon, 25 Jul 2016 22:17:47 +0200

> On 25 Jul 2016, at 21:00, Andy Bierman <[email-masked]> wrote:
> 
> 
> 
> Only XML passes everything as a string.  YANG union was designed only to work
> robustly with XML.

I believe for JSON it is perfectly robust. The general rule should be this: use whatever information you have to determine whether the received value is valid for the first member type. If not, try the second member etc.

Lada

> The current solution is just as good and just as broken as the JSON solution,
> and we already moved ahead with that in NETMOD WG. (So I guess that means yes?)
> 
> 
> Andy
>  
> >
> > ENUMERATION EXAMPLE
> >
> > typedef A {
> >  type union {
> >    type enumeration {
> >      enum white {
> >        value 0;
> >      }
> >      enum black {
> >        value 1;
> >      }
> >    }
> >    type enumeration {
> >      enum red {
> >        value 0;
> >      }
> >      enum green {
> >        value 1;
> >      }
> >      enum blue {
> >        value 2;
> >      }
> >    }
> >  }
> > }
>

---

From: Andy Bierman <[email-masked]>
Date: Mon, 25 Jul 2016 13:21:29 -0700

On Mon, Jul 25, 2016 at 1:17 PM, Ladislav Lhotka <[email-masked]> wrote:

> I believe for JSON it is perfectly robust. The general rule should be
> this: use whatever information you have to determine whether the received
> value is valid for the first member type. If not, try the second member etc.
>
>
believe what youi want:

   type union {
       type string;
       type int32;
   }

JSON produces different results than XML.

If client A sets number 42 in JSON, then client B will read back string
"42" in XML.
If the XML client sets string "42", the JSON client will read back string
"42".
This is the exact same issue facing CBOR conversion

Andy

> >
> >
> > > If this is the case, the currently proposed solution to tag some
> specific datatype (bits,  enumeration, identityref, instance-identifier)
> when used in a union resolve any potential ambiguities.
> >
> > > Yes, but, for instance, the "inet:host" type in RFC 6991 has
> overlapping members: "1.2.3.4" is valid as both "inet:ip-address" and
> "inet:domain-name".
> > >
> > > Lada
> > >

---

From: Ladislav Lhotka <[email-masked]>
Date: Mon, 25 Jul 2016 22:33:42 +0200

> On 25 Jul 2016, at 21:12, Michel Veillette <[email-masked]> wrote:
> 
> Hi Lada
> 

I think I already expressed support to this solution in an earlier thread. This should solve most real-life cases. It is not completely general though because there may be unions inside unions.

Lada

---

From: Ladislav Lhotka <[email-masked]>
Date: Mon, 25 Jul 2016 22:53:50 +0200

> On 25 Jul 2016, at 22:21, Andy Bierman <[email-masked]> wrote:
> 
> 
> 
> If client A sets number 42 in JSON, then client B will read back string "42" in XML.
> If the XML client sets string "42", the JSON client will read back string "42".

In my view, most implementations will eventually use just one encoding, and this problem won't exist for them.

XML, JSON and CBOR in reality pass different information, it is not like UTF-8 versus UTF-16. Restricting all alternative media types only to features supported in XML doesn't make much sense.

Lada

---

From: Andy Bierman <[email-masked]>
Date: Mon, 25 Jul 2016 20:20:04 -0700

Hi,

I think this "union type" corner-case is not bad enough to
justify a really complicated design like tracking the order
of union type syntax changes.  We need to remove order
dependencies, not add new ones.  I should have realized
this is too complicated.

So I think the current tagging solution is good enough.
Do not change it to count and order the member type-stmts.

I will write YANG guidelines in 6087bis to attempt to prevent these
corner-cases from occurring in IETF modules.

Andy

On Mon, Jul 25, 2016 at 1:53 PM, Ladislav Lhotka <[email-masked]> wrote:

> > > > - In a union, is it valid to define multiple enumerations which
> reuse the same value(s) with a different semantic meaning?
> > > > - In a union, is it valid to define multiple bits which reuse the
> same position(s) with a different semantic meaning?
> > >
> > > I don't think so. The member types are checked in the same order as
> they appear in the union's definition, and the first type for which a given
> value is valid is used.
> > >
> > >
> > >
> > > That's the issue here.
> XML, JSON and CBOR in reality pass different information, it is not like
> UTF-8 versus UTF-16. Restricting all alternative media types only to
> features supported in XML doesn't make much sense.
>
> Lada
>
> >
> >
> >
> > Lada
> >
> >
> > Andy
> >
> > >
> > >
> > > Yes, I think this is a good idea because otherwise resolving the
> member types may be difficult. Moreover, sometimes the sending side might
> want to override the resolution algorithm - e.g., if a client wants to set
> a leaf with "inet:host" type to "1.2.3.4" and interpret it as a domain name
> rather than IP address.
> > >
> type

---

From: Michel Veillette <[email-masked]>
Date: Tue, 26 Jul 2016 03:31:01 +0000

+1

Le 25 juil. 2016 à 23:20, Andy Bierman <[email-masked]<mailto:[email-masked]>> a écrit :

Hi,

Andy

On Mon, Jul 25, 2016 at 1:53 PM, Ladislav Lhotka <[email-masked]<mailto:[email-masked]>> wrote:

> On 25 Jul 2016, at 22:21, Andy Bierman <[email-masked]<mailto:[email-masked]>> wrote:
>
>
>
> On Mon, Jul 25, 2016 at 1:17 PM, Ladislav Lhotka <[email-masked]<mailto:[email-masked]>> wrote:
>
> > On 25 Jul 2016, at 21:00, Andy Bierman <[email-masked]<mailto:[email-masked]>> wrote:
> >
> >
> >
> > On Mon, Jul 25, 2016 at 11:51 AM, Ladislav Lhotka <[email-masked]<mailto:[email-masked]>> wrote:
> >
> > > On 25 Jul 2016, at 18:05, Michel Veillette <[email-masked]<mailto:[email-masked]>> wrote:
> > >
> > > Hi Lada, Hi Andy
> > >

Lada