Subject: [netmod] Potential additions to rfc6087bis: RegEx guidelines

From: Xufeng Liu <[email-masked]>
Date: Wed, 23 Aug 2017 21:20:36 +0000

Members of Routing Area Yang DT have had some discussions about the handling of various variants of regular expressions. The followings are the current state, and we are thinking that if this topic can be added to RFC6087bis:

1. Regular Expression Usage
YANG uses regular expressions to restrict string values. Such a restriction can be a part of a "pattern" statement or a string matching function. [RFC7950] specifies that YANG regular expressions will conform to Appendix F in [XSD-TYPES].
YANG models have been implemented in many different environments and the XSD variant of the regular expressions is not supported in many of these environments. There are currently more than a dozen popular regular expression variants implemented in various environments. While the usage of the XSD variant of regular expression described in [RFC7950] remains the preferred standard, a few conventions are prescribed to maximize the portability of YANG models between environments.

1.1. Regular Expression Variant Choice Precedence
YANG model designers SHOULD use the most portable syntax whenever possible. Under the condition that XSD compliance is satisfied and there are multiple choices for a given expression, the following precedence SHOULD be used to choose a regular expressions variant:

o    POSIX base

o    POSIX extended

o    BSD

o    GNU Regular Expression Extensions

o    C++ Regular Expressions with std::regex

o    Others

For example, either \d or [0-9] can be used with equivalent semantics and they are both compliant to [XSD-TYPES]. [0-9] is recommended because [0-9] is supported by POSIX base but \d is not.

1.2.  Convention Guidelines
1.2.1. Avoid Character Category Escapes
For example, in XSD regular expression, \d is a Character Category Escape denoting the range of digits, i.e.,  [0-9]. To maximize portability, the model designers SHOULD use [0-9] instead of \d.

1.2.2. Avoid Unicode Characters
Unicode characters are allowed in XSD regular expressions, but are not supported in the POSIX variant. If possible, the model designers SHOULD avoid using Unicode characters, such as: \p{L} and \p{N}.

1.3. Conversion Tools
Tools can automatically convert regular expressions from one variant to another. When a YANG model is implemented in an environment where XSD regular expressions are not supported, the recommended approach is to use a conversion tool. For example, if needed, anchor position characters, i.e., '^' and '$', can be added by a regular expression conversion tool.

1.4. Validation Tools
Tools can be used to validate regular expressions in YANG files. The followings are some of these tools:

o    YANG W3C Regex Expression Validator: https://yangcatalog.org/yangre

This an on-line tool with a WEB interface.

o    yangre as a part of the libyang package:
https://github.com/CESNET/libyang

This is an open source tool with a command line interface.

Usage:

    yangre [-hvV] -p <regexp1> [-i] [-p <regexp2> [-i] ...] <string>

Returns 0 if string matches the pattern(s), 1 if not and -1 on error.

Options:

  -h, --help              Show this help message and exit.

  -v, --version           Show version number and exit.

  -V, --verbose           Print the processing information.

  -i, --invert-match      Invert-match modifier for the closest preceeding

                          pattern.

  -p, --pattern="REGEXP"  Regular expression including the quoting,

                          which is applied the same way as in a YANG module.

Examples:

  pattern "[0-9a-fA-F]*";      -> yangre -p '"[0-9a-fA-F]*"' '1F'

  pattern '[a-zA-Z0-9\-_.]*';  -> yangre -p "'[a-zA-Z0-9\-_.]*'" 'a-b'

  pattern [xX][mM][lL].*;      -> yangre -p '[xX][mM][lL].*' 'xml-encoding'

References

[XSD-TYPES] Biron, P. and A. Malhotra, "XML Schema Part 2: Datatypes Second Edition", World Wide Web Consortium Recommendation REC-xmlschema-2-20041028, October 2004, <http://www.w3.org/TR/2004/REC-xmlschema-2-20041028>

[POSIX]   IEEE Std 1003.1-2008, 2016, <https://standards.ieee.org/findstds/standard/1003.1-2008.html>

[BSD]     Regular Expression, <http://www.bsd.org/regexintro.html>

[C++]     ISO/IEC DIS 14882: Programming Languages - C++, 2017.

Thanks,
- Xufeng

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Thu, 24 Aug 2017 08:09:00 +0200

> YANG models have been implemented in many different environments and the XSD variant of the regular expressions is not supported in many of these environments. There are currently more than a dozen…

I strongly disagree with this statement. The standard format are XSD
regular expressions. RFC 7950 section 9.4.5:

   The "pattern" statement, which is an optional substatement to the
   "type" statement, takes as an argument a regular expression string,
   as defined in [XSD-TYPES].

There is no notion of a 'preferred' standard.

> o    C++ Regular Expressions with std::regex

Strongly disagree. You either write YANG or something different. There
is no way to recognize what kind of regular expressions have been used
by the model designer. The value of a standard is that everybody does
the same.

> Tools can automatically convert regular expressions from one variant to another. When a YANG model is implemented in an environment where XSD regular expressions are not supported, the recommended…

If conversion tools exist that can convert, then by all means use XSD
in the YANG model and use tools to convert to whatever format your
implementation prefers to use.

/js

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1 | 28759 Bremen | Germany
Fax:   +49 421 200 3103         <http://www.jacobs-university.de/>

---

From: Per Hedeland <[email-masked]>
Date: Thu, 24 Aug 2017 09:24:59 +0200

I strongly agree with all of Juergen's statements, and disagree also
with the suggestion to include the parts of the text that he didn't
specifically disagree with. And I'd like to add that the "lack of XSD
support" argument is pretty weak - there exists at least one freely
available implementation in the form of libxml2, which is actually
present by default in basically all "normal" Linux installations.
It is portable C code, and the parts needed for regexp matching amount
to just above 100 kB of compiled code on an x86_64 CPU.

--Per

---

From: Ladislav Lhotka <[email-masked]>
Date: Thu, 24 Aug 2017 17:54:05 +0200

> to just above 100 kB of compiled code on an x86_64 CPU.

I wouldn't be so strict here. Libxml2 has its share of problems - for
one, its "official" bindings do not support Python3, so e.g. in Yangson
I had to use PyXB package instead and pyang gives up pattern validation
in Python 3 entirely. 

That being said, there doesn't seem to be a clearly superior
replacement, and some aspects of XSD regexes, such as support for
Unicode and the absence of ^ and $ anchors, make a lot of sense in
YANG. So I am also not in favour of the proposed change.

BTW, it is actually a shame that there is no standard regex language
that could be easily used in all programming languages. Oh well ...

Lada

> _______________________________________________
> netmod mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/netmod

-- 
Ladislav Lhotka
Head, CZ.NIC Labs
PGP Key ID: 0xB8F92B08A9F76C67

---

From: Per Hedeland <[email-masked]>
Date: Thu, 24 Aug 2017 19:14:57 +0200

> I had to use PyXB package instead and pyang gives up pattern validation

I don't really see how claiming that the "lack of XSD support" argument
is weak amounts to being "strict" - and I suspect that the claim is
valid even considering the amount of pattern-validating server and
client implementations written in Python3. For a validation/translation
tool such as pyang, having a minuscule C program that is invoked for
validation would seem to be a reasonable implementation if no other
options exist, though admittedly it is an annoyance.

> YANG. So I am also not in favour of the proposed change.

I did not see a proposed change to the standard YANG specification
regarding the regexp flavor, only a proposal that module authors SHOULD
show consideration for implementations that don't comply with the
standard.

--Per

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 24 Aug 2017 20:10:40 +0200

> Unicode characters are allowed in XSD regular expressions, but are not supported in the POSIX variant. If possible, the model designers SHOULD avoid using Unicode characters, such as: \p{L} and \p{…

All ASCII characters are Unicode Characters.  
I think ASCII characters are useful and should be allowed.

(Is this maybe about Unicode categories and character classes?)

Grüße, Carsten

---

From: Xufeng Liu <[email-masked]>
Date: Fri, 25 Aug 2017 12:40:18 +0000

> -----Original Message-----
> From: Per Hedeland [mailto:[email-masked]]
> Sent: Thursday, August 24, 2017 1:15 PM
> To: Ladislav Lhotka <[email-masked]>
> Cc: [email-masked]' <[email-masked]>; Xufeng Liu <[email-masked]>
> Subject: Re: [netmod] Potential additions to rfc6087bis: RegEx guidelines
> 
> >> It is portable C code, and the parts needed for regexp matching
> >> amount to just above 100 kB of compiled code on an x86_64 CPU.
> >
> > one, its "official" bindings do not support Python3, so e.g. in
> > Yangson I had to use PyXB package instead and pyang gives up pattern
> > validation in Python 3 entirely.
> 
> I don't really see how claiming that the "lack of XSD support" argument is weak
> amounts to being "strict" - and I suspect that the claim is valid even considering
> the amount of pattern-validating server and client implementations written in
> Python3. For a validation/translation tool such as pyang, having a minuscule C
> program that is invoked for validation would seem to be a reasonable
> implementation if no other options exist, though admittedly it is an annoyance.
> 
[Xufeng] Besides the language issue, there are situations where XML is not used, so that it is not desirable to include the libxml2 library.

> I did not see a proposed change to the standard YANG specification regarding
> the regexp flavor, only a proposal that module authors SHOULD show
> consideration for implementations that don't comply with the standard.
> 
[Xufeng] This is the point.

> >>>> Members of Routing Area Yang DT have had some discussions about the
> handling of various variants of regular expressions. The followings are the
> current state, and we are thinking that if this topic can be added to RFC6087bis:
> >>>>
> >>>> YANG uses regular expressions to restrict string values. Such a restriction
> can be a part of a "pattern" statement or a string matching function. [RFC7950]
> specifies that YANG regular expressions will conform to Appendix F in [XSD-
> TYPES].
> >>>> YANG models have been implemented in many different environments and
> the XSD variant of the regular expressions is not supported in many of these
> environments. There are currently more than a dozen popular regular expression
> variants implemented in various environments. While the usage of the XSD
> variant of regular expression described in [RFC7950] remains the preferred
> standard, a few conventions are prescribed to maximize the portability of YANG
> models between environments.
> >>>>
> >>>
> >>>> 1.1. Regular Expression Variant Choice Precedence YANG model
> >>>> designers SHOULD use the most portable syntax whenever possible. Under
> the condition that XSD compliance is satisfied and there are multiple choices for
> a given expression, the following precedence SHOULD be used to choose a
> regular expressions variant:
> >>>>
> >>>> o    POSIX base
> >>>>
> >>>> o    POSIX extended
> >>>>
> >>>> o    BSD
> >>>>
> >>> Strongly disagree. You either write YANG or something different.
> >>> There is no way to recognize what kind of regular expressions have
> >>> been used by the model designer. The value of a standard is that
> >>> everybody does the same.
> >>>
> >>>> For example, either \d or [0-9] can be used with equivalent semantics and
> they are both compliant to [XSD-TYPES]. [0-9] is recommended because [0-9] is
> supported by POSIX base but \d is not.
> >>>>
> >>>> 1.2.1. Avoid Character Category Escapes For example, in XSD regular
> >>>> expression, \d is a Character Category Escape denoting the range of digits,
> i.e.,  [0-9]. To maximize portability, the model designers SHOULD use [0-9]
> instead of \d.
> >>>>
> >>>> Unicode characters are allowed in XSD regular expressions, but are not
> supported in the POSIX variant. If possible, the model designers SHOULD avoid
> using Unicode characters, such as: \p{L} and \p{N}.
> >>>>
> >>>> 1.3. Conversion Tools
> >>>> Tools can automatically convert regular expressions from one variant to
> another. When a YANG model is implemented in an environment where XSD
> regular expressions are not supported, the recommended approach is to use a
> conversion tool. For example, if needed, anchor position characters, i.e., '^' and
> '$', can be added by a regular expression conversion tool.
> >>>
> >>> If conversion tools exist that can convert, then by all means use
> >>> XSD in the YANG model and use tools to convert to whatever format
> >>> your implementation prefers to use.
> >>>
> >>> /js
> >>>
> >>

---

From: Xufeng Liu <[email-masked]>
Date: Fri, 25 Aug 2017 12:42:39 +0000

> From: Carsten Bormann [mailto:[email-masked]]
> Sent: Thursday, August 24, 2017 2:11 PM
> To: Xufeng Liu <[email-masked]>
> Cc: [email-masked]
[Xufeng] Certainly you are right. It was meant to say non-ASCII Unicode characters.

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Fri, 25 Aug 2017 14:52:54 +0200

On Fri, Aug 25, 2017 at 12:40:18PM +0000, Xufeng Liu wrote:
> > regarding the regexp flavor, only a proposal that module authors
> > SHOULD show consideration for implementations that don't comply
> > with the standard.
> >

> [Xufeng] This is the point.
 
Perhaps this did not come out properly.

Anyway, why would I as a YANG author have to replace \d with [0..9] so
that implementations that can't handle \d are happy? An implementation
can easily do this substitution itself before passing the pattern to
the regex engine it prefers to use. (I think this is what libyang is
actually doing, I think it uses pcre internally.)

YANG 1.0 and 1.1 are pretty clear which pattern syntax they use.
Implementations should try to support that.

/js

---

From: Xufeng Liu <[email-masked]>
Date: Mon, 28 Aug 2017 12:58:59 +0000

> From: Juergen Schoenwaelder [mailto:[email-masked]]
> Sent: Friday, August 25, 2017 8:53 AM
> Cc: Per Hedeland <[email-masked]>; Ladislav Lhotka <[email-masked]>;
> [email-masked]' <[email-masked]>
> > > SHOULD show consideration for implementations that don't comply with
> > > the standard.
> > >
> 
> Anyway, why would I as a YANG author have to replace \d with [0..9] so that
> implementations that can't handle \d are happy? An implementation can easily
> do this substitution itself before passing the pattern to the regex engine it
> prefers to use. (I think this is what libyang is actually doing, I think it uses pcre
> internally.)
> 

[Xufeng] [0..9] is still compliant with the XSD pattern specified by YANG 1.0 and 1.1. Using [0..9] instead of [\d] will make the implementations with native POSIX RegEx easier without the need for a tool to inspect every element of the RegEx pattern.

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Mon, 28 Aug 2017 17:46:40 +0200

On Mon, Aug 28, 2017 at 12:58:59PM +0000, Xufeng Liu wrote:
> 
> [Xufeng] [0..9] is still compliant with the XSD pattern specified by
> YANG 1.0 and 1.1. Using [0..9] instead of [\d] will make the
> implementations with native POSIX RegEx easier without the need for
> a tool to inspect every element of the RegEx pattern.

Yes, but then \d is legal in YANG (and it is used in a couple of
published RFCs).

Educating _all_ module authors to write [0..9] instead of \d will
likely be more expensive than improving the code of implementations
that did not implement YANG entirely to accept \d.

/js

---

From: Benoit Claise <[email-masked]>
Date: Tue, 29 Aug 2017 09:39:05 +0200

In this discussion, let's keep in mind that the openconfig modules use 
the POSIX regex while the IETF uses the W3C regex.
So for operators that have to deal with a mix of openconfig and IETF 
modules, this type of advice could be handy from a tooling point of 
view. Such advice, if not in RFC6087bis, could be provided in the yangre 
tool or in its GUI equivalent: https://yangcatalog.org/yangre

Regards, Benoit

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Tue, 29 Aug 2017 09:51:12 +0200

On Tue, Aug 29, 2017 at 09:39:05AM +0200, Benoit Claise wrote:
> In this discussion, let's keep in mind that the openconfig modules use the
> POSIX regex while the IETF uses the W3C regex.
> modules, this type of advice could be handy from a tooling point of view.
> Such advice, if not in RFC6087bis, could be provided in the yangre tool or
> in its GUI equivalent: https://yangcatalog.org/yangre

But we might make things worse. POSIX regex != XSD regex and trying to
make them look more similar likely worsens the confusion by making it
even more subtle that there are differences. From this perspective,
using \d instead of [0-9] is actually a good thing.

/js

---

From: Robert Wilton <[email-masked]>
Date: Tue, 29 Aug 2017 15:27:03 +0100

> Yes, but then \d is legal in YANG (and it is used in a couple of
I entirely agree that YANG regular expressions must be legal XML Schema 
regular expressions.

However, I don't think that the majority of YANG implementations are 
going to want to either use libxml or write their own implementation of 
the XML RE language.  Instead it is desirable that they can use whatever 
standard regex implementation comes with their language, or is readily 
available in a library.

Most of the pattern statements I see in YANG modules use a basic subset 
of regular expressions, and hence it looks like they can often be used 
by most RE engines, perhaps with some trivial tweaks or conversions.  
However, there is no formal guidance recommending that pattern 
statements in standard modules are restricted to a subset of XML RE.

Hence, ideally I would like 6087bis to state that pattern statements 
SHOULD also conform to the following additional RE syntax restrictions, 
which I think should make them easy to convert to most other standard 
regex implementations (subject to unicode support limitations):

(1) Allow \d, \D, \s, \S, \w and \W; but not inside character classes.
(2) Disallow \i, \c; and their negative equivalents.
(3) Disallow character class subtraction (e.g. "[A-Z-[RW]]").
(4) Limit the supported unicode categories to only the following 8. Both 
\p and \P syntax is supported, but not inside character classes:
   \p{L} or any kind of letter from any language.
   \p{Ll} a lowercase letter that has an uppercase variant.
   \p{Lu} an uppercase letter that has a lowercase variant.
   \p{Z} any kind of whitespace or invisible separator.
   \p{Zs} a whitespace character that is invisible, but does take up space.
   \p{Zl} a line separator character U+2028.
   \p{N} any kind of numeric character in any script.
   \p{Nd}: a digit zero through nine in any script except ideographic
(5) Disallow matching of unicode blocks.

Thanks,
Rob

---

From: Andy Bierman <[email-masked]>
Date: Tue, 29 Aug 2017 09:15:40 -0700

Hi,

I agree with Juergen that these proposed guidelines are not a good idea.
The priority order for YANG is (1) readers (2) writers and (3) toolmakers.
It seems trivial for group (3) to convert the XSD pattern to some other
format.
It seems difficult to train all the people in groups (1) and (2) that there
are lots of
special new rules to learn.

Andy

On Tue, Aug 29, 2017 at 7:27 AM, Robert Wilton <[email-masked]> wrote:

> However, I don't think that the majority of YANG implementations are going
> to want to either use libxml or write their own implementation of the XML
> RE language.  Instead it is desirable that they can use whatever standard
> regex implementation comes with their language, or is readily available in
> a library.
>
> Most of the pattern statements I see in YANG modules use a basic subset of
> regular expressions, and hence it looks like they can often be used by most
> RE engines, perhaps with some trivial tweaks or conversions.  However,
> there is no formal guidance recommending that pattern statements in
> standard modules are restricted to a subset of XML RE.
>
> which I think should make them easy to convert to most other standard regex
> implementations (subject to unicode support limitations):
>

---

From: Robert Wilton <[email-masked]>
Date: Wed, 30 Aug 2017 10:16:30 +0100

Hi Andy,

What I am suggesting makes it easier for readers, because I am a 
proponent of simpler regular expressions that are easy to read and 
understand.

For example, I wonder how many YANG model readers would immediately 
comprehend what this pattern statement means:

pattern "\p{Sc}\p{Zs}?\p{Nd}+\.\p{Nd}{2}"?

Does allowing such patterns really make it easier for model readers?

The proposes guidelines obviously make it easier (or at least no harder) 
for tool makers.

I agree that there is an minor impact to model writers, but really only 
in the sense that the guidelines would be telling them not to use the 
esoteric options of the XML regex syntax that they probably don't know 
about anyway.

If explicitly putting this in the YANG author guidelines is not liked, 
then another possible option could be a softer recommendation in the 
guidelines RFC, with some more explicit examples of stuff to avoid on an 
YANG FAQ Wiki page.

Thanks,
Rob

On 29/08/2017 17:15, Andy Bierman wrote:
> It seems trivial for group (3) to convert the XSD pattern to some 
> other format.
> It seems difficult to train all the people in groups (1) and (2) that 
> there are lots of
> On Tue, Aug 29, 2017 at 7:27 AM, Robert Wilton <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>
>     I entirely agree that YANG regular expressions must be legal XML
>     Schema regular expressions.
>
>     However, I don't think that the majority of YANG implementations
>     are going to want to either use libxml or write their own
>     implementation of the XML RE language.  Instead it is desirable
>     that they can use whatever standard regex implementation comes
>     with their language, or is readily available in a library.
>
>     Most of the pattern statements I see in YANG modules use a basic
>     subset of regular expressions, and hence it looks like they can
>     often be used by most RE engines, perhaps with some trivial tweaks
>     or conversions.  However, there is no formal guidance recommending
>     that pattern statements in standard modules are restricted to a
>     subset of XML RE.
>
>     Hence, ideally I would like 6087bis to state that pattern
>     statements SHOULD also conform to the following additional RE
>     syntax restrictions, which I think should make them easy to
>     convert to most other standard regex implementations (subject to
>     unicode support limitations):
>
>     (4) Limit the supported unicode categories to only the following
>     8.  Both \p and \P syntax is supported, but not inside character
>     classes:
>     [email-masked] <mailto:[email-masked]>
>     <https://www.ietf.org/mailman/listinfo/netmod>
>
>

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Wed, 30 Aug 2017 12:29:02 +0200

On Wed, Aug 30, 2017 at 10:16:30AM +0100, Robert Wilton wrote:
> Hi Andy,
> 
> What I am suggesting makes it easier for readers, because I am a proponent
> of simpler regular expressions that are easy to read and understand.
> 

This is always difficult to judge but to be fair you have to show how
you express _the same_ (and not a subset) with some other kind of
regular expressions. (My understanding is that \p{Sc} is a currency
symbol.)

> The proposes guidelines obviously make it easier (or at least no harder) for
> tool makers.
> 
> I agree that there is an minor impact to model writers, but really only in
> the sense that the guidelines would be telling them not to use the esoteric
> options of the XML regex syntax that they probably don't know about anyway.

What is 'esoteric' largely depends on your language environment. What
you are saying by 'do not use \p{}' is essentially 'do not use any
unicode long live ASCII'.
 
/js

---

From: Robert Wilton <[email-masked]>
Date: Wed, 30 Aug 2017 12:48:19 +0100

> regular expressions. (My understanding is that \p{Sc} is a currency
Yes, the expression would cover a currency amount, along with associated 
symbol (e.g. "$200.00").

If I was writing a module, I would probably use the following pattern 
statement instead, which I think a lot more people would likely be able 
to comprehend:

pattern "[A-Z]{3}\s?\d+\.\d{2}", using the 3 letter, ISO 4217, currency codes.  e.g. ("USD 200.00")

> unicode long live ASCII'.
No, that is not my intention, i.e. I'm not suggesting banning all use of 
\p{}, but instead limiting it to the character classes that seem like 
they may plausibly be used in standardized YANG modules.

I'm not trying to change what 6020/7950 defines the pattern statement 
as, just give what I perceive as some pragmatic guidance as to what 
parts of XML RE it makes sense to use in standardized YANG modules, 
making it easier for readers and implementations.

I think that it is fine for companies, vendors, etc to use the full 
breadth of XML RE if they wish.

Thanks,
Rob

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Wed, 30 Aug 2017 14:31:56 +0200

On Wed, Aug 30, 2017 at 12:48:19PM +0100, Robert Wilton wrote:
> statement instead, which I think a lot more people would likely be able to
> comprehend:
> 

But that is not the same. Apples versus oranges. (I expect people to
tell me that (i) currency is irrelevant and (ii) that three ASCII
letter currency acronyms are better than currency symbols anyway but
this is a separate discussion I am not interested in.)

> \p{}, but instead limiting it to the character classes that seem like they
> may plausibly be used in standardized YANG modules.

This is entirely subjective. And if you still allow some \p{}, what is
the point of the exercise?

> I'm not trying to change what 6020/7950 defines the pattern statement as,
> just give what I perceive as some pragmatic guidance as to what parts of XML
> RE it makes sense to use in standardized YANG modules, making it easier for
> readers and implementations.
> 
> I think that it is fine for companies, vendors, etc to use the full breadth
> of XML RE if they wish.

Implementations have to be prepared to handle XSD pattern if they
claim compliance to YANG 1.0 and 1.1. So all this only helps
non-compliant implementations. This may indeed be a goal - but then we
should spell this out as such - this helps non-compliant
implementations (and they may still fail on the first \p{} that
you still allow).

If implementations do not implement the YANG pattern statement but
something else, then then they should ignore patterns they can't
understand and treat the pattern as if it would have been in a
description clause - i.e., leave it to humans to write the code that
implements the pattern correctly. Note that YANG does not say anything
how stuff is implemented.

/js

---

From: Andy Bierman <[email-masked]>
Date: Wed, 30 Aug 2017 07:52:01 -0700

On Wed, Aug 30, 2017 at 5:31 AM, Juergen Schoenwaelder <
[email-masked]> wrote:

> proponent
> to
> they

This does not work.
There are 3 outcomes from the regex compiler

1) proper syntax was used and accepted; pattern matches correctly
2) improper syntax was used and accepted; pattern matches incorrectly
3) improper syntax was used and rejected; compiler error generated

Case (2) is the really bad one and we have seen in in bug reports.

This issue was discussed in detail for almost 2 years and the conclusion was
that a YANG extension would be used to specify other pattern types than
the XSD pattern mandated by the standard.

Andy

---

From: Robert Wilton <[email-masked]>
Date: Wed, 30 Aug 2017 17:44:01 +0100

Hi,

On 30/08/2017 15:52, Andy Bierman wrote:
>
>
> On Wed, Aug 30, 2017 at 5:31 AM, Juergen Schoenwaelder 
> <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
I actually think that XML RE is a good choice for YANG pattern 
statements (because it is one of the more simple RE languages), I just 
don't think that we need all of it.

First question: How many pattern statements in draft and standard IETF 
YANG modules actually use Unicode properties (e.g \p{}).
Answer: Just 2.  To add a zone at the end of the IPv4/IPv6 address.

E.g.       pattern
         '(([0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5])\.){3}'
       +  '([0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5])'
       + '(%[\p{N}\p{L}]+)?';

This could quite possibly have been written just as 
"\d{1,3}\.{3}\d{1,3)(%\w+)?" and not use Unicode properties at all.

There a couple more occurrences of Unicode character classes in the 
vendor models on github, but only to restrict them to the ASCII 
character set (oh the irony), which I believe can be accomplished 
without resorting to Unicode properties.

Another question: How often is character class subtraction (e.g. 
[A-Z-[PQ]] used in standard & the github YANG modules?
Answer: 0.  AFAICT, it isn't used at all, anywhere ...

Now, I'm not proposing using a different regex syntax for pattern 
statements, just a sensible subset of XSD RE, such that it easier for 
folks to read/review pattern statements, and it is easier for client and 
server implementations to translate into other common regex 
implementations if they so wish.

Of course, as part of that translation, I would expect a translation 
function to check and generate an error if the translation cannot handle 
the input regex (e.g. if it uses an obscure unmatched unicode property 
or a unicode block, or character class subtraction syntax).  This really 
doesn't seem hard to me.

But the XML RE language has stuff in it that I don't think anyone is 
ever going to use in a standardized network management YANG model. 
Forcing everyone to implement support for this stuff just seems like a 
complete waste of time and effort.  Looking at the regex info website it 
looks like there are about 143 unicode properties and blocks defined (it 
may be incomplete), or which I think that 135+ of these probably have no 
relevance in network management YANG modules, and the benefit of the 
remaining ones is pretty suspect.

I mean, how many network management YANG modules really need a pattern 
statement that only matches Runic characters?  Perhaps someone out there 
is busy defining "middle-earth.yang" ;-)

If I am the only person opposed to making life unnecessarily difficult 
to readers of YANG models, and client/server tool implementors 
interacting with YANG then it is probably time to give up this 
discussion. ;-)

Python, quite likely a common tool for client side network management, 
also doesn't seem to have any support of unicode properties or blocks.  
Perhaps implementations will hook it up to libxml2 instead, or write a 
full translation XML RE to Python RE conversion tool.  But probably most 
people will just feed the pattern statement into the native Python regex 
engine, and my guess is that this will probably work 95% of the time.  
The other 5% ... who knows what will happen ... oh well, better to try 
and fail than to not try at all.

Apologies if this email comes across as a rant.

Rob

>     Fax:   +49 421 200 3103         <http://www.jacobs-university.de/
>     <http://www.jacobs-university.de/>>
>
>

---

From: Kent Watsen <[email-masked]>
Date: Wed, 30 Aug 2017 20:03:44 +0000

As Andy says, readability is #1, and it follows that a restricted subset would be more understandable.  Standardizing this would require an update to RFC 7950 (read: not going to happen anytime soon).  Maybe we could start with just having a tool detect when something outside the common-subset is used.   Can a "common subset" be well-defined?  - "common" between how many engines? - would it be forever evolving?

K. // contributor

On 8/30/17, 12:44 PM, "netmod on behalf of Robert Wilton" <[email-masked]<mailto:[email-masked]> on behalf of [email-masked]<mailto:[email-masked]>> wrote:

I actually think that XML RE is a good choice for YANG pattern statements (because it is one of the more simple RE languages), I just don't think that we need all of it.

First question: How many pattern statements in draft and standard IETF YANG modules actually use Unicode properties (e.g \p{}).
Answer: Just 2.  To add a zone at the end of the IPv4/IPv6 address.

E.g.       pattern

This could quite possibly have been written just as "\d{1,3}\.{3}\d{1,3)(%\w+)?" and not use Unicode properties at all.

There a couple more occurrences of Unicode character classes in the vendor models on github, but only to restrict them to the ASCII character set (oh the irony), which I believe can be accomplished without resorting to Unicode properties.

Another question: How often is character class subtraction (e.g. [A-Z-[PQ]] used in standard & the github YANG modules?
Answer: 0.  AFAICT, it isn't used at all, anywhere ...

Now, I'm not proposing using a different regex syntax for pattern statements, just a sensible subset of XSD RE, such that it easier for folks to read/review pattern statements, and it is easier for client and server implementations to translate into other common regex implementations if they so wish.

Of course, as part of that translation, I would expect a translation function to check and generate an error if the translation cannot handle the input regex (e.g. if it uses an obscure unmatched unicode property or a unicode block, or character class subtraction syntax).  This really doesn't seem hard to me.

But the XML RE language has stuff in it that I don't think anyone is ever going to use in a standardized network management YANG model.   Forcing everyone to implement support for this stuff just seems like a complete waste of time and effort.  Looking at the regex info website it looks like there are about 143 unicode properties and blocks defined (it may be incomplete), or which I think that 135+ of these probably have no relevance in network management YANG modules, and the benefit of the remaining ones is pretty suspect.

I mean, how many network management YANG modules really need a pattern statement that only matches Runic characters?  Perhaps someone out there is busy defining "middle-earth.yang" ;-)

If I am the only person opposed to making life unnecessarily difficult to readers of YANG models, and client/server tool implementors interacting with YANG then it is probably time to give up this discussion. ;-)

Python, quite likely a common tool for client side network management, also doesn't seem to have any support of unicode properties or blocks.  Perhaps implementations will hook it up to libxml2 instead, or write a full translation XML RE to Python RE conversion tool.  But probably most people will just feed the pattern statement into the native Python regex engine, and my guess is that this will probably work 95% of the time.  The other 5% ... who knows what will happen ... oh well, better to try and fail than to not try at all.

Rob

---

From: Lou Berger <[email-masked]>
Date: Wed, 30 Aug 2017 16:28:51 -0400

Rob

Speaking as a contributor.

> Answer: Just 2.  To add a zone at the end of the IPv4/IPv6 address.

I agree with you 100%

And I see Xufeng's proposal for 6087bis as an attempt at putting some 
language together to support this desire. Perhaps you can suggest alternate 
language.

Lou

> ----------

---

From: Andy Bierman <[email-masked]>
Date: Wed, 30 Aug 2017 13:43:22 -0700

Hi,

The burden this would place on YANG writers would be excessive.
We learned in SNMP-land about CLRs (clever little rules) and how they need
to be avoided. We learned that special-casing and sub-setting technology has
its own costs, which are usually more than the problem they solved
(e.g., counter names MUST be in the plural form).

Andy

On Wed, Aug 30, 2017 at 1:03 PM, Kent Watsen <[email-masked]> wrote:

>
>
> As Andy says, readability is #1, and it follows that a restricted subset
> would be more understandable.  Standardizing this would require an update
> to RFC 7950 (read: not going to happen anytime soon).  Maybe we could start
> with just having a tool detect when something outside the common-subset is
> used.   Can a "common subset" be well-defined?  - "common" between how many
> engines? - would it be forever evolving?
>
>
>
> K. // contributor
>
>
>
>
>
> On 8/30/17, 12:44 PM, "netmod on behalf of Robert Wilton" <
> [email-masked] on behalf of [email-masked]> wrote:
>
>
>
> I actually think that XML RE is a good choice for YANG pattern statements
> (because it is one of the more simple RE languages), I just don't think
> that we need all of it.
>
>
> There a couple more occurrences of Unicode character classes in the vendor
> models on github, but only to restrict them to the ASCII character set (oh
> the irony), which I believe can be accomplished without resorting to
> Unicode properties.
>
>
> statements, just a sensible subset of XSD RE, such that it easier for folks
> to read/review pattern statements, and it is easier for client and server
> implementations to translate into other common regex implementations if
> they so wish.
>
> the input regex (e.g. if it uses an obscure unmatched unicode property or a
> unicode block, or character class subtraction syntax).  This really doesn't
> seem hard to me.
>
> But the XML RE language has stuff in it that I don't think anyone is ever
> going to use in a standardized network management YANG model.   Forcing
> everyone to implement support for this stuff just seems like a complete
> waste of time and effort.  Looking at the regex info website it looks like
> there are about 143 unicode properties and blocks defined (it may be
> incomplete), or which I think that 135+ of these probably have no relevance
> in network management YANG modules, and the benefit of the remaining ones
> is pretty suspect.
>
> statement that only matches Runic characters?  Perhaps someone out there is
> busy defining "middle-earth.yang" ;-)
>
> If I am the only person opposed to making life unnecessarily difficult to
> readers of YANG models, and client/server tool implementors interacting
> with YANG then it is probably time to give up this discussion. ;-)
>
> Perhaps implementations will hook it up to libxml2 instead, or write a full
> translation XML RE to Python RE conversion tool.  But probably most people
> will just feed the pattern statement into the native Python regex engine,
> and my guess is that this will probably work 95% of the time.  The other 5%
> ... who knows what will happen ... oh well, better to try and fail than to
> not try at all.
>

---

From: Robert Wilton <[email-masked]>
Date: Thu, 31 Aug 2017 14:53:35 +0100

Hi Lou, all

Proposed 6087bis text inline below.

On 30/08/2017 21:28, Lou Berger wrote:
> >>     >
> >>     >
> >>     > On 30/08/2017 11:29, Juergen Schoenwaelder wrote:
> >>     > > On Wed, Aug 30, 2017 at 10:16:30AM +0100, Robert Wilton wrote:
> >>     > > > Hi Andy,
> >>     > > >
> >>     > > > What I am suggesting makes it easier for readers, because I
> >>     am a proponent
> >>     > > > of simpler regular expressions that are easy to read and
> >>     understand.
> >>     > > >
> >>     > > > For example, I wonder how many YANG model readers would
> >>     immediately
> >>     > > > comprehend what this pattern statement means:
> >>     > > >
> >>     > > > pattern "\p{Sc}\p{Zs}?\p{Nd}+\.\p{Nd}{2}"?
> >>     > > >
> >>     > > > Does allowing such patterns really make it easier for model
> >>     readers?
> >>     > > This is always difficult to judge but to be fair you have to
> >>     show how
> >>     > > you express _the same_ (and not a subset) with some other 
> kind of
> >>     > > regular expressions. (My understanding is that \p{Sc} is a
> >>     currency
> >>     > > symbol.)
> >>     > Yes, the expression would cover a currency amount, along with
> >>     associated
> >>     > symbol (e.g. "$200.00").
> >>     >
> >>     > If I was writing a module, I would probably use the following
> >>     pattern
> >>     > statement instead, which I think a lot more people would likely
> >>     be able to
> >>     > comprehend:
> >>     >
> >>     > pattern "[A-Z]{3}\s?\d+\.\d{2}", using the 3 letter, ISO 4217,
> >>     currency codes.  e.g. ("USD 200.00")
> >>
> >>     > >
> >>     > > > The proposes guidelines obviously make it easier (or at
> >>     least no harder) for
> >>     > > > tool makers.
> >>     > > >
> >>     > > > I agree that there is an minor impact to model writers, but
> >>     really only in
> >>     > > > the sense that the guidelines would be telling them not to
> >>     use the esoteric
> >>     > > > options of the XML regex syntax that they probably don't
> >>     know about anyway.
> >>     > > What is 'esoteric' largely depends on your language
> >>     environment. What
> >>     > > you are saying by 'do not use \p{}' is essentially 'do not 
> use any
> >>     > > unicode long live ASCII'.
> >>     > No, that is not my intention, i.e. I'm not suggesting banning
> >>     all use of
> >>     > \p{}, but instead limiting it to the character classes that seem
> >>     like they
> >>     > may plausibly be used in standardized YANG modules.
> >>
> >>     > I'm not trying to change what 6020/7950 defines the pattern
> >>     statement as,
> >>     > just give what I perceive as some pragmatic guidance as to what
> >>     parts of XML
> >>     > RE it makes sense to use in standardized YANG modules, making it
> >>     easier for
> >>     > readers and implementations.
> >>     >
> >>     > I think that it is fine for companies, vendors, etc to use the
> >>     full breadth
> >>     > of XML RE if they wish.
> >>
> language together to support this desire. Perhaps you can suggest 
> alternate language.
>
I propose adding the following 2 paragraphs to 6087bis section on 
pattern and ranges:

NEW:
To ensure patterns are easy to read and implement, authors SHOULD
restrict themselves to the parts of the XML schema regular expression
language that are common across most regular expression languages.  In
particular, pattern statements SHOULD avoid using 'character class
subtraction' (e.g. '[a-z-[aeiou]]').  They SHOULD avoid matching
unicode properties and blocks (e.g. '\p{L} or \p{IsBasic_Latin}').
They MAY use the '\d', '\w', '\s' character class shorthands and their
negated versions, where appropriate, but SHOULD avoid other character
class shorthands.  To match ASCII digits 0-9 the character class
'[0-9]' MUST be used instead of the '\d' character class shorthand
that matches Unicode digits in all scripts.

Pattern statements do not have to strictly restrict numerical values,
and a simple less specific pattern may be preferable over a more
complex and precise pattern, e.g. as illustrated in the
'ipv4-address-no-zone' example pattern below.

Or, in context of the existing text 6087bis text:

*** Patterns and Ranges

For string data types, if a machine-readable pattern
can be defined for the desired semantics, then
one or more pattern statements SHOULD be present.
A single quoted string SHOULD be used to specify the pattern,
since a double-quoted string can modify the content.

To ensure patterns are easy to read and implement, authors SHOULD
restrict themselves to the parts of the XML schema regular expression
language that are common across most regular expression languages.  In
particular, pattern statements SHOULD avoid using 'character class
subtraction' (e.g. '[a-z-[aeiou]]').  They SHOULD avoid matching
unicode properties and blocks (e.g. '\p{L} or \p{IsBasic_Latin}').
They MAY use the '\d', '\w', '\s' character class shorthands and their
negated versions, where appropriate, but SHOULD avoid other character
class shorthands.  To match ASCII digits 0-9 the character class
'[0-9]' MUST be used instead of the '\d' character class shorthand
that also matches Unicode digits in all scripts.

Pattern statements do not have to strictly restrict numerical values,
and a simple less specific pattern may be preferable over a more
complex and precise pattern, e.g. as illustrated in the
'ipv4-address-no-zone' example pattern below.

The following typedef from ^RFC6991^ demonstrates the proper
use of the "pattern" statement:

     typedef ipv4-address-no-zone {
       type inet:ipv4-address {
         pattern '[0-9\.]*';
       }
       ...
     }

For string data types, if the length of the string
is required to be bounded in all implementations,
then a length statement MUST be present.

The following typedef from ^RFC6991^ demonstrates the proper
use of the "length" statement:

     typedef yang-identifier {
       type string {
         length "1..max";
         pattern '[a-zA-Z_][a-zA-Z0-9\-_.]*';
         pattern '.|..|[^xX].*|.[^mM].*|..[^lL].*';
       }
       ...
     }

For numeric data types, if the values allowed
by the intended semantics are different than
those allowed by the unbounded intrinsic data
type (e.g., 'int32'), then a range statement SHOULD be present.

The following typedef from ^RFC6991^ demonstrates the proper
use of the "range" statement:

     typedef dscp {
       type uint8 {
          range "0..63";
       }
       ...
     }

Thanks,
Rob

---

From: Alex Campbell <[email-masked]>
Date: Thu, 31 Aug 2017 23:57:34 +0000

Hi,

I'd be very wary of adding guidelines that restrict the regex syntax.

A tool that supports YANG must implement the full regex language anyway (or ignore regexes altogether if they are not relevant to the tool's function). However, these guidelines will inevitably encourage some tool authors to take shortcuts.

I'm sure there are already tools that take shortcuts - but if the shortcuts cause problems, right now it's squarely the tool's fault, rather than the model being processed.

If there are official guidelines saying not to use such-and-such regex feature, then the tool author can point to the guideline and say "See? You aren't supposed to be using that".

We may end up with a compatibility nightmare on our hands where certain modules are only compatible with certain tools, even more-so than is the case now.

The only way I'd ever approve of this is if it was a MUST requirement in a new version of YANG.

________________________________
From: netmod <[email-masked]> on behalf of Robert Wilton <[email-masked]>
Sent: Thursday, 31 August 2017 4:44 a.m.
To: Andy Bierman; Juergen Schoenwaelder; Xufeng Liu; [email-masked]
Subject: Re: [netmod] Potential additions to rfc6087bis: RegEx guidelines

Hi,

On Wed, Aug 30, 2017 at 5:31 AM, Juergen Schoenwaelder <[email-masked]<mailto:[email-masked]>> wrote:

This does not work.

E.g.       pattern

Rob

/js

Andy