Subject: [Tools-discuss] Advice on DT schema changes

From: "Salz, Rich" <[email-masked]>
Date: Sun, 17 Nov 2019 07:35:21 +0000

I want to add a uniform and extensible way of attaching “external references” to DT objects: groups, people, documents. For example, I’d like us to be able to say “this is the GitHub repository for this draft,” “my GitLab username is rsalz,” and “the homepage for this WG is https:…..”

I’m attaching two files that I’d like some comments on.   The ext-schema.sql file is a SQL script that shows the tables I am thinking of adding.  The data-changes.py file is a combination of python and pseudo-code that shows how I’d migrate the database. Is this the best place to ask?

Now, I know that this isn’t the way we’d really do things. Instead I’d write Django models, update the view scripts, and a command-line tool to do the migration. But before I dig into that (and learn enough Django so that I can :) I’d like comments – think of them as the high-level design work, for example.

One of my motivations for this (BTW, my first notes on this are on letterhead from the Bangkok hotel :), is so that I can enhance my GitHub scripts, https://github.com/richsalz/ietf-gh-scripts, to automatically pull/push data between GH and DT.

Thanks for any feedback.
                /r$

---

From: Michael Richardson <[email-masked]>
Date: Sun, 17 Nov 2019 22:43:23 +0800

Salz, Rich <[email-masked]> wrote:
    > I want to add a uniform and extensible way of attaching “external
    > references” to DT objects: groups, people, documents. For example, I’d
    > like us to be able to say “this is the GitHub repository for this
    > draft,” “my GitLab username is rsalz,” and “the homepage for this WG is
    > https:…..”

The SQL seems fine.
I didn't quite understand your migration python code.
I got the general gist of it, and it seems right, but it's certainly not in
migration style.  I guess you are initializing things from ref_names: value
side with a unique id saved into cache, but I didn't see where you used that.

--
]               Never tell me the odds!                 | ipv6 mesh networks [
]   Michael Richardson, Sandelman Software Works        | network architect  [
]     [email-masked]  http://www.sandelman.ca/        |   ruby on rails    [

---

From: "Salz, Rich" <[email-masked]>
Date: Sun, 17 Nov 2019 22:05:52 +0000

What is migration style?  Where can I find "typical" examples to emulate?

The idea is that existing "additional URLs" fields -- for example, look at https://datatracker.ietf.org/wg/quic/about/ or https://datatracker.ietf.org/doc/draft-ietf-tls-subcerts/ -- will have those fields that are common (as specified in the python script) moved into the new tables. I am calling those new commonly-named things "extref" for "external referfence"  The value is just a string, but the idea is that it is an identifier that is useful in some external context, and having common names for these thing makes more automation possible.

For example, a "GitHub Repository" extref.  We want that name normalized, so that tooling can fetch it. Right now the additional URL's are a set of text fields, and the names are not consistent: "Github repo" "GitHub Repository" "Repository", etc., are all used.  That is the point of the ref_names map, to capture all the existing names via regexp and map them into a common name which gets an ID in the external_refs table, which is then used as foreign keys in the "list of external references" for each of the person, group, doc, entries. It will not be perfect, but I looked at all the names and in use (the tables mentioned in the python outline) and I think it gets almost all things right.  I am unsure, for example, of the YANG links; what tooling of theirs breaks if the name becomes generic?  Perhaps I just don't remove a link when migrating.

What is not shown or done is mapping these extrefs into the source, such as ietf/doc/models.py, and displaying them. That's the real work, I just want commentary on the data structures first.  For example, in https://datatracker.ietf.org/person/[email-masked] I could add a "standard" extref that says "On github I am richsalz"  Then this could be used by scripts to populate repo's and auto-link back to the DT accounts.

Hope that helps you help me help the IETF. :)

---

From: Michael Richardson <[email-masked]>
Date: Mon, 18 Nov 2019 15:14:12 +0800

Salz, Rich <[email-masked]> wrote:
    > The idea is that existing "additional URLs" fields -- for example, look
    > at https://datatracker.ietf.org/wg/quic/about/ or
    > https://datatracker.ietf.org/doc/draft-ietf-tls-subcerts/ -- will have
    > those fields that are common (as specified in the python script) moved
    > into the new tables. I am calling those new commonly-named things
    > "extref" for "external referfence" The value is just a string, but the
    > idea is that it is an identifier that is useful in some external
    > context, and having common names for these thing makes more automation
    > possible.

    > For example, a "GitHub Repository" extref.  We want that name
    > normalized, so that tooling can fetch it. Right now the additional
    > URL's are a set of text fields, and the names are not consistent:
    > "Github repo" "GitHub Repository" "Repository", etc., are all used.
    > That is the point of the ref_names map, to capture all the existing
    > names via regexp and map them into a common name which gets an ID in
    > the external_refs table, which is then used as foreign keys in the
    > "list of external references" for each of the person, group, doc,

Oh, I understand now.
I didn't know that there were existing data in a free-form field.