Subject: [codesprints] 7.39.1.dev1

From: Robert Sparks <[email-masked]>
Date: Tue, 2 Nov 2021 15:46:18 -0500

I've merged what we've all committed so far, and made a new development 
tag at 7.39.1.dev1.

You can get a copy of it as follows: In your current checkout top level, 
run:

$ ./bin/mkdevbranch (your branch root)
$  svn co ^/personal/(your branch root)/7.39.1.dev1 ../7.39.1.dev1

RjS

---

From: Robert Sparks <[email-masked]>
Date: Mon, 8 Nov 2021 14:59:17 -0600

7.39.1.dev2 is now available. Modify the below instructions accordingly.

RjS

On 11/2/21 3:46 PM, Robert Sparks wrote:
> I've merged what we've all committed so far, and made a new 
> development tag at 7.39.1.dev1.
>
> You can get a copy of it as follows: In your current checkout top 
> level, run:
>

---

From: Robert Sparks <[email-masked]>
Date: Tue, 9 Nov 2021 16:22:43 -0600

Yes, I know these are coming close together, but a lot is coming 
together closely.

7.39.1.dev3 is now available.

This brings in a major refactor of the Meeting app to give Purpose to 
Sessions, and to remove the old schedule editor.

Instructions to get a new working copy are again similar to what's below.

A dump of what the changelog will look like (so far) of the next release 
is below. It's going to be a big release.

>> $  svn co ^/personal/(your branch root)/7.39.1.dev1 ../7.39.1.dev1
(env) rjsparks@c445f8b9a0b7:~/ietf/datatracker/trunk$ bin/commitlog

   * Merged branch ^/personal/lars/7.39.1.dev0@19465 from [email-masked]:
     Rework the docker container.

   * Merged in [19466] from [email-masked]:
     Fix HTML by closing some <li> tags.

   * Merged in [19467] from [email-masked]:
     Add a 'sign in' button to the menu bar when the user is not signed in.
     This will hopefully make it more noticeable when one got logged out.

   * Merged in [19468] from [email-masked]:
     Insert a missing </button>.

   * Merged in [19469] from [email-masked]:
     <span> can't have <div> in it. Fix that.

   * Merged in [19470] from [email-masked]:
     Change some popup texts and icons for session buttons.

   * Merged in [19471] from [email-masked]:
     More conformant HTML for the 'sign in' button.

   * Merged in [19472] from [email-masked]:
     Move letter icons consistently after the name.
     Use the correct attribute for an extresource value.

   * Merged in [19473] from [email-masked]:
     Fix search layout.

   * Merged in [19476] from [email-masked]:
     Add svn to image. Grab idnits from GitHub. Try to speed up database
     import.

   * Merged in [19480] from [email-masked]:
     Simple entity caching for the recent drafts view.

   * Merged in [19482] from [email-masked]:
     No need to load d3 or jquery for these templates.

   * Merged in [19483] from [email-masked]:
     Restructure the I-D submission page.
     While there is no change in functionality, we now by default only show
     an option to submit XML, with other formats hidden behind a 'Submit
     other formats' checkbox. I also simplified the text on the page.

   * Merged in [19484] from [email-masked]:
     Remove the Note Well from the datatracker and point to the IETF web
     site.
     One less thing to keep in sync if the Note Well changes.

   * Merged in [19485] from [email-masked]:
     Revamp the menu structure some, and remove a bunch of unneeded files.

   * Merged in [19486] from [email-masked]:
     Remove the ability to submit Postscript I-Ds.
     Fixes #3448.

   * Merged in [19487] from [email-masked]:
     Remove workarounds for ancient IE versions.

   * Merged in [19488] from [email-masked]:
     Change view /ipr/by-draft-recursive to use static content

   * Merged in [19489] from [email-masked]:
     Caching of the latest document view work. May add database stress to
     other views.

   * Merged in [19490] from [email-masked]:
     Refactor expirable_drafts() and its usage to speed up
     doc.views_search.recent_drafts view (and discourage inefficient use 
of the
     method).

   * Merged in [19493] from [email-masked]:
     Create management command base class that sends emails on exceptions.
     Fixes #3356 and #3357.

   * Merged in [19494] from [email-masked]:
     URL-encode the link to a person's public profile, fixing validation
     errors.

   * Merged in [19497] from [email-masked]:
     Change email copy settings.

   * Merged in [19505] from [email-masked]:
     Bump selenium requirement. Forgotten in r19432.

   * Merged in [19506] from [email-masked]:
     Remove duplicate words in shepherds writeup template. Fixes #3453.

   * Merged in ^/personal/nick/7.39.1.dev0@19535 from [email-masked]:
     Provide a docker environment that is friendly to vs code.

   * Merged in ^/personal/nick/7.39.1.dev1@19547 from [email-masked]:
      Downgrade to Python 3.6 to match production. Downgrade to selenium
     3.141.0 as a consequence.

   * Merged in [19541] from [email-masked]:
     Print a line than can actually be copied & pasted to start the
     datatracker.

   * Merged in [19552] from [email-masked]:
     Adjust when we truncate search results.

   * Merged in [19555] from [email-masked]:
     Use temporary directories instead of 'real' filesystem for tests. Fixes
     #3414.

   * Merged in [19544] from [email-masked]:
     Adjust dots claim based on 112 Plenary feedback

   * Merged in [19553] from [email-masked]:
     Gather related document information a little more efficiently for the
     draft main view

   * Merged in [19554] from [email-masked]:
     Separating the relateddocument query results in python when building
     the document main page is slightly faster

   * Revert merge of r19554. Speed comparison was flawed, and moving the few
     comparisons into python made no relevant difference.

   * Merged in ^/personal/jennifer/7.39.1.dev2 from
     [email-masked]:
     Create dev branch for session purpose work (from revision [19414])
     Snapshot of dev work to add session purpose annotation (from
     revision [19415])
     Allow non-WG-like groups to request additional sessions/durations and
     bypass approval (from revision [19424])
     Add 'closed' session purpose, assign purposes for nomcom groups, and
     update
     schedule editor to enforce timeslot type and allow blurring sessions by
     purpose (from revision [19427])
     Add management command to set up timeslots/sessions for testing/demoing
     'purpose' field (from revision [19430])
     Update session purposes and group type -> purpose map to match notes
     page,
     change 'session' purpose to 'regular' (from revision [19433])
     Redirect edit_schedule urls to edit_meeting_schedule view (from
     revision [19434])
     Allow hiding/blurring sessions and timeslots based on TimeSlotType in
     the schedule editor (from revision [19438])
     Disable session purpose/timeslot type hiding on schedule editor when
     only 0 or 1 options (from revision [19439])
     Improvements to the timeslot and schedule editors (move new toggles to
     modals, handle overflowing session names, fix timeslot editor 
scrolling,
     add buttons to quickly create single timeslot, accept trailing slash on
     edit URL) (from revision [19449])
     Update purpose/types after discussions, add on_agenda Session field,
     prevent session requests for groups with no allowed purpose, handle
     addition fields in session request, fix editing session requests, add
     session edit form/access from schedule editor, eliminate
     TimeSlotTypeName
     'private' field, add server-side timeslot type filtering to schedule
     editor
     (from revision [19549])
     Eliminate the officehours timeslot type, update/renumber migrations,
     mark
     offagenda/reserved TimeSlotTypeNames as not used, add a 'none'
     SessionPurposeName and disallow null, update agenda filter
     keywords/filter
     helpers, fix broken tests and general debugging (from revision [19550])
     Tear out old meeting schedule editor and related code (from
     revision [19551])
     Fix merge errors in preceding commits (from revision [19556])
     Add missing ifSeleniumEnabled guard for
     meeting.tests_js.EditTimeslotsTests
     (from revision [19592])
     Remove 'before' version of migrations that were renumbered before 
merge.
     (from revision [19595])

   * Merged in [19579] from [email-masked]:
     Update docker/README.md to reflect the merge of Lars' branch.

   * Merged in [19581] from [email-masked]:
     fix: docker build - cannot change locale warnings

  -- Robert Sparks <[email-masked]>  09 Nov 2021 22:18:12 +0000

---

From: Robert Sparks <[email-masked]>
Date: Thu, 11 Nov 2021 13:27:12 -0600

All -

This update unifies Nick's and Lars' work on the development 
containerization and shifts us to docker-compose so that we can manage 
the development database separately from the code.

It also brings in a few changes to the actual code.

Russ - I've tested that this tag's container actually works on os/x up 
through running the tests.

Note that the instructions for using the container have changed 
_completely_. Please read through 
https://trac.ietf.org/trac/ietfdb/browser/tags/dev/7.39.1.dev4/docker/README.md 
before proceeding.

Consider cleaning out old datatracker docker images and containers.

The first build takes about 20 minutes. After that, getting in and out 
of the container is very quick.

Tests in the container take 20 minutes to complete on my machine, which 
is about twice as long as they take natively.

RjS

Since .dev3:

  * Merged in ^/personal/nick/7.39.1.dev3@19629 from [email-masked]:
     feat: Single docker configuration for VS Code and generic dev [19604]
     fix: remove deprecated files from single docker setup changes [19605]
     fix: finalize docker setup consolidation [19622]
     misc: add cleandb script for docker + fix utf8 db charset params 
[19628]

   * Docker related changes: Mark cleandb executable. Clean settings_local
     and settings_local_sqlitetest of flakes and types issues.

   * Adjust docker volume mounting strategy.

   * Merged in [19621] from [email-masked]:
     Use rfc8989 for 2022 nomcom eligibility. Fixes #3442.

   * Merged in [19636] from [email-masked]:
     Update groupfeatures_session_purposes field migration to match change
     to model.

   * Add a command to clean all datatracker related docker resources (from
     [email-masked])

On 11/9/21 4:22 PM, Robert Sparks wrote:
> A dump of what the changelog will look like (so far) of the next 
> release is below. It's going to be a big release.
>

---

From: Robert Sparks <[email-masked]>
Date: Thu, 11 Nov 2021 13:30:14 -0600

Btw - we will, when we can, be building these images automatically and 
continuously. The pain of this first 20m build will go away (but will be 
replaced with some initial download time that's on the order of the time 
taken to download the database snapshots).

RjS

On 11/11/21 1:27 PM, Robert Sparks wrote:
> Tests in the container take 20 minutes to complete on my machine, 
> which is about twice as long as they take natively.
>
> RjS
>
> Since .dev3:
>
> _______________________________________________
> codesprints mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/codesprints

---

From: Lars Eggert <[email-masked]>
Date: Fri, 12 Nov 2021 08:35:02 +0200

Looks good, will try!

Could we add a third optional image to the setup that has a canned copy of all the extra assets? It’s a bit of a pain to maintain local copies. This would be especially helpful once we hook up the image build to CI, so fresh copies can simply be pulled rather than built locally.

Thanks,
Lars

-- 
Sent from a mobile device; please excuse typos.

> On Nov 11, 2021, at 21:27, Robert Sparks <[email-masked]> wrote:
> 
> All -
> 
> This update unifies Nick's and Lars' work on the development containerization and shifts us to docker-compose so that we can manage the development database separately from the code.
> 
> Russ - I've tested that this tag's container actually works on os/x up through running the tests.
> 
> Note that the instructions for using the container have changed _completely_. Please read through https://trac.ietf.org/trac/ietfdb/browser/tags/dev/7.39.1.dev4/docker/README.md before proceeding.
> 
> The first build takes about 20 minutes. After that, getting in and out of the container is very quick.
> 
> Tests in the container take 20 minutes to complete on my machine, which is about twice as long as they take natively.
> 
> RjS
> 
> Since .dev3:
> 
>     misc: add cleandb script for docker + fix utf8 db charset params [19628]
> 
>> Yes, I know these are coming close together, but a lot is coming together closely.
>> 
>> This brings in a major refactor of the Meeting app to give Purpose to Sessions, and to remove the old schedule editor.
>> 
>> A dump of what the changelog will look like (so far) of the next release is below. It's going to be a big release.
>> 
>>>> I've merged what we've all committed so far, and made a new development tag at 7.39.1.dev1.
>>>> 
>>>> You can get a copy of it as follows: In your current checkout top level, run:
>>>>

---

From: Lars Eggert <[email-masked]>
Date: Fri, 12 Nov 2021 08:36:33 +0200

It will be faster - the download time seems to be bound by the IETF.org pipe currently, whereas I can pull from ghcr.io at tens of MB/s.

> On Nov 11, 2021, at 21:30, Robert Sparks <[email-masked]> wrote:
> 
> Btw - we will, when we can, be building these images automatically and continuously. The pain of this first 20m build will go away (but will be replaced with some initial download time that's on the order of the time taken to download the database snapshots).
> 
> RjS
> 
>

---

From: "Nick" <[email-masked]>
Date: Tue, 16 Nov 2021 22:56:27 -0500

The Markdown package has been updated today to 3.3.5 and it prevents the server from starting.
Pinning the package version to 3.3.4 in requirements.txt resolves the issue.

@Robert Sparks:
By default, Docker for Desktop (mac and Windows) sets a limit on the number of CPU cores and RAM available to containers.
This is likely why you see a performance impact when running the tests inside the container vs on the host.

On Mac, you can increase these settings under Preferences > Advanced, in Docker for Desktop.

Nick

-----Original Message-----
From: Lars Eggert <[email-masked]> 
Sent: November 12, 2021 1:37 AM
To: Robert Sparks <[email-masked]>
Cc: [email-masked]
Subject: Re: [codesprints] 7.39.1.dev4

>>>>> $ ./bin/mkdevbranch (your branch root) $  svn co ^/personal/(your 
>>>>> branch root)/7.39.1.dev1 ../7.39.1.dev1
>>>>> 
>>>>> RjS
>>>>> 
>>> 
>>

---

From: Robert Sparks <[email-masked]>
Date: Wed, 17 Nov 2021 08:10:41 -0600

> On Mac, you can increase these settings under Preferences > Advanced, in Docker for Desktop.

Already done, long ago. The slowdown is, as far as I've been able to 
discern, in the filesystem sync done into the containers through the VM 
docker necessarily runs on OSX. (Windows requires one too, of course).

I'm curious what the total test time is on your windows host?

Fwiw, on my 2.4G 8core i9, 64Gb macbook pro, running natively, the test 
suite takes just under 10 minutes.

I'm currently giving docker 8 cpus, 18Gb ram, and 2Gb swap. There the 
test suite takes just over 20 minutes.

Incrementing the resources above has not had any appreciable effect.

RjS

---

From: Robert Sparks <[email-masked]>
Date: Wed, 17 Nov 2021 09:39:47 -0600

Fact-checking myself:

Native : 926.222s

Docker: 1196.506s

(Native without selenium: 607.834s)

So, really, the slowdown is only 30%

RjS

On 11/17/21 8:10 AM, Robert Sparks wrote:
>> By default, Docker for Desktop (mac and Windows) sets a limit on the 
>> number of CPU cores and RAM available to containers.
>> This is likely why you see a performance impact when running the 
>> tests inside the container vs on the host.
>>
> discern, in the filesystem sync done into the containers through the 
> VM docker necessarily runs on OSX. (Windows requires one too, of course).
>
> Fwiw, on my 2.4G 8core i9, 64Gb macbook pro, running natively, the 
> test suite takes just under 10 minutes.
>
>> It will be faster - the download time seems to be bound by the 
>> IETF.org pipe currently, whereas I can pull from ghcr.io at tens of 
>> MB/s.
>>
>> -- 
>>> Btw - we will, when we can, be building these images automatically 
>>> and continuously. The pain of this first 20m build will go away (but 
>>> will be replaced with some initial download time that's on the order 
>>> of the time taken to download the database snapshots).
>>>
>>> RjS
>>>
>>>
>>>> containerization and shifts us to docker-compose so that we can 
>>>> manage the development database separately from the code.
>>>>
>>>>> This brings in a major refactor of the Meeting app to give Purpose 
>>>>> to Sessions, and to remove the old schedule editor.
>>>>>