Subject: [codesprints] Docker mount options

From: Tero Kivinen <[email-masked]>
Date: Thu, 3 Jun 2021 23:12:53 +0300

In codesprint there was discussion that it would be faster to use
:delegated on the database mounts, so I added an option to docker/run
that allows setting mount option and run some test.

I then run ./docker/updatedb with delegated, cached and default and
there was no real difference in speeds for those runs. All of them
took 140-150 minutes on my laptop.

There might of course be differences on other platforms or hardware
(this was on linux using old slow laptop).

As there was no real difference I do not think it is worthwhile to add
that --db-opt option I created to docker/run...
-- 
[email-masked]

---

From: Lars Eggert <[email-masked]>
Date: Fri, 4 Jun 2021 12:10:45 +0300

Hi,

> that --db-opt option I created to docker/run...

hm. That's a disappointment.

Are there other mounted volumes? Maybe add :delegated to all of them, to see if that makes any difference?

If I understand the DT docker setup correctly, nothing executes in the host (= everything executes in containers), so :delegated should always be a no-op or an improvement.

Thanks,
Lars

---

From: Tero Kivinen <[email-masked]>
Date: Fri, 4 Jun 2021 17:26:59 +0300

Lars Eggert writes:
> Are there other mounted volumes? Maybe add :delegated to all of
> them, to see if that makes any difference? 

There is two mounted volumes, one for the home directory and another
for the database. Home directory is mounted with :cached on MacOs X
and with default on other environments. Database was always mounted
with default setting. 

> If I understand the DT docker setup correctly, nothing executes in
> the host (= everything executes in containers), so :delegated should
> always be a no-op or an improvement.

It might also be that my machine is just so slow that there is no
real difference on the mount options...

I will attach the diff that will add -D option to the docker/run, and
after that you can try:

./docker/run -c -D cached

or

./docker/run -c -D delegated

(or without -D option). The -c option forces :cached on the home
directory.

Index: docker/run
===================================================================
--- docker/run	(revision 19048)
+++ docker/run	(working copy)
@@ -75,8 +75,8 @@
 # Option parsing
 
 # Options
-shortopts=dhi:m:Mp:r:t:vVu:cC
-longopts=download-data,help,ietfdb-url=,mysqldata=,no-mysqldir,port=,docker-repo=,tag=,verbose,version,user=,cached,no-cached
+shortopts=dhi:m:Mp:r:t:vVu:cCD:
+longopts=download-data,help,ietfdb-url=,mysqldata=,no-mysqldir,port=,docker-repo=,tag=,verbose,version,user=,cached,no-cached,db-opt=
 
 # Default values
 MYSQLDIR=$parent/data/mysql
@@ -86,6 +86,7 @@
 DBURL=https://www.ietf.org/lib/dt/sprint/ietf_utf8.bin.tar.bz2
 WHO=$(whoami)
 CACHED=''
+DBOPT=''
 
 if   [ "$(uname)" = "Linux" ]; then
     args=$(getopt -o "$shortopts" --long "$longopts" -n '$program' -- $SV "$@")
@@ -108,6 +109,7 @@
 	-c| --cached)	CACHED=':cached';;	# Use cached disk access to reduce system load
 	-C| --no-cached)	CACHED='';;	# Use fully synchronized disk access
 	-d| --download-data) DOWNLOAD=1;;	# Download and set up the database files
+	-D| --db-opt)	DBOPT=":$2"; shift;;	# Use delegated/cached/etc disk access for db to reduce system load
 	-h| --help)	usage; exit;;		# Show this help, then exit
 	-f| --filedir)	FILEDIR=$2; shift;;	# Set the desired location of drafts, charters etc.
 	-i| --ietfdb-url)	DBURL=$2; shift;;	# Use an alternative database tarball URL
@@ -263,7 +265,7 @@
 	    "$REPO:$TAG" "$@"
     else
 	docker run -ti -p $PORT:8000 -v "$HOME:/home/$WHO$CACHED"                       \
-	    -v "$MYSQLDIR:/var/lib/mysql" -e USER="$WHO"                                \
+	    -v "$MYSQLDIR:/var/lib/mysql$DBOPT" -e USER="$WHO"                          \
 	    -e DATADIR="${parent#$HOME/}/data" -e CWD="${PWD#$HOME/}" -e TAG="$TAG"     \
 	    -e FILEDIR=${FILEDIR#$HOME} -e UID="$WHOUID" -e GID="$WHOGID"               \
 	    "$REPO:$TAG" "$@"
-- 
[email-masked]

---

From: Robert Sparks <[email-masked]>
Date: Fri, 4 Jun 2021 10:16:35 -0500

As I noted on the sprint, I got a pretty significant boost out of 
unchecking "Use gRPC FUSE for file sharing" in docker desktop's options.

I'm sure that comes with warts related to the legacy osxfs sharing 
mechanism, but those have not hurt me.

I am not on a recent enough os to try the new virtualization framework 
mentioned in the experimental section.

RjS

On 6/4/21 9:26 AM, Tero Kivinen wrote:
> _______________________________________________
> codesprints mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/codesprints

---

From: Robert Sparks <[email-masked]>
Date: Fri, 4 Jun 2021 10:28:29 -0500

> If I understand the DT docker setup correctly, nothing executes in the host (= everything executes in containers), so :delegated should always be a no-op or an improvement.

Well, it might be safe, but...

The current design of the container keeps the _source tree_ on the host 
filesystem and volume mounts it into the container so that it is 
convenient to use editors you have on the host on the code. Using 
:delegated here _could_ lead to surprising synchronization effects as 
the editor makes changes to those files from the host's perspective.

Similarly, the filesystem objects (drafts, reviews, photos, etc) are 
volume mounted in, and someone may refresh those using, say, rsync on 
the host.

No massive dangers there I think, but opportunity for confusion when 
sync doesn't look like you think it should.