Apptainer / Singularity insufficient build encapsulation #50

New issue

Open

opened 2024-06-11 17:27:22 +00:00 by mslw · 1 comment

mslw commented

2024-06-11 17:27:22 +00:00

(Migrated from github.com)

I can build the container image with apptainer (everything applies for singularity, too):

❱ apptainer build --fakeroot icf.sif singularity/icf.def
(...)
INFO:    Build complete: icf.sif

When, for some reason, I want to build the image again, it fails:

❱ apptainer build --fakeroot icf2.sif singularity/icf.def
(...)
2024-06-11T17:12:28+0000 [INFO    ] datalad_installer: git-annex is now installed at /tmp/dl-build-5n4aqld0/git-annex.linux/git-annex
+ cut -d : -f 1
+ cut -d = -f 2
+ mv /tmp/dl-build-c26wd0t_/git-annex.linux /tmp/dl-build-5n4aqld0/git-annex.linux /git-annex
mv: target '/git-annex': No such file or directory
FATAL:   While performing build: while running engine: exit status 1

The runs leave behind, in the host system's /tmp these files, produced by datalad-installer:

❱ ls /tmp | grep dl
dl-build-5n4aqld0
dl-build-c26wd0t_
dlinstaller_env.sh
❱ cat /tmp/dlinstaller_env.sh
export PATH=/tmp/dl-build-c26wd0t_/git-annex.linux:"$PATH"
export PATH=/tmp/dl-build-5n4aqld0/git-annex.linux:"$PATH"

Removing the /tmp/dlinstaller_env.sh file allows the new build to proceed:

❱ rm /tmp/dlinstaller_env.sh
❱ apptainer build --fakeroot icf2.sif singularity/icf.def
(...)
INFO:    Build complete: icf2.sif

Crucially, the same would hapen if, e.g. /tmp/dlinstaller_env.sh was already present for unrelated reasons (e,g. created by datalad-installer).

This is probably not very alarming (given that /tmp is usually wiped on reboot) but maybe we can improve on build encapsulation.

Note: datalad-installer appends to the /tmp/dlinstaller_env.sh file; the cut command used in the definition file to get the temporary directory produces two lines of output if the file has two lines.

Reported by Stefanie

I can build the container image with `apptainer` (everything applies for `singularity`, too): ``` ❱ apptainer build --fakeroot icf.sif singularity/icf.def (...) INFO: Build complete: icf.sif ``` When, for some reason, I want to build the image again, it fails: ``` ❱ apptainer build --fakeroot icf2.sif singularity/icf.def (...) 2024-06-11T17:12:28+0000 [INFO ] datalad_installer: git-annex is now installed at /tmp/dl-build-5n4aqld0/git-annex.linux/git-annex + cut -d : -f 1 + cut -d = -f 2 + mv /tmp/dl-build-c26wd0t_/git-annex.linux /tmp/dl-build-5n4aqld0/git-annex.linux /git-annex mv: target '/git-annex': No such file or directory FATAL: While performing build: while running engine: exit status 1 ``` The runs leave behind, *in the host system's `/tmp`* these files, produced by `datalad-installer`: ``` ❱ ls /tmp | grep dl dl-build-5n4aqld0 dl-build-c26wd0t_ dlinstaller_env.sh ❱ cat /tmp/dlinstaller_env.sh export PATH=/tmp/dl-build-c26wd0t_/git-annex.linux:"$PATH" export PATH=/tmp/dl-build-5n4aqld0/git-annex.linux:"$PATH" ``` Removing the `/tmp/dlinstaller_env.sh` file allows the new build to proceed: ``` ❱ rm /tmp/dlinstaller_env.sh ❱ apptainer build --fakeroot icf2.sif singularity/icf.def (...) INFO: Build complete: icf2.sif ``` Crucially, the same would hapen if, e.g. `/tmp/dlinstaller_env.sh` was already present for unrelated reasons (e,g. created by datalad-installer). This is probably not very alarming (given that `/tmp` is usually wiped on reboot) but maybe we can improve on build encapsulation. Note: `datalad-installer` *appends* to the `/tmp/dlinstaller_env.sh` file; the `cut` command used in the definition file to get the temporary directory produces two lines of output if the file has two lines. Reported by Stefanie

mslw commented

2024-07-02 16:38:59 +00:00

(Migrated from github.com)

Note: datalad-installer appends to the /tmp/dlinstaller_env.sh file; the cut command used in the definition file to get the temporary directory produces two lines of output if the file has two lines.

this should work:

tail -n 1 /tmp/dlinstaller_env.sh | cut -d = -f 2 | cut -d : -f 1

but I am also tempting to do what datalad-installer does:

2024-07-02T15:22:53+0000 [INFO    ] datalad_installer: Installing git-annex via snapshot
2024-07-02T15:22:53+0000 [INFO    ] datalad_installer: Downloading and extracting under /tmp/dl-build-xu2_movo/git-annex.linux
2024-07-02T15:22:53+0000 [INFO    ] datalad_installer: Downloading https://downloads.kitenet.net/git-annex/linux/current/git-annex-standalone-amd64.tar.gz
2024-07-02T15:23:05+0000 [INFO    ] datalad_installer: Running: tar -C /tmp/dl-build-xu2_movo -xzf /tmp/dl-build-xu2_movo/git-annex-standalone-amd64.tar.gz
2024-07-02T15:23:07+0000 [INFO    ] datalad_installer: git-annex is now installed at /tmp/dl-build-xu2_movo/git-annex.linux/git-annex

(plus, probably, setting the env variable explicitly)

> Note: datalad-installer appends to the /tmp/dlinstaller_env.sh file; the cut command used in the definition file to get the temporary directory produces two lines of output if the file has two lines. this should work: ``` tail -n 1 /tmp/dlinstaller_env.sh | cut -d = -f 2 | cut -d : -f 1 ``` but I am also tempting to do what datalad-installer does: ``` 2024-07-02T15:22:53+0000 [INFO ] datalad_installer: Installing git-annex via snapshot 2024-07-02T15:22:53+0000 [INFO ] datalad_installer: Downloading and extracting under /tmp/dl-build-xu2_movo/git-annex.linux 2024-07-02T15:22:53+0000 [INFO ] datalad_installer: Downloading https://downloads.kitenet.net/git-annex/linux/current/git-annex-standalone-amd64.tar.gz 2024-07-02T15:23:05+0000 [INFO ] datalad_installer: Running: tar -C /tmp/dl-build-xu2_movo -xzf /tmp/dl-build-xu2_movo/git-annex-standalone-amd64.tar.gz 2024-07-02T15:23:07+0000 [INFO ] datalad_installer: git-annex is now installed at /tmp/dl-build-xu2_movo/git-annex.linux/git-annex ``` (plus, probably, setting the env variable explicitly)