[BUG]: Jobs fail with new `condor_submit_dag` changes #233

Merged

synchon merged 2 commits from fix/htcondor-submit-fix into main

2023-07-27 10:00:30 +00:00

synchon commented

2023-07-26 09:30:54 +00:00

(Migrated from github.com)

Is there an existing issue for this?

I have searched the existing issues

Current Behavior

Datalad fails to work as it tries to make a directory in /.

This is likely related to the new features of HTCondor: https://htcondor.readthedocs.io/en/latest/version-history/feature-versions-10-x.html#version-10-4-0

Indeed, the issue appears only when a job us submitted using a DAG (junifer queue)

Submitting an interactive job and then using junifer run works well.

Expected Behavior

I expect junifer to run wihout issue.

Steps To Reproduce

Install junifer in an enviroment in juseless
Create a yaml with the following content

workdir: /tmp

datagrabber:
  kind: DataladHCP1200
  ica_fix: true
  tasks:
    - REST1
markers:
  - name: ALFF-Power2011
    kind: ALFFSpheres
    coords: "Power"
    fractional: false
    use_afni: true
    highpass: 0.01
    lowpass: 0.08
    tr: 0.72
    method: mean
    radius: 5 
    masks: 
      - fetch_icbm152_brain_gm_mask
storage:
  kind: HDF5FeatureStorage
  uri: /data/group/appliedml/fraimondo/junifer_test/test2.sqlite
queue:
  jobname: TestJuniferQueue
  kind: HTCondor
  env:
    kind: conda
    name: junifer
  pre_run: |
    source /data/group/appliedml/tools/afni/afni.sh
  mem: 20G
  disk: 5G
  verbose: 8

junifer queue test.yaml --element 100206,REST1,LR --verbose debug --overwrite --submit
See the failure

Workarounds:

Go the junifer_jobs/TestJuniferQueue directory and submit the dag again using condor_submit_dag -include_env "HOME" TestJuniferQueue.dag

This time it will run, but AFNI will fail. Since we don't have any info on the output of AFNI (due to a bug in the afni output capture), I don't know why this is failing.

Environment

junifer:
  version: 0.0.3.dev72
python:
  version: 3.11.3
  implementation: CPython
dependencies:
  click: 8.1.3
  numpy: 1.23.5
  datalad: 0.18.2+59.gc5054cb91
  pandas: 1.5.3
  nibabel: 4.0.2
  nilearn: 0.10.0
  sqlalchemy: 1.4.48
  ruamel.yaml: 0.17.31
system:
  platform: Linux-6.1.0-9-amd64-x86_64-with-glibc2.36
environment:
  LC_CTYPE: en_US.UTF-8
  PATH: 
    /home/fraimondo/miniconda3/envs/junifer/bin:/home/fraimondo/miniconda3/condabin:/home/fraimondo/.dotfiles/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/X11R6/bin:/usr/local/games:/usr/games

Relevant log output

Traceback (most recent call last):
  File "/home/fraimondo/miniconda3/envs/junifer/bin/junifer", line 5, in <module>
    from junifer.api.cli import cli
  File "/home/fraimondo/dev/tbox/junifer/junifer/__init__.py", line 7, in <module>
    from . import (
  File "/home/fraimondo/dev/tbox/junifer/junifer/api/__init__.py", line 7, in <module>
    from . import decorators
  File "/home/fraimondo/dev/tbox/junifer/junifer/api/decorators.py", line 10, in <module>
    from ..pipeline.registry import register
  File "/home/fraimondo/dev/tbox/junifer/junifer/pipeline/__init__.py", line 6, in <module>
    from . import registry
  File "/home/fraimondo/dev/tbox/junifer/junifer/pipeline/registry.py", line 10, in <module>
    from ..utils.logging import logger, raise_error
  File "/home/fraimondo/dev/tbox/junifer/junifer/utils/__init__.py", line 8, in <module>
    from .logging import configure_logging, logger, raise_error, warn_with_log
  File "/home/fraimondo/dev/tbox/junifer/junifer/utils/logging.py", line 15, in <module>
    import datalad
  File "/home/fraimondo/dev/tbox/datalad/datalad/__init__.py", line 113, in <module>
    cfg = ConfigManager()
          ^^^^^^^^^^^^^^^
  File "/home/fraimondo/dev/tbox/datalad/datalad/config.py", line 401, in __init__
    self.reload(force=True)
  File "/home/fraimondo/dev/tbox/datalad/datalad/config.py", line 476, in reload
    self._stores[store_id] = self._reload(runargs)
                             ^^^^^^^^^^^^^^^^^^^^^
  File "/home/fraimondo/dev/tbox/datalad/datalad/config.py", line 504, in _reload
    stdout, stderr = self._run(
                     ^^^^^^^^^^
  File "/home/fraimondo/dev/tbox/datalad/datalad/config.py", line 885, in _run
    out = self._runner.run(self._config_cmd + args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fraimondo/dev/tbox/datalad/datalad/runner/runner.py", line 223, in run
    raise CommandError(
datalad.runner.exception.CommandError: CommandError: 'git --git-dir=/dev/null config -z -l --show-origin' failed with exitcode 1 [err: 'mkdir: cannot create directory ‘/.cache’: Permission denied
Unable to write to /.cache/git-annex/locales; can't continue!']

Anything else?

No response

### Is there an existing issue for this? - [X] I have searched the existing issues ### Current Behavior Datalad fails to work as it tries to make a directory in `/`. This is likely related to the new features of HTCondor: https://htcondor.readthedocs.io/en/latest/version-history/feature-versions-10-x.html#version-10-4-0 Indeed, the issue appears only when a job us submitted using a DAG (`junifer queue`) Submitting an interactive job and then using `junifer run` works well. ### Expected Behavior I expect junifer to run wihout issue. ### Steps To Reproduce 1. Install junifer in an enviroment in juseless 2. Create a yaml with the following content ``` workdir: /tmp datagrabber: kind: DataladHCP1200 ica_fix: true tasks: - REST1 markers: - name: ALFF-Power2011 kind: ALFFSpheres coords: "Power" fractional: false use_afni: true highpass: 0.01 lowpass: 0.08 tr: 0.72 method: mean radius: 5 masks: - fetch_icbm152_brain_gm_mask storage: kind: HDF5FeatureStorage uri: /data/group/appliedml/fraimondo/junifer_test/test2.sqlite queue: jobname: TestJuniferQueue kind: HTCondor env: kind: conda name: junifer pre_run: | source /data/group/appliedml/tools/afni/afni.sh mem: 20G disk: 5G verbose: 8 ``` 3. `junifer queue test.yaml --element 100206,REST1,LR --verbose debug --overwrite --submit` 4. See the failure Workarounds: 1) Go the `junifer_jobs/TestJuniferQueue` directory and submit the dag again using `condor_submit_dag -include_env "HOME" TestJuniferQueue.dag ` This time it will run, but AFNI will fail. Since we don't have any info on the output of AFNI (due to a bug in the afni output capture), I don't know why this is failing. ### Environment ```markdown junifer: version: 0.0.3.dev72 python: version: 3.11.3 implementation: CPython dependencies: click: 8.1.3 numpy: 1.23.5 datalad: 0.18.2+59.gc5054cb91 pandas: 1.5.3 nibabel: 4.0.2 nilearn: 0.10.0 sqlalchemy: 1.4.48 ruamel.yaml: 0.17.31 system: platform: Linux-6.1.0-9-amd64-x86_64-with-glibc2.36 environment: LC_CTYPE: en_US.UTF-8 PATH: /home/fraimondo/miniconda3/envs/junifer/bin:/home/fraimondo/miniconda3/condabin:/home/fraimondo/.dotfiles/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/X11R6/bin:/usr/local/games:/usr/games ``` ### Relevant log output ```shell Traceback (most recent call last): File "/home/fraimondo/miniconda3/envs/junifer/bin/junifer", line 5, in <module> from junifer.api.cli import cli File "/home/fraimondo/dev/tbox/junifer/junifer/__init__.py", line 7, in <module> from . import ( File "/home/fraimondo/dev/tbox/junifer/junifer/api/__init__.py", line 7, in <module> from . import decorators File "/home/fraimondo/dev/tbox/junifer/junifer/api/decorators.py", line 10, in <module> from ..pipeline.registry import register File "/home/fraimondo/dev/tbox/junifer/junifer/pipeline/__init__.py", line 6, in <module> from . import registry File "/home/fraimondo/dev/tbox/junifer/junifer/pipeline/registry.py", line 10, in <module> from ..utils.logging import logger, raise_error File "/home/fraimondo/dev/tbox/junifer/junifer/utils/__init__.py", line 8, in <module> from .logging import configure_logging, logger, raise_error, warn_with_log File "/home/fraimondo/dev/tbox/junifer/junifer/utils/logging.py", line 15, in <module> import datalad File "/home/fraimondo/dev/tbox/datalad/datalad/__init__.py", line 113, in <module> cfg = ConfigManager() ^^^^^^^^^^^^^^^ File "/home/fraimondo/dev/tbox/datalad/datalad/config.py", line 401, in __init__ self.reload(force=True) File "/home/fraimondo/dev/tbox/datalad/datalad/config.py", line 476, in reload self._stores[store_id] = self._reload(runargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/fraimondo/dev/tbox/datalad/datalad/config.py", line 504, in _reload stdout, stderr = self._run( ^^^^^^^^^^ File "/home/fraimondo/dev/tbox/datalad/datalad/config.py", line 885, in _run out = self._runner.run(self._config_cmd + args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/fraimondo/dev/tbox/datalad/datalad/runner/runner.py", line 223, in run raise CommandError( datalad.runner.exception.CommandError: CommandError: 'git --git-dir=/dev/null config -z -l --show-origin' failed with exitcode 1 [err: 'mkdir: cannot create directory ‘/.cache’: Permission denied Unable to write to /.cache/git-annex/locales; can't continue!'] ``` ### Anything else? _No response_

github-actions[bot] commented

2023-07-26 09:37:55 +00:00

(Migrated from github.com)

PR Preview Action v1.4.4
Preview removed because the pull request was closed.
2023-07-27 10:06 UTC

[PR Preview Action](https://github.com/rossjrw/pr-preview-action) v1.4.4 :---: Preview removed because the pull request was closed. 2023-07-27 10:06 UTC

codecov[bot] commented

2023-07-26 09:48:52 +00:00

(Migrated from github.com)

Codecov Report

Merging #233 (98c1c73) into main (550e620) will not change coverage.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##             main     #233   +/-   ##
=======================================
  Coverage   93.10%   93.10%           
=======================================
  Files          84       84           
  Lines        3714     3714           
  Branches      722      722           
=======================================
  Hits         3458     3458           
  Misses        160      160           
  Partials       96       96

Flag	Coverage Δ
docs	`100.00% <ø> (ø)`
junifer	`93.10% <100.00%> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Changed	Coverage Δ
junifer/api/functions.py	`96.23% <100.00%> (ø)`

## [Codecov](https://app.codecov.io/gh/juaml/junifer/pull/233?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml) Report > Merging [#233](https://app.codecov.io/gh/juaml/junifer/pull/233?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml) (98c1c73) into [main](https://app.codecov.io/gh/juaml/junifer/commit/550e620a3cf726be9f8294fdd1a230ef5bd4bc0e?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml) (550e620) will **not change** coverage. > The diff coverage is `100.00%`. [![Impacted file tree graph](https://app.codecov.io/gh/juaml/junifer/pull/233/graphs/tree.svg?width=650&height=150&src=pr&token=5H21JuZXMw&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml)](https://app.codecov.io/gh/juaml/junifer/pull/233?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml) ```diff @@ Coverage Diff @@ ## main #233 +/- ## ======================================= Coverage 93.10% 93.10% ======================================= Files 84 84 Lines 3714 3714 Branches 722 722 ======================================= Hits 3458 3458 Misses 160 160 Partials 96 96 ``` | Flag | Coverage Δ | | |---|---|---| | docs | `100.00% <ø> (ø)` | | | junifer | `93.10% <100.00%> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml#carryforward-flags-in-the-pull-request-comment) to find out more. | [Files Changed](https://app.codecov.io/gh/juaml/junifer/pull/233?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml) | Coverage Δ | | |---|---|---| | [junifer/api/functions.py](https://app.codecov.io/gh/juaml/junifer/pull/233?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml#diff-anVuaWZlci9hcGkvZnVuY3Rpb25zLnB5) | `96.23% <100.00%> (ø)` | |

fraimondo (Migrated from github.com) approved these changes

2023-07-27 09:57:57 +00:00