juaml/junifer
Watch
3
0
Fork
You've already forked junifer
 0
Code Issues 14 Pull requests 2 Packages Activity Actions

[BUG]: Merge parcellations fail with extra dimensions #331

Merged
synchon merged 3 commits from fix/extra-parc-dim-in-merge into main 2024-05-06 10:28:28 +00:00
Conversation 5 Commits 3 Files changed 2 +4 -1
synchon commented 2024-04-15 10:49:04 +00:00 (Migrated from github.com)
Copy link

Is there an existing issue for this?

  • I have searched the existing issues

Current Behavior

Running a simple FC marker with two parcellations fails:

/home/fraimondo/miniconda3/envs/neurodc/lib/python3.11/site-packages/junifer/data/parcellations.py:1785: RuntimeWarning: The parcellations have different resolutions!Resampling all parcellations to the first one in the list.
  warn_with_log(
Traceback (most recent call last):
  File "/home/fraimondo/miniconda3/envs/neurodc/bin/junifer", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File "/home/fraimondo/miniconda3/envs/neurodc/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fraimondo/miniconda3/envs/neurodc/lib/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/home/fraimondo/miniconda3/envs/neurodc/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fraimondo/miniconda3/envs/neurodc/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fraimondo/miniconda3/envs/neurodc/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fraimondo/miniconda3/envs/neurodc/lib/python3.11/site-packages/junifer/api/cli.py", line 223, in run
    api_run(
  File "/home/fraimondo/miniconda3/envs/neurodc/lib/python3.11/site-packages/junifer/api/functions.py", line 176, in run
    mc.fit(datagrabber_object[t_element])
  File "/home/fraimondo/miniconda3/envs/neurodc/lib/python3.11/site-packages/junifer/markers/collection.py", line 101, in fit
    m_value = marker.fit_transform(data, storage=self._storage)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fraimondo/miniconda3/envs/neurodc/lib/python3.11/site-packages/junifer/pipeline/pipeline_step_mixin.py", line 229, in fit_transform
    return self._fit_transform(input=input, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fraimondo/miniconda3/envs/neurodc/lib/python3.11/site-packages/junifer/markers/base.py", line 201, in _fit_transform
    t_out = self.compute(input=t_input, extra_input=extra_input)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fraimondo/miniconda3/envs/neurodc/lib/python3.11/site-packages/junifer/markers/functional_connectivity/functional_connectivity_base.py", line 138, in compute
    aggregation = self.aggregate(input, extra_input=extra_input)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fraimondo/miniconda3/envs/neurodc/lib/python3.11/site-packages/junifer/markers/functional_connectivity/functional_connectivity_parcels.py", line 102, in aggregate
    return parcel_aggregation.compute(input, extra_input=extra_input)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fraimondo/miniconda3/envs/neurodc/lib/python3.11/site-packages/junifer/markers/parcel_aggregation.py", line 191, in compute
    parcellation_img, labels = get_parcellation(
                               ^^^^^^^^^^^^^^^^^
  File "/home/fraimondo/miniconda3/envs/neurodc/lib/python3.11/site-packages/junifer/data/parcellations.py", line 348, in get_parcellation
    resampled_parcellation_img, labels = merge_parcellations(
                                         ^^^^^^^^^^^^^^^^^^^^
  File "/home/fraimondo/miniconda3/envs/neurodc/lib/python3.11/site-packages/junifer/data/parcellations.py", line 1805, in merge_parcellations
    parc_data[parc_data == 0] += t_parc_data[parc_data == 0]
                                 ~~~~~~~~~~~^^^^^^^^^^^^^^^^
IndexError: too many indices for array: array is 3-dimensional, but 4 were indexed

Expected Behavior

to run withour error.

Steps To Reproduce

  1. Install latest (--pre) junifer from pip.
  2. run this yaml:
workdir: /tmp

datagrabber:
  kind: DataladAOMICID1000
  types: 
    - BOLD
    - BOLD_confounds
    - BOLD_mask
preprocess:
    kind: fMRIPrepConfoundRemover
    detrend: true
    standardize: true
    strategy:
        motion: full
        wm_csf: full
        global_signal: full
    low_pass: 0.08
    high_pass: 0.01
    masks:
      - inherit
      - compute_epi_mask
      - threshold: 0
markers:
  - name: parccortical-Schaefer100x17-TianxS2x3T
    kind: FunctionalConnectivityParcels
    parcellation: 
        - Schaefer100x17
        - TianxS2x3TxMNInonlinear2009cAsym
    cor_method: correlation
    cor_method_params:
      empirical: true
    masks:
      - inherit

storage:
  kind: HDF5FeatureStorage
  uri: "/data/group/riseml/fraimondo/ds003097_FC/ds003097_FC.hdf5"
queue:
  jobname: aomic_fc
  kind: HTCondor
  env:
    kind: conda
    name: neurodc
  mem: 8G
  disk: 5G
  verbose: info
  pre_run: |
    source /data/group/appliedml/tools/afni_23.1.10/afni.sh
    source /data/group/appliedml/tools/ants_2.5.0/ants.sh
    source /data/group/appliedml/tools/fsl_6.0.4-patched2/fsl.sh

Environment

junifer:
  version: 0.0.5.dev27
python:
  version: 3.11.8
  implementation: CPython
dependencies:
  click: 8.1.7
  numpy: 1.26.4
  scipy: 1.11.4
  datalad: 0.19.6
  pandas: 2.1.4
  nibabel: 5.2.1
  nilearn: 0.10.2
  sqlalchemy: 2.0.29
  ruamel.yaml: 0.17.40
  httpx: 0.26.0
  tqdm: 4.66.1
  templateflow: 24.2.0
  looseversion: None
system:
  platform: Linux-6.6.13+bpo-amd64-x86_64-with-glibc2.36
environment:
  LC_CTYPE: en_US.UTF-8
  PATH: 
    /home/fraimondo/miniconda3/envs/neurodc/bin:/home/fraimondo/miniconda3/condabin:/home/fraimondo/.dotfiles/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/X11R6/bin:/usr/local/games:/usr/games

Relevant log output

Activating neurodc
Sourcing pre_run.sh
Script dir from afni.sh: /data/group/appliedml/tools/afni_23.1.10
Path after sourcing afni.sh: /data/group/appliedml/tools/afni_23.1.10/binaries:/home/fraimondo/miniconda3/envs/neurodc/bin:/home/fraimondo/miniconda3/condabin:/home/fraimondo/miniconda3/envs/neurodc/bin:/home/fraimondo/miniconda3/condabin:/home/fraimondo/.dotfiles/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/X11R6/bin:/usr/local/games:/usr/games
Script dir from ants.sh: /data/group/appliedml/tools/ants_2.5.0
Path after sourcing ants.sh: /data/group/appliedml/tools/ants_2.5.0/binaries:/data/group/appliedml/tools/afni_23.1.10/binaries:/home/fraimondo/miniconda3/envs/neurodc/bin:/home/fraimondo/miniconda3/condabin:/home/fraimondo/miniconda3/envs/neurodc/bin:/home/fraimondo/miniconda3/condabin:/home/fraimondo/.dotfiles/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/X11R6/bin:/usr/local/games:/usr/games
Script dir from fsl.sh: /data/group/appliedml/tools/fsl_6.0.4-patched2
Path after sourcing fsl.sh: /data/group/appliedml/tools/fsl_6.0.4-patched2/binaries:/data/group/appliedml/tools/ants_2.5.0/binaries:/data/group/appliedml/tools/afni_23.1.10/binaries:/home/fraimondo/miniconda3/envs/neurodc/bin:/home/fraimondo/miniconda3/condabin:/home/fraimondo/miniconda3/envs/neurodc/bin:/home/fraimondo/miniconda3/condabin:/home/fraimondo/.dotfiles/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/X11R6/bin:/usr/local/games:/usr/games
Running junifer run /home/fraimondo/dev/scratch/neurodc/junifer_jobs/aomic_fc/config.yaml --verbose info --element sub-0014 in conda environment
2024-04-12 22:25:36,051 - JUNIFER - INFO - ===== Lib Versions =====
2024-04-12 22:25:36,051 - JUNIFER - INFO - numpy: 1.26.4
2024-04-12 22:25:36,051 - JUNIFER - INFO - scipy: 1.11.4
2024-04-12 22:25:36,051 - JUNIFER - INFO - pandas: 2.1.4
2024-04-12 22:25:36,051 - JUNIFER - INFO - nilearn: 0.10.2
2024-04-12 22:25:36,051 - JUNIFER - INFO - nibabel: 5.2.1
2024-04-12 22:25:36,051 - JUNIFER - INFO - junifer: 0.0.5.dev27
2024-04-12 22:25:36,051 - JUNIFER - INFO - ========================
2024-04-12 22:25:36,051 - JUNIFER - INFO - Parsing yaml file: /home/fraimondo/dev/scratch/neurodc/junifer_jobs/aomic_fc/config.yaml
2024-04-12 22:25:36,058 - JUNIFER - INFO - `datadir` is None, creating a temporary directory
2024-04-12 22:25:36,058 - JUNIFER - INFO - `datadir` set to /tmp/tmpgd5w41ln/datadir
2024-04-12 22:25:36,289 - JUNIFER - INFO - Validating Marker Collection
2024-04-12 22:25:36,289 - JUNIFER - INFO - DataGrabber output type: ['BOLD', 'BOLD_confounds', 'BOLD_mask']
2024-04-12 22:25:36,289 - JUNIFER - INFO - Validating Data Reader:
2024-04-12 22:25:36,289 - JUNIFER - INFO - Data Reader output type: ['BOLD', 'BOLD_confounds', 'BOLD_mask']
2024-04-12 22:25:36,289 - JUNIFER - INFO - Validating Preprocessor: fMRIPrepConfoundRemover
2024-04-12 22:25:36,289 - JUNIFER - INFO - Preprocess output type: ['BOLD']
2024-04-12 22:25:36,289 - JUNIFER - INFO - Validating Marker: parccortical-Schaefer100x17-TianxS2x3T
2024-04-12 22:25:36,386 - JUNIFER - INFO - Marker output type: ['matrix']
2024-04-12 22:25:36,386 - JUNIFER - INFO - Validating storage for parccortical-Schaefer100x17-TianxS2x3T
2024-04-12 22:27:39,404 - JUNIFER - INFO - Getting element sub-0014
2024-04-12 22:28:45,106 - JUNIFER - INFO - Fitting pipeline
2024-04-12 22:28:45,106 - JUNIFER - INFO - Reading BOLD from /tmp/tmpgd5w41ln/datadir/derivatives/fmriprep/sub-0014/func/sub-0014_task-moviewatching_space-MNI152NLin2009cAsym_desc-preproc_bold.nii.gz
2024-04-12 22:28:45,106 - JUNIFER - INFO - BOLD is type NIFTI
2024-04-12 22:28:45,107 - JUNIFER - INFO - Reading BOLD_confounds from /tmp/tmpgd5w41ln/datadir/derivatives/fmriprep/sub-0014/func/sub-0014_task-moviewatching_desc-confounds_regressors.tsv
2024-04-12 22:28:45,107 - JUNIFER - INFO - BOLD_confounds is type TSV
2024-04-12 22:28:45,123 - JUNIFER - INFO - Reading BOLD_mask from /tmp/tmpgd5w41ln/datadir/derivatives/fmriprep/sub-0014/func/sub-0014_task-moviewatching_space-MNI152NLin2009cAsym_desc-brain_mask.nii.gz
2024-04-12 22:28:45,123 - JUNIFER - INFO - BOLD_mask is type NIFTI
2024-04-12 22:28:45,124 - JUNIFER - INFO - Preprocessing data with fMRIPrepConfoundRemover
2024-04-12 22:28:45,124 - JUNIFER - INFO - Preprocessing BOLD
2024-04-12 22:28:47,902 - JUNIFER - INFO - No `t_r` specified, using t_r from NIfTI header
2024-04-12 22:28:47,903 - JUNIFER - INFO - Read t_r from NIfTI header: 2.200000047683716
2024-04-12 22:28:48,266 - JUNIFER - INFO - Cleaning image using nilearn
2024-04-12 22:28:54,670 - JUNIFER - INFO - Fitting marker parccortical-Schaefer100x17-TianxS2x3T
2024-04-12 22:28:54,778 - JUNIFER - INFO - Computing BOLD
2024-04-12 22:28:54,779 - JUNIFER - INFO - Fetching one of Schaefer parcellations.
2024-04-12 22:28:54,779 - JUNIFER - INFO - Parcellation parameters:
2024-04-12 22:28:54,779 - JUNIFER - INFO - 	resolution: 3.0
2024-04-12 22:28:54,779 - JUNIFER - INFO - 	n_rois: 100
2024-04-12 22:28:54,779 - JUNIFER - INFO - 	yeo_networks: 17
2024-04-12 22:28:54,781 - JUNIFER - INFO - Loading parcellation /home/fraimondo/junifer/data/parcellations/schaefer_2018/Schaefer2018_100Parcels_17Networks_order_FSLMNI152_2mm.nii.gz
2024-04-12 22:28:54,816 - JUNIFER - INFO - Found existing xfm file for MNI152NLin6Asym to MNI152NLin2009cAsym at /home/fraimondo/junifer/data/xfms/MNI152NLin6Asym_to_MNI152NLin2009cAsym/MNI152NLin6Asym_to_MNI152NLin2009cAsym_Composite.h5
2024-04-12 22:28:55,773 - JUNIFER - INFO - Downloading template MNI152NLin2009cAsym in resolution 2
2024-04-12 22:28:55,892 - JUNIFER - INFO - antsApplyTransforms command to be executed:
antsApplyTransforms -d 3 -e 3 -n 'GenericLabel[NearestNeighbor]' -i /tmp/tmpol3919xi/parcellationsupbm3qm_/Schaefer100x17.nii.gz -r /tmp/tmpol3919xi/parcellationsupbm3qm_/MNI152NLin2009cAsym_T1w_3.0.nii.gz -t /home/fraimondo/junifer/data/xfms/MNI152NLin6Asym_to_MNI152NLin2009cAsym/MNI152NLin6Asym_to_MNI152NLin2009cAsym_Composite.h5 -o /tmp/tmpyk7pw003/parcellationskeha8hp0/Schaefer100x17_warped_from_MNI152NLin6Asym_to_MNI152NLin2009cAsym.nii.gz
2024-04-12 22:29:00,187 - JUNIFER - INFO - antsApplyTransforms command succeeded with the following output:
b'/tmp/tmpol3919xi/parcellationsupbm3qm_/Schaefer100x17.nii.gz is a file\n/tmp/tmpol3919xi/parcellationsupbm3qm_/MNI152NLin2009cAsym_T1w_3.0.nii.gz is a file\n/home/fraimondo/junifer/data/xfms/MNI152NLin6Asym_to_MNI152NLin2009cAsym/MNI152NLin6Asym_to_MNI152NLin2009cAsym_Composite.h5 is a file\n/tmp/tmpyk7pw003/parcellationskeha8hp0/Schaefer100x17_warped_from_MNI152NLin6Asym_to_MNI152NLin2009cAsym.nii.gz is a prefix\nSingularity args: --bind /tmp/tmpol3919xi/parcellationsupbm3qm_:/data/mount_1 --bind /tmp/tmpol3919xi/parcellationsupbm3qm_:/data/mount_2 --bind /home/fraimondo/junifer/data/xfms/MNI152NLin6Asym_to_MNI152NLin2009cAsym:/data/mount_3 --bind /tmp/tmpyk7pw003/parcellationskeha8hp0:/data/mount_4\nCorrected args for ANTS: antsApplyTransforms -d 3 -e 3 -n GenericLabel[NearestNeighbor] -i /data/mount_1/Schaefer100x17.nii.gz -r /data/mount_2/MNI152NLin2009cAsym_T1w_3.0.nii.gz -t /data/mount_3/MNI152NLin6Asym_to_MNI152NLin2009cAsym_Composite.h5 -o /data/mount_4/Schaefer100x17_warped_from_MNI152NLin6Asym_to_MNI152NLin2009cAsym.nii.gz\nRunning command: /data/group/appliedml/tools/ants_2.5.0/singularity_cmd exec --cleanenv --bind /tmp/tmpol3919xi/parcellationsupbm3qm_:/data/mount_1 --bind /tmp/tmpol3919xi/parcellationsupbm3qm_:/data/mount_2 --bind /home/fraimondo/junifer/data/xfms/MNI152NLin6Asym_to_MNI152NLin2009cAsym:/data/mount_3 --bind /tmp/tmpyk7pw003/parcellationskeha8hp0:/data/mount_4 /data/group/appliedml/tools/ants_2.5.0/ants_v2.5.0.sif antsApplyTransforms -d 3 -e 3 -n GenericLabel[NearestNeighbor] -i /data/mount_1/Schaefer100x17.nii.gz -r /data/mount_2/MNI152NLin2009cAsym_T1w_3.0.nii.gz -t /data/mount_3/MNI152NLin6Asym_to_MNI152NLin2009cAsym_Composite.h5 -o /data/mount_4/Schaefer100x17_warped_from_MNI152NLin6Asym_to_MNI152NLin2009cAsym.nii.gz\n'
2024-04-12 22:29:00,249 - JUNIFER - INFO - Fetching one of Tian parcellations.
2024-04-12 22:29:00,249 - JUNIFER - INFO - Parcellation parameters:
2024-04-12 22:29:00,249 - JUNIFER - INFO - 	resolution: 3.0
2024-04-12 22:29:00,249 - JUNIFER - INFO - 	scale: 2
2024-04-12 22:29:00,249 - JUNIFER - INFO - 	space: MNI152NLin2009cAsym
2024-04-12 22:29:00,249 - JUNIFER - INFO - 	magneticfield: 3T
2024-04-12 22:29:00,251 - JUNIFER - INFO - Loading parcellation /home/fraimondo/junifer/data/parcellations/Tian2020MSA_v1.1/3T/Subcortex-Only/Tian_Subcortex_S2_3T_2009cAsym.nii.gz
2024-04-12 22:29:00,334 - JUNIFER - WARNING - The parcellations have different resolutions!Resampling all parcellations to the first one in the list.

Anything else?

No response

### Is there an existing issue for this? - [X] I have searched the existing issues ### Current Behavior Running a simple FC marker with two parcellations fails: ``` /home/fraimondo/miniconda3/envs/neurodc/lib/python3.11/site-packages/junifer/data/parcellations.py:1785: RuntimeWarning: The parcellations have different resolutions!Resampling all parcellations to the first one in the list. warn_with_log( Traceback (most recent call last): File "/home/fraimondo/miniconda3/envs/neurodc/bin/junifer", line 8, in <module> sys.exit(cli()) ^^^^^ File "/home/fraimondo/miniconda3/envs/neurodc/lib/python3.11/site-packages/click/core.py", line 1157, in __call__ return self.main(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/fraimondo/miniconda3/envs/neurodc/lib/python3.11/site-packages/click/core.py", line 1078, in main rv = self.invoke(ctx) ^^^^^^^^^^^^^^^^ File "/home/fraimondo/miniconda3/envs/neurodc/lib/python3.11/site-packages/click/core.py", line 1688, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/fraimondo/miniconda3/envs/neurodc/lib/python3.11/site-packages/click/core.py", line 1434, in invoke return ctx.invoke(self.callback, **ctx.params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/fraimondo/miniconda3/envs/neurodc/lib/python3.11/site-packages/click/core.py", line 783, in invoke return __callback(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/fraimondo/miniconda3/envs/neurodc/lib/python3.11/site-packages/junifer/api/cli.py", line 223, in run api_run( File "/home/fraimondo/miniconda3/envs/neurodc/lib/python3.11/site-packages/junifer/api/functions.py", line 176, in run mc.fit(datagrabber_object[t_element]) File "/home/fraimondo/miniconda3/envs/neurodc/lib/python3.11/site-packages/junifer/markers/collection.py", line 101, in fit m_value = marker.fit_transform(data, storage=self._storage) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/fraimondo/miniconda3/envs/neurodc/lib/python3.11/site-packages/junifer/pipeline/pipeline_step_mixin.py", line 229, in fit_transform return self._fit_transform(input=input, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/fraimondo/miniconda3/envs/neurodc/lib/python3.11/site-packages/junifer/markers/base.py", line 201, in _fit_transform t_out = self.compute(input=t_input, extra_input=extra_input) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/fraimondo/miniconda3/envs/neurodc/lib/python3.11/site-packages/junifer/markers/functional_connectivity/functional_connectivity_base.py", line 138, in compute aggregation = self.aggregate(input, extra_input=extra_input) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/fraimondo/miniconda3/envs/neurodc/lib/python3.11/site-packages/junifer/markers/functional_connectivity/functional_connectivity_parcels.py", line 102, in aggregate return parcel_aggregation.compute(input, extra_input=extra_input) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/fraimondo/miniconda3/envs/neurodc/lib/python3.11/site-packages/junifer/markers/parcel_aggregation.py", line 191, in compute parcellation_img, labels = get_parcellation( ^^^^^^^^^^^^^^^^^ File "/home/fraimondo/miniconda3/envs/neurodc/lib/python3.11/site-packages/junifer/data/parcellations.py", line 348, in get_parcellation resampled_parcellation_img, labels = merge_parcellations( ^^^^^^^^^^^^^^^^^^^^ File "/home/fraimondo/miniconda3/envs/neurodc/lib/python3.11/site-packages/junifer/data/parcellations.py", line 1805, in merge_parcellations parc_data[parc_data == 0] += t_parc_data[parc_data == 0] ~~~~~~~~~~~^^^^^^^^^^^^^^^^ IndexError: too many indices for array: array is 3-dimensional, but 4 were indexed ``` ### Expected Behavior to run withour error. ### Steps To Reproduce 1. Install latest (--pre) junifer from pip. 2. run this yaml: ``` yaml workdir: /tmp datagrabber: kind: DataladAOMICID1000 types: - BOLD - BOLD_confounds - BOLD_mask preprocess: kind: fMRIPrepConfoundRemover detrend: true standardize: true strategy: motion: full wm_csf: full global_signal: full low_pass: 0.08 high_pass: 0.01 masks: - inherit - compute_epi_mask - threshold: 0 markers: - name: parccortical-Schaefer100x17-TianxS2x3T kind: FunctionalConnectivityParcels parcellation: - Schaefer100x17 - TianxS2x3TxMNInonlinear2009cAsym cor_method: correlation cor_method_params: empirical: true masks: - inherit storage: kind: HDF5FeatureStorage uri: "/data/group/riseml/fraimondo/ds003097_FC/ds003097_FC.hdf5" queue: jobname: aomic_fc kind: HTCondor env: kind: conda name: neurodc mem: 8G disk: 5G verbose: info pre_run: | source /data/group/appliedml/tools/afni_23.1.10/afni.sh source /data/group/appliedml/tools/ants_2.5.0/ants.sh source /data/group/appliedml/tools/fsl_6.0.4-patched2/fsl.sh ``` ### Environment ```markdown junifer: version: 0.0.5.dev27 python: version: 3.11.8 implementation: CPython dependencies: click: 8.1.7 numpy: 1.26.4 scipy: 1.11.4 datalad: 0.19.6 pandas: 2.1.4 nibabel: 5.2.1 nilearn: 0.10.2 sqlalchemy: 2.0.29 ruamel.yaml: 0.17.40 httpx: 0.26.0 tqdm: 4.66.1 templateflow: 24.2.0 looseversion: None system: platform: Linux-6.6.13+bpo-amd64-x86_64-with-glibc2.36 environment: LC_CTYPE: en_US.UTF-8 PATH: /home/fraimondo/miniconda3/envs/neurodc/bin:/home/fraimondo/miniconda3/condabin:/home/fraimondo/.dotfiles/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/X11R6/bin:/usr/local/games:/usr/games ``` ### Relevant log output ```shell Activating neurodc Sourcing pre_run.sh Script dir from afni.sh: /data/group/appliedml/tools/afni_23.1.10 Path after sourcing afni.sh: /data/group/appliedml/tools/afni_23.1.10/binaries:/home/fraimondo/miniconda3/envs/neurodc/bin:/home/fraimondo/miniconda3/condabin:/home/fraimondo/miniconda3/envs/neurodc/bin:/home/fraimondo/miniconda3/condabin:/home/fraimondo/.dotfiles/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/X11R6/bin:/usr/local/games:/usr/games Script dir from ants.sh: /data/group/appliedml/tools/ants_2.5.0 Path after sourcing ants.sh: /data/group/appliedml/tools/ants_2.5.0/binaries:/data/group/appliedml/tools/afni_23.1.10/binaries:/home/fraimondo/miniconda3/envs/neurodc/bin:/home/fraimondo/miniconda3/condabin:/home/fraimondo/miniconda3/envs/neurodc/bin:/home/fraimondo/miniconda3/condabin:/home/fraimondo/.dotfiles/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/X11R6/bin:/usr/local/games:/usr/games Script dir from fsl.sh: /data/group/appliedml/tools/fsl_6.0.4-patched2 Path after sourcing fsl.sh: /data/group/appliedml/tools/fsl_6.0.4-patched2/binaries:/data/group/appliedml/tools/ants_2.5.0/binaries:/data/group/appliedml/tools/afni_23.1.10/binaries:/home/fraimondo/miniconda3/envs/neurodc/bin:/home/fraimondo/miniconda3/condabin:/home/fraimondo/miniconda3/envs/neurodc/bin:/home/fraimondo/miniconda3/condabin:/home/fraimondo/.dotfiles/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/X11R6/bin:/usr/local/games:/usr/games Running junifer run /home/fraimondo/dev/scratch/neurodc/junifer_jobs/aomic_fc/config.yaml --verbose info --element sub-0014 in conda environment 2024-04-12 22:25:36,051 - JUNIFER - INFO - ===== Lib Versions ===== 2024-04-12 22:25:36,051 - JUNIFER - INFO - numpy: 1.26.4 2024-04-12 22:25:36,051 - JUNIFER - INFO - scipy: 1.11.4 2024-04-12 22:25:36,051 - JUNIFER - INFO - pandas: 2.1.4 2024-04-12 22:25:36,051 - JUNIFER - INFO - nilearn: 0.10.2 2024-04-12 22:25:36,051 - JUNIFER - INFO - nibabel: 5.2.1 2024-04-12 22:25:36,051 - JUNIFER - INFO - junifer: 0.0.5.dev27 2024-04-12 22:25:36,051 - JUNIFER - INFO - ======================== 2024-04-12 22:25:36,051 - JUNIFER - INFO - Parsing yaml file: /home/fraimondo/dev/scratch/neurodc/junifer_jobs/aomic_fc/config.yaml 2024-04-12 22:25:36,058 - JUNIFER - INFO - `datadir` is None, creating a temporary directory 2024-04-12 22:25:36,058 - JUNIFER - INFO - `datadir` set to /tmp/tmpgd5w41ln/datadir 2024-04-12 22:25:36,289 - JUNIFER - INFO - Validating Marker Collection 2024-04-12 22:25:36,289 - JUNIFER - INFO - DataGrabber output type: ['BOLD', 'BOLD_confounds', 'BOLD_mask'] 2024-04-12 22:25:36,289 - JUNIFER - INFO - Validating Data Reader: 2024-04-12 22:25:36,289 - JUNIFER - INFO - Data Reader output type: ['BOLD', 'BOLD_confounds', 'BOLD_mask'] 2024-04-12 22:25:36,289 - JUNIFER - INFO - Validating Preprocessor: fMRIPrepConfoundRemover 2024-04-12 22:25:36,289 - JUNIFER - INFO - Preprocess output type: ['BOLD'] 2024-04-12 22:25:36,289 - JUNIFER - INFO - Validating Marker: parccortical-Schaefer100x17-TianxS2x3T 2024-04-12 22:25:36,386 - JUNIFER - INFO - Marker output type: ['matrix'] 2024-04-12 22:25:36,386 - JUNIFER - INFO - Validating storage for parccortical-Schaefer100x17-TianxS2x3T 2024-04-12 22:27:39,404 - JUNIFER - INFO - Getting element sub-0014 2024-04-12 22:28:45,106 - JUNIFER - INFO - Fitting pipeline 2024-04-12 22:28:45,106 - JUNIFER - INFO - Reading BOLD from /tmp/tmpgd5w41ln/datadir/derivatives/fmriprep/sub-0014/func/sub-0014_task-moviewatching_space-MNI152NLin2009cAsym_desc-preproc_bold.nii.gz 2024-04-12 22:28:45,106 - JUNIFER - INFO - BOLD is type NIFTI 2024-04-12 22:28:45,107 - JUNIFER - INFO - Reading BOLD_confounds from /tmp/tmpgd5w41ln/datadir/derivatives/fmriprep/sub-0014/func/sub-0014_task-moviewatching_desc-confounds_regressors.tsv 2024-04-12 22:28:45,107 - JUNIFER - INFO - BOLD_confounds is type TSV 2024-04-12 22:28:45,123 - JUNIFER - INFO - Reading BOLD_mask from /tmp/tmpgd5w41ln/datadir/derivatives/fmriprep/sub-0014/func/sub-0014_task-moviewatching_space-MNI152NLin2009cAsym_desc-brain_mask.nii.gz 2024-04-12 22:28:45,123 - JUNIFER - INFO - BOLD_mask is type NIFTI 2024-04-12 22:28:45,124 - JUNIFER - INFO - Preprocessing data with fMRIPrepConfoundRemover 2024-04-12 22:28:45,124 - JUNIFER - INFO - Preprocessing BOLD 2024-04-12 22:28:47,902 - JUNIFER - INFO - No `t_r` specified, using t_r from NIfTI header 2024-04-12 22:28:47,903 - JUNIFER - INFO - Read t_r from NIfTI header: 2.200000047683716 2024-04-12 22:28:48,266 - JUNIFER - INFO - Cleaning image using nilearn 2024-04-12 22:28:54,670 - JUNIFER - INFO - Fitting marker parccortical-Schaefer100x17-TianxS2x3T 2024-04-12 22:28:54,778 - JUNIFER - INFO - Computing BOLD 2024-04-12 22:28:54,779 - JUNIFER - INFO - Fetching one of Schaefer parcellations. 2024-04-12 22:28:54,779 - JUNIFER - INFO - Parcellation parameters: 2024-04-12 22:28:54,779 - JUNIFER - INFO - resolution: 3.0 2024-04-12 22:28:54,779 - JUNIFER - INFO - n_rois: 100 2024-04-12 22:28:54,779 - JUNIFER - INFO - yeo_networks: 17 2024-04-12 22:28:54,781 - JUNIFER - INFO - Loading parcellation /home/fraimondo/junifer/data/parcellations/schaefer_2018/Schaefer2018_100Parcels_17Networks_order_FSLMNI152_2mm.nii.gz 2024-04-12 22:28:54,816 - JUNIFER - INFO - Found existing xfm file for MNI152NLin6Asym to MNI152NLin2009cAsym at /home/fraimondo/junifer/data/xfms/MNI152NLin6Asym_to_MNI152NLin2009cAsym/MNI152NLin6Asym_to_MNI152NLin2009cAsym_Composite.h5 2024-04-12 22:28:55,773 - JUNIFER - INFO - Downloading template MNI152NLin2009cAsym in resolution 2 2024-04-12 22:28:55,892 - JUNIFER - INFO - antsApplyTransforms command to be executed: antsApplyTransforms -d 3 -e 3 -n 'GenericLabel[NearestNeighbor]' -i /tmp/tmpol3919xi/parcellationsupbm3qm_/Schaefer100x17.nii.gz -r /tmp/tmpol3919xi/parcellationsupbm3qm_/MNI152NLin2009cAsym_T1w_3.0.nii.gz -t /home/fraimondo/junifer/data/xfms/MNI152NLin6Asym_to_MNI152NLin2009cAsym/MNI152NLin6Asym_to_MNI152NLin2009cAsym_Composite.h5 -o /tmp/tmpyk7pw003/parcellationskeha8hp0/Schaefer100x17_warped_from_MNI152NLin6Asym_to_MNI152NLin2009cAsym.nii.gz 2024-04-12 22:29:00,187 - JUNIFER - INFO - antsApplyTransforms command succeeded with the following output: b'/tmp/tmpol3919xi/parcellationsupbm3qm_/Schaefer100x17.nii.gz is a file\n/tmp/tmpol3919xi/parcellationsupbm3qm_/MNI152NLin2009cAsym_T1w_3.0.nii.gz is a file\n/home/fraimondo/junifer/data/xfms/MNI152NLin6Asym_to_MNI152NLin2009cAsym/MNI152NLin6Asym_to_MNI152NLin2009cAsym_Composite.h5 is a file\n/tmp/tmpyk7pw003/parcellationskeha8hp0/Schaefer100x17_warped_from_MNI152NLin6Asym_to_MNI152NLin2009cAsym.nii.gz is a prefix\nSingularity args: --bind /tmp/tmpol3919xi/parcellationsupbm3qm_:/data/mount_1 --bind /tmp/tmpol3919xi/parcellationsupbm3qm_:/data/mount_2 --bind /home/fraimondo/junifer/data/xfms/MNI152NLin6Asym_to_MNI152NLin2009cAsym:/data/mount_3 --bind /tmp/tmpyk7pw003/parcellationskeha8hp0:/data/mount_4\nCorrected args for ANTS: antsApplyTransforms -d 3 -e 3 -n GenericLabel[NearestNeighbor] -i /data/mount_1/Schaefer100x17.nii.gz -r /data/mount_2/MNI152NLin2009cAsym_T1w_3.0.nii.gz -t /data/mount_3/MNI152NLin6Asym_to_MNI152NLin2009cAsym_Composite.h5 -o /data/mount_4/Schaefer100x17_warped_from_MNI152NLin6Asym_to_MNI152NLin2009cAsym.nii.gz\nRunning command: /data/group/appliedml/tools/ants_2.5.0/singularity_cmd exec --cleanenv --bind /tmp/tmpol3919xi/parcellationsupbm3qm_:/data/mount_1 --bind /tmp/tmpol3919xi/parcellationsupbm3qm_:/data/mount_2 --bind /home/fraimondo/junifer/data/xfms/MNI152NLin6Asym_to_MNI152NLin2009cAsym:/data/mount_3 --bind /tmp/tmpyk7pw003/parcellationskeha8hp0:/data/mount_4 /data/group/appliedml/tools/ants_2.5.0/ants_v2.5.0.sif antsApplyTransforms -d 3 -e 3 -n GenericLabel[NearestNeighbor] -i /data/mount_1/Schaefer100x17.nii.gz -r /data/mount_2/MNI152NLin2009cAsym_T1w_3.0.nii.gz -t /data/mount_3/MNI152NLin6Asym_to_MNI152NLin2009cAsym_Composite.h5 -o /data/mount_4/Schaefer100x17_warped_from_MNI152NLin6Asym_to_MNI152NLin2009cAsym.nii.gz\n' 2024-04-12 22:29:00,249 - JUNIFER - INFO - Fetching one of Tian parcellations. 2024-04-12 22:29:00,249 - JUNIFER - INFO - Parcellation parameters: 2024-04-12 22:29:00,249 - JUNIFER - INFO - resolution: 3.0 2024-04-12 22:29:00,249 - JUNIFER - INFO - scale: 2 2024-04-12 22:29:00,249 - JUNIFER - INFO - space: MNI152NLin2009cAsym 2024-04-12 22:29:00,249 - JUNIFER - INFO - magneticfield: 3T 2024-04-12 22:29:00,251 - JUNIFER - INFO - Loading parcellation /home/fraimondo/junifer/data/parcellations/Tian2020MSA_v1.1/3T/Subcortex-Only/Tian_Subcortex_S2_3T_2009cAsym.nii.gz 2024-04-12 22:29:00,334 - JUNIFER - WARNING - The parcellations have different resolutions!Resampling all parcellations to the first one in the list. ``` ### Anything else? _No response_
synchon commented 2024-04-15 10:50:01 +00:00 (Migrated from github.com)
Copy link

@fraimondo Can you check this branch out and see if the issue is solved?

@fraimondo Can you check this branch out and see if the issue is solved?
codecov[bot] commented 2024-04-15 10:52:18 +00:00 (Migrated from github.com)
Copy link

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 100.00%. Comparing base (a13e587) to head (54e6fb8).

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff            @@
##              main      #331   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files            1         1           
  Lines            1         1           
=========================================
  Hits             1         1
Flag Coverage Δ
docs 100.00% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

## [Codecov](https://app.codecov.io/gh/juaml/junifer/pull/331?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml) Report All modified and coverable lines are covered by tests :white_check_mark: > Project coverage is 100.00%. Comparing base [(`a13e587`)](https://app.codecov.io/gh/juaml/junifer/commit/a13e587b027739c77ebf41046b86328f9af4be94?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml) to head [(`54e6fb8`)](https://app.codecov.io/gh/juaml/junifer/pull/331?dropdown=coverage&src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml). <details><summary>Additional details and impacted files</summary> [![Impacted file tree graph](https://app.codecov.io/gh/juaml/junifer/pull/331/graphs/tree.svg?width=650&height=150&src=pr&token=5H21JuZXMw&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml)](https://app.codecov.io/gh/juaml/junifer/pull/331?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml) ```diff @@ Coverage Diff @@ ## main #331 +/- ## ========================================= Coverage 100.00% 100.00% ========================================= Files 1 1 Lines 1 1 ========================================= Hits 1 1 ``` | [Flag](https://app.codecov.io/gh/juaml/junifer/pull/331/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml) | Coverage Δ | | |---|---|---| | [docs](https://app.codecov.io/gh/juaml/junifer/pull/331/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml) | `100.00% <ø> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml#carryforward-flags-in-the-pull-request-comment) to find out more. </details>
github-actions[bot] commented 2024-04-15 11:10:25 +00:00 (Migrated from github.com)
Copy link
PR Preview Action v1.4.7
Preview removed because the pull request was closed.
2024-05-06 10:44 UTC
[PR Preview Action](https://github.com/rossjrw/pr-preview-action) v1.4.7 :---: Preview removed because the pull request was closed. 2024-05-06 10:44 UTC <!-- Sticky Pull Request Commentpr-preview -->
fraimondo commented 2024-05-06 09:11:23 +00:00 (Migrated from github.com)
Copy link
workdir: /tmp

datagrabber:
  kind: DataladAOMICID1000
  types: 
    - BOLD
    - BOLD_confounds
    - BOLD_mask
preprocess:
    kind: fMRIPrepConfoundRemover
    detrend: true
    standardize: true
    strategy:
        motion: full
        wm_csf: full
        global_signal: full
    low_pass: 0.08
    high_pass: 0.01
    masks:
      - inherit
      - compute_epi_mask
      - threshold: 0
markers:
  - name: parccortical-Schaefer100x17-TianxS2x3T
    kind: FunctionalConnectivityParcels
    parcellation: 
        - Schaefer100x17
        - TianxS2x3TxMNInonlinear2009cAsym
    cor_method: correlation
    cor_method_params:
      empirical: true
    masks:
      - inherit

storage:
  kind: HDF5FeatureStorage
  uri: "/data/group/riseml/fraimondo/ds003097_FC/ds003097_FC.hdf5"
queue:
  jobname: aomic_fc
  kind: HTCondor
  env:
    kind: conda
    name: neurodc
  mem: 8G
  disk: 5G
  verbose: info
  pre_run: |
    source /data/group/appliedml/tools/afni_23.1.10/afni.sh
    source /data/group/appliedml/tools/ants_2.5.0/ants.sh
    source /data/group/appliedml/tools/fsl_6.0.4-patched2/fsl.sh

Yes, it works now.

> ```yaml > workdir: /tmp > > datagrabber: > kind: DataladAOMICID1000 > types: > - BOLD > - BOLD_confounds > - BOLD_mask > preprocess: > kind: fMRIPrepConfoundRemover > detrend: true > standardize: true > strategy: > motion: full > wm_csf: full > global_signal: full > low_pass: 0.08 > high_pass: 0.01 > masks: > - inherit > - compute_epi_mask > - threshold: 0 > markers: > - name: parccortical-Schaefer100x17-TianxS2x3T > kind: FunctionalConnectivityParcels > parcellation: > - Schaefer100x17 > - TianxS2x3TxMNInonlinear2009cAsym > cor_method: correlation > cor_method_params: > empirical: true > masks: > - inherit > > storage: > kind: HDF5FeatureStorage > uri: "/data/group/riseml/fraimondo/ds003097_FC/ds003097_FC.hdf5" > queue: > jobname: aomic_fc > kind: HTCondor > env: > kind: conda > name: neurodc > mem: 8G > disk: 5G > verbose: info > pre_run: | > source /data/group/appliedml/tools/afni_23.1.10/afni.sh > source /data/group/appliedml/tools/ants_2.5.0/ants.sh > source /data/group/appliedml/tools/fsl_6.0.4-patched2/fsl.sh > ``` Yes, it works now.
👍 1
fraimondo (Migrated from github.com) approved these changes 2024-05-06 09:11:38 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
fraimondo
Labels
Clear labels   
CRITICAL
High Priority Issue

  
Stale

   
WIP
Work in progress

  
bug
Something isn't working

  
concept
Design/Implementation concepts

  
coordinate
Issues or pull requests related to coordinates (ROIs)

  
dataset
Issues or pull requests related to datagrabbers or datasets

  
dependencies
Pull requests that update a dependency file

  
documentation
Improvements or additions to documentation

  
duplicate
This issue or pull request already exists

  
enhancement
New feature or request

  
github_actions
Pull requests that update GitHub Actions code

  
good first issue
Good for newcomers

  
help wanted
Extra attention is needed

  
invalid
This doesn't seem right

  
maintenance
Issues or pull requests related to maintenance

  
maps
Issues or pull requests related to maps (probabilistic atlases)

  
marker
Issues or pull requests related to markers

  
mask
Issues or pull requests related to masks

  
on hold
Issue waiting on other issue(s)

  
parcellation
Issues or pull requests related to parcellations (deterministic atlases)

  
preprocess
Issues or pull requests related to preprocessors

  
question
Further information is requested

  
ready
Pull request is ready for review and merge

  
storage
Issues or pull requests related to marker storage or retrieval

  
template-space
Issues or pull requests related to template spaces

  
triage
New issues waiting to be reviewed

  
wontfix
This will not be worked on

No labels
CRITICAL
Stale
WIP
bug
concept
coordinate
dataset
dependencies
documentation
duplicate
enhancement
github_actions
good first issue
help wanted
invalid
maintenance
maps
marker
mask
on hold
parcellation
preprocess
question
ready
storage
template-space
triage
wontfix
Milestone
Clear milestone
No items
No milestone
0.0.5 (alpha 4)
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
juaml/junifer!331
No description provided.