juaml/junifer
Watch
3
0
Fork
You've already forked junifer
 0
Code Issues 14 Pull requests 2 Packages Activity Actions

[BUG]: Extra dimension added to parcellation after warping to another template space via ANTs #324

Merged
synchon merged 2 commits from fix/parcellation-warp-incorrect-ndim into main 2024-04-10 16:33:31 +00:00
Conversation 8 Commits 2 Files changed 2 +4 -1
synchon commented 2024-04-10 12:42:10 +00:00 (Migrated from github.com)
Copy link

Is there an existing issue for this?

  • I have searched the existing issues

Current Behavior

When I apply a mask in FC extraction using a YAML pipeline it fails.

Expected Behavior

I would like it to apply the mask correctly and not fail.

Steps To Reproduce

  1. See environment below.
  2. See the Yaml here:
workdir: /tmp

datagrabber:
    kind: DataladAOMICID1000


preprocess:
    kind: Smoothing
    using: "nilearn"
    "on": "BOLD"
    smoothing_params:
      fwhm: 10

storage:
  kind: HDF5FeatureStorage
  uri: data/aomic_id1000.hdf5


markers:

  - name: parccortical-Schaefer100x17FSLMNI_marker-empiricalFC
    kind: FunctionalConnectivityParcels
    parcellation: Schaefer100x17
    cor_method: correlation
    cor_method_params:
      empirical: true
    masks:
      - inherit

        
queue:
  jobname: smoothing_test
  kind: HTCondor
  collect: on_success_only
  env:
    kind: venv
    name: env
  mem: 20G
  verbose: 8
  pre_run: |
    source /data/group/appliedml/tools/ants_2.5.0/ants.sh

Environment

❱ junifer wtf
junifer:
  version: 0.0.4.dev822
python:
  version: 3.10.10
  implementation: CPython
dependencies:
  click: 8.1.7
  numpy: 1.26.4
  scipy: 1.11.4
  datalad: 0.19.6
  pandas: 2.1.4
  nibabel: 5.2.1
  nilearn: 0.10.2
  sqlalchemy: 2.0.28
  ruamel.yaml: 0.17.21
  httpx: 0.26.0
  tqdm: 4.66.1
  templateflow: 24.2.0
  looseversion: None
system:
  platform: Linux-6.6.13+bpo-amd64-x86_64-with-glibc2.36
environment:
  LC_CTYPE: en_US.UTF-8
  PATH: /home/lsasse/miniconda3/bin:/home/lsasse/miniconda3/condabin:/home/lsasse/bin:/home/lsasse/.dotfiles/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/X11R6/bin:/usr/local/games:/usr/games

Relevant log output

It results in the following error:

[DEBUG] Done dropping for Dataset(/tmp/tmplwwl0ock/datadir) 
[DEBUG] Finished dropping datasets at Dataset(/tmp/tmplwwl0ock/datadir) 
Traceback (most recent call last):
  File "/data/project/identification_prediction/projects/test_junifer_smoothing/env/bin/junifer", line 8, in <module>
    sys.exit(cli())
  File "/data/project/identification_prediction/projects/test_junifer_smoothing/env/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/data/project/identification_prediction/projects/test_junifer_smoothing/env/lib/python3.10/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/data/project/identification_prediction/projects/test_junifer_smoothing/env/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/data/project/identification_prediction/projects/test_junifer_smoothing/env/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/data/project/identification_prediction/projects/test_junifer_smoothing/env/lib/python3.10/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/data/project/identification_prediction/projects/test_junifer_smoothing/env/lib/python3.10/site-packages/junifer/api/cli.py", line 224, in run
    api_run(
  File "/data/project/identification_prediction/projects/test_junifer_smoothing/env/lib/python3.10/site-packages/junifer/api/functions.py", line 174, in run
    mc.fit(datagrabber_object[t_element])
  File "/data/project/identification_prediction/projects/test_junifer_smoothing/env/lib/python3.10/site-packages/junifer/markers/collection.py", line 101, in fit
    m_value = marker.fit_transform(data, storage=self._storage)
  File "/data/project/identification_prediction/projects/test_junifer_smoothing/env/lib/python3.10/site-packages/junifer/pipeline/pipeline_step_mixin.py", line 229, in fit_transform
    return self._fit_transform(input=input, **kwargs)
  File "/data/project/identification_prediction/projects/test_junifer_smoothing/env/lib/python3.10/site-packages/junifer/markers/base.py", line 201, in _fit_transform
    t_out = self.compute(input=t_input, extra_input=extra_input)
  File "/data/project/identification_prediction/projects/test_junifer_smoothing/env/lib/python3.10/site-packages/junifer/markers/functional_connectivity/functional_connectivity_base.py", line 138, in compute
    aggregation = self.aggregate(input, extra_input=extra_input)
  File "/data/project/identification_prediction/projects/test_junifer_smoothing/env/lib/python3.10/site-packages/junifer/markers/functional_connectivity/functional_connectivity_parcels.py", line 102, in aggregate
    return parcel_aggregation.compute(input, extra_input=extra_input)
  File "/data/project/identification_prediction/projects/test_junifer_smoothing/env/lib/python3.10/site-packages/junifer/markers/parcel_aggregation.py", line 191, in compute
    parcellation_bin = math_img(
  File "/data/project/identification_prediction/projects/test_junifer_smoothing/env/lib/python3.10/site-packages/nilearn/image/image.py", line 1059, in math_img
    result = eval(formula, data_dict)
  File "<string>", line 1, in <module>
ValueError: ("Input formula couldn't be processed, you provided 'np.logical_and(img, mask)',", 'operands could not be broadcast together with shapes (60,77,65,1) (60,77,65) ')





### Anything else?

I guess it is missing np.squeeze() somewhere.
### Is there an existing issue for this? - [X] I have searched the existing issues ### Current Behavior When I apply a mask in FC extraction using a YAML pipeline it fails. ### Expected Behavior I would like it to apply the mask correctly and not fail. ### Steps To Reproduce 1. See environment below. 2. See the Yaml here: ``` workdir: /tmp datagrabber: kind: DataladAOMICID1000 preprocess: kind: Smoothing using: "nilearn" "on": "BOLD" smoothing_params: fwhm: 10 storage: kind: HDF5FeatureStorage uri: data/aomic_id1000.hdf5 markers: - name: parccortical-Schaefer100x17FSLMNI_marker-empiricalFC kind: FunctionalConnectivityParcels parcellation: Schaefer100x17 cor_method: correlation cor_method_params: empirical: true masks: - inherit queue: jobname: smoothing_test kind: HTCondor collect: on_success_only env: kind: venv name: env mem: 20G verbose: 8 pre_run: | source /data/group/appliedml/tools/ants_2.5.0/ants.sh ``` ### Environment ```markdown ❱ junifer wtf junifer: version: 0.0.4.dev822 python: version: 3.10.10 implementation: CPython dependencies: click: 8.1.7 numpy: 1.26.4 scipy: 1.11.4 datalad: 0.19.6 pandas: 2.1.4 nibabel: 5.2.1 nilearn: 0.10.2 sqlalchemy: 2.0.28 ruamel.yaml: 0.17.21 httpx: 0.26.0 tqdm: 4.66.1 templateflow: 24.2.0 looseversion: None system: platform: Linux-6.6.13+bpo-amd64-x86_64-with-glibc2.36 environment: LC_CTYPE: en_US.UTF-8 PATH: /home/lsasse/miniconda3/bin:/home/lsasse/miniconda3/condabin:/home/lsasse/bin:/home/lsasse/.dotfiles/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/X11R6/bin:/usr/local/games:/usr/games ``` ### Relevant log output ```shell It results in the following error: [DEBUG] Done dropping for Dataset(/tmp/tmplwwl0ock/datadir) [DEBUG] Finished dropping datasets at Dataset(/tmp/tmplwwl0ock/datadir) Traceback (most recent call last): File "/data/project/identification_prediction/projects/test_junifer_smoothing/env/bin/junifer", line 8, in <module> sys.exit(cli()) File "/data/project/identification_prediction/projects/test_junifer_smoothing/env/lib/python3.10/site-packages/click/core.py", line 1157, in __call__ return self.main(*args, **kwargs) File "/data/project/identification_prediction/projects/test_junifer_smoothing/env/lib/python3.10/site-packages/click/core.py", line 1078, in main rv = self.invoke(ctx) File "/data/project/identification_prediction/projects/test_junifer_smoothing/env/lib/python3.10/site-packages/click/core.py", line 1688, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/data/project/identification_prediction/projects/test_junifer_smoothing/env/lib/python3.10/site-packages/click/core.py", line 1434, in invoke return ctx.invoke(self.callback, **ctx.params) File "/data/project/identification_prediction/projects/test_junifer_smoothing/env/lib/python3.10/site-packages/click/core.py", line 783, in invoke return __callback(*args, **kwargs) File "/data/project/identification_prediction/projects/test_junifer_smoothing/env/lib/python3.10/site-packages/junifer/api/cli.py", line 224, in run api_run( File "/data/project/identification_prediction/projects/test_junifer_smoothing/env/lib/python3.10/site-packages/junifer/api/functions.py", line 174, in run mc.fit(datagrabber_object[t_element]) File "/data/project/identification_prediction/projects/test_junifer_smoothing/env/lib/python3.10/site-packages/junifer/markers/collection.py", line 101, in fit m_value = marker.fit_transform(data, storage=self._storage) File "/data/project/identification_prediction/projects/test_junifer_smoothing/env/lib/python3.10/site-packages/junifer/pipeline/pipeline_step_mixin.py", line 229, in fit_transform return self._fit_transform(input=input, **kwargs) File "/data/project/identification_prediction/projects/test_junifer_smoothing/env/lib/python3.10/site-packages/junifer/markers/base.py", line 201, in _fit_transform t_out = self.compute(input=t_input, extra_input=extra_input) File "/data/project/identification_prediction/projects/test_junifer_smoothing/env/lib/python3.10/site-packages/junifer/markers/functional_connectivity/functional_connectivity_base.py", line 138, in compute aggregation = self.aggregate(input, extra_input=extra_input) File "/data/project/identification_prediction/projects/test_junifer_smoothing/env/lib/python3.10/site-packages/junifer/markers/functional_connectivity/functional_connectivity_parcels.py", line 102, in aggregate return parcel_aggregation.compute(input, extra_input=extra_input) File "/data/project/identification_prediction/projects/test_junifer_smoothing/env/lib/python3.10/site-packages/junifer/markers/parcel_aggregation.py", line 191, in compute parcellation_bin = math_img( File "/data/project/identification_prediction/projects/test_junifer_smoothing/env/lib/python3.10/site-packages/nilearn/image/image.py", line 1059, in math_img result = eval(formula, data_dict) File "<string>", line 1, in <module> ValueError: ("Input formula couldn't be processed, you provided 'np.logical_and(img, mask)',", 'operands could not be broadcast together with shapes (60,77,65,1) (60,77,65) ') ``` ``` ### Anything else? I guess it is missing np.squeeze() somewhere.
LeSasse commented 2024-04-05 12:00:34 +00:00 (Migrated from github.com)
Copy link

When I remove

    masks:
      - inherit

from the marker it works.

When I remove ``` masks: - inherit ``` from the marker it works.
LeSasse commented 2024-04-05 12:03:01 +00:00 (Migrated from github.com)
Copy link

I did not actually define a mask in the preprocessor.

I did not actually define a mask in the preprocessor.
LeSasse commented 2024-04-05 12:14:07 +00:00 (Migrated from github.com)
Copy link

I have re-opened it, because the AOMIC ID1000 datagrabber does have BOLD_masks that I should be able to inherit in the marker extraction. But I think the preprocessor for Smoothing from #161 is messing with it.

I have re-opened it, because the AOMIC ID1000 datagrabber does have BOLD_masks that I should be able to inherit in the marker extraction. But I think the preprocessor for Smoothing from #161 is messing with it.
synchon commented 2024-04-10 12:45:23 +00:00 (Migrated from github.com)
Copy link

@LeSasse I believe I've found the bug and it's that when we warp parcellation via ANTs, it somehow adds an extra axis when loading via nibabel. If you try with a parcellation without the need to warp, it'll be ok. So, I've added an explicit np.squeeze in nilearn's math_img. Check this branch out for your run and let me know if it solves it and you get correct results.

@LeSasse I believe I've found the bug and it's that when we warp parcellation via ANTs, it somehow adds an extra axis when loading via nibabel. If you try with a parcellation without the need to warp, it'll be ok. So, I've added an explicit `np.squeeze` in nilearn's `math_img`. Check this branch out for your run and let me know if it solves it and you get correct results.
github-actions[bot] commented 2024-04-10 13:07:27 +00:00 (Migrated from github.com)
Copy link
PR Preview Action v1.4.7
Preview removed because the pull request was closed.
2024-04-10 17:09 UTC
[PR Preview Action](https://github.com/rossjrw/pr-preview-action) v1.4.7 :---: Preview removed because the pull request was closed. 2024-04-10 17:09 UTC <!-- Sticky Pull Request Commentpr-preview -->
LeSasse commented 2024-04-10 13:12:34 +00:00 (Migrated from github.com)
Copy link

Works like a charm!

Works like a charm!
LeSasse (Migrated from github.com) approved these changes 2024-04-10 13:20:25 +00:00
LeSasse (Migrated from github.com) left a comment
Copy link

This now works, the np.squeeze makes sense to me

This now works, the np.squeeze makes sense to me
codecov[bot] commented 2024-04-10 14:25:50 +00:00 (Migrated from github.com)
Copy link

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 100.00%. Comparing base (0fde637) to head (c0c893b).

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff            @@
##              main      #324   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files            1         1           
  Lines            1         1           
=========================================
  Hits             1         1
Flag Coverage Δ
docs 100.00% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

## [Codecov](https://app.codecov.io/gh/juaml/junifer/pull/324?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml) Report All modified and coverable lines are covered by tests :white_check_mark: > Project coverage is 100.00%. Comparing base [(`0fde637`)](https://app.codecov.io/gh/juaml/junifer/commit/0fde6372bf62144eb989fd31da09f68354fc71d1?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml) to head [(`c0c893b`)](https://app.codecov.io/gh/juaml/junifer/pull/324?dropdown=coverage&src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml). <details><summary>Additional details and impacted files</summary> [![Impacted file tree graph](https://app.codecov.io/gh/juaml/junifer/pull/324/graphs/tree.svg?width=650&height=150&src=pr&token=5H21JuZXMw&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml)](https://app.codecov.io/gh/juaml/junifer/pull/324?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml) ```diff @@ Coverage Diff @@ ## main #324 +/- ## ========================================= Coverage 100.00% 100.00% ========================================= Files 1 1 Lines 1 1 ========================================= Hits 1 1 ``` | [Flag](https://app.codecov.io/gh/juaml/junifer/pull/324/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml) | Coverage Δ | | |---|---|---| | [docs](https://app.codecov.io/gh/juaml/junifer/pull/324/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml) | `100.00% <ø> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml#carryforward-flags-in-the-pull-request-comment) to find out more. </details>
Sign in to join this conversation.
Reviewers
No reviewers
LeSasse
Labels
Clear labels   
CRITICAL
High Priority Issue

  
Stale

   
WIP
Work in progress

  
bug
Something isn't working

  
concept
Design/Implementation concepts

  
coordinate
Issues or pull requests related to coordinates (ROIs)

  
dataset
Issues or pull requests related to datagrabbers or datasets

  
dependencies
Pull requests that update a dependency file

  
documentation
Improvements or additions to documentation

  
duplicate
This issue or pull request already exists

  
enhancement
New feature or request

  
github_actions
Pull requests that update GitHub Actions code

  
good first issue
Good for newcomers

  
help wanted
Extra attention is needed

  
invalid
This doesn't seem right

  
maintenance
Issues or pull requests related to maintenance

  
maps
Issues or pull requests related to maps (probabilistic atlases)

  
marker
Issues or pull requests related to markers

  
mask
Issues or pull requests related to masks

  
on hold
Issue waiting on other issue(s)

  
parcellation
Issues or pull requests related to parcellations (deterministic atlases)

  
preprocess
Issues or pull requests related to preprocessors

  
question
Further information is requested

  
ready
Pull request is ready for review and merge

  
storage
Issues or pull requests related to marker storage or retrieval

  
template-space
Issues or pull requests related to template spaces

  
triage
New issues waiting to be reviewed

  
wontfix
This will not be worked on

No labels
CRITICAL
Stale
WIP
bug
concept
coordinate
dataset
dependencies
documentation
duplicate
enhancement
github_actions
good first issue
help wanted
invalid
maintenance
maps
marker
mask
on hold
parcellation
preprocess
question
ready
storage
template-space
triage
wontfix
Milestone
Clear milestone
No items
No milestone
0.0.5 (alpha 4)
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
juaml/junifer!324
No description provided.