[ENH]: Update Warp datatype schema and logic #390

Merged
synchon merged 20 commits from refactor/warp-dtype into main 2024-11-13 11:55:48 +00:00
synchon commented 2024-11-08 11:00:57 +00:00 (Migrated from github.com)
  1. Modify the Warp datatype to be a "list" of mappings from / to spaces.
  2. Add a "warper" key to the Warp datatype indicating which warper was used: fsl or ants for the moment.
  3. Modify the SpaceWarper to allow to "auto" select the warper based on the dataset. This should require both FSL and ANTs as external dependencies. This only works if reference is "T1w": "auto" is not defined for junifer internal transforms (i.e. between template spaces)
  4. Modify coordinates/masks/parcellations so they check for the right transform and use the "warper" parameter accordingly.
1) Modify the `Warp` datatype to be a "list" of mappings from / to spaces. 2) Add a `"warper"` key to the `Warp` datatype indicating which warper was used: `fsl` or `ants` for the moment. 3) Modify the `SpaceWarper` to allow to `"auto"` select the warper based on the dataset. This should require both FSL and ANTs as external dependencies. This only works if reference is `"T1w"`: `"auto"` is not defined for junifer internal transforms (i.e. between template spaces) 4) Modify coordinates/masks/parcellations so they check for the right transform and use the "warper" parameter accordingly.
codecov[bot] commented 2024-11-08 11:01:38 +00:00 (Migrated from github.com)

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 87.48%. Comparing base (96f693e) to head (f5e5792).
Report is 21 commits behind head on main.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main     #390      +/-   ##
==========================================
+ Coverage   87.31%   87.48%   +0.16%     
==========================================
  Files         129      129              
  Lines        5235     5154      -81     
  Branches      857      828      -29     
==========================================
- Hits         4571     4509      -62     
+ Misses        491      475      -16     
+ Partials      173      170       -3     
Flag Coverage Δ
docs 100.00% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
...nifer/data/coordinates/_ants_coordinates_warper.py 50.00% <ø> (ø)
junifer/data/coordinates/_coordinates.py 86.20% <ø> (+5.56%) ⬆️
...unifer/data/coordinates/_fsl_coordinates_warper.py 50.00% <ø> (ø)
junifer/data/masks/_ants_mask_warper.py 29.62% <ø> (-1.41%) ⬇️
junifer/data/masks/_fsl_mask_warper.py 50.00% <ø> (ø)
junifer/data/masks/_masks.py 83.33% <ø> (+2.95%) ⬆️
...er/data/parcellations/_ants_parcellation_warper.py 77.77% <ø> (+1.91%) ⬆️
...fer/data/parcellations/_fsl_parcellation_warper.py 50.00% <ø> (ø)
junifer/data/parcellations/_parcellations.py 89.53% <ø> (+0.87%) ⬆️
junifer/data/utils.py 100.00% <ø> (ø)
... and 14 more
## [Codecov](https://app.codecov.io/gh/juaml/junifer/pull/390?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml) Report All modified and coverable lines are covered by tests :white_check_mark: > Project coverage is 87.48%. Comparing base [(`96f693e`)](https://app.codecov.io/gh/juaml/junifer/commit/96f693ea1821e98c60408eb2d876bdac755ddddb?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml) to head [(`f5e5792`)](https://app.codecov.io/gh/juaml/junifer/commit/f5e5792a24302f7055823ea82c0bbf5540e2fe43?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml). > Report is 21 commits behind head on main. <details><summary>Additional details and impacted files</summary> [![Impacted file tree graph](https://app.codecov.io/gh/juaml/junifer/pull/390/graphs/tree.svg?width=650&height=150&src=pr&token=5H21JuZXMw&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml)](https://app.codecov.io/gh/juaml/junifer/pull/390?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml) ```diff @@ Coverage Diff @@ ## main #390 +/- ## ========================================== + Coverage 87.31% 87.48% +0.16% ========================================== Files 129 129 Lines 5235 5154 -81 Branches 857 828 -29 ========================================== - Hits 4571 4509 -62 + Misses 491 475 -16 + Partials 173 170 -3 ``` | [Flag](https://app.codecov.io/gh/juaml/junifer/pull/390/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml) | Coverage Δ | | |---|---|---| | [docs](https://app.codecov.io/gh/juaml/junifer/pull/390/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml) | `100.00% <ø> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml#carryforward-flags-in-the-pull-request-comment) to find out more. | [Files with missing lines](https://app.codecov.io/gh/juaml/junifer/pull/390?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml) | Coverage Δ | | |---|---|---| | [...nifer/data/coordinates/\_ants\_coordinates\_warper.py](https://app.codecov.io/gh/juaml/junifer/pull/390?src=pr&el=tree&filepath=junifer%2Fdata%2Fcoordinates%2F_ants_coordinates_warper.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml#diff-anVuaWZlci9kYXRhL2Nvb3JkaW5hdGVzL19hbnRzX2Nvb3JkaW5hdGVzX3dhcnBlci5weQ==) | `50.00% <ø> (ø)` | | | [junifer/data/coordinates/\_coordinates.py](https://app.codecov.io/gh/juaml/junifer/pull/390?src=pr&el=tree&filepath=junifer%2Fdata%2Fcoordinates%2F_coordinates.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml#diff-anVuaWZlci9kYXRhL2Nvb3JkaW5hdGVzL19jb29yZGluYXRlcy5weQ==) | `86.20% <ø> (+5.56%)` | :arrow_up: | | [...unifer/data/coordinates/\_fsl\_coordinates\_warper.py](https://app.codecov.io/gh/juaml/junifer/pull/390?src=pr&el=tree&filepath=junifer%2Fdata%2Fcoordinates%2F_fsl_coordinates_warper.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml#diff-anVuaWZlci9kYXRhL2Nvb3JkaW5hdGVzL19mc2xfY29vcmRpbmF0ZXNfd2FycGVyLnB5) | `50.00% <ø> (ø)` | | | [junifer/data/masks/\_ants\_mask\_warper.py](https://app.codecov.io/gh/juaml/junifer/pull/390?src=pr&el=tree&filepath=junifer%2Fdata%2Fmasks%2F_ants_mask_warper.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml#diff-anVuaWZlci9kYXRhL21hc2tzL19hbnRzX21hc2tfd2FycGVyLnB5) | `29.62% <ø> (-1.41%)` | :arrow_down: | | [junifer/data/masks/\_fsl\_mask\_warper.py](https://app.codecov.io/gh/juaml/junifer/pull/390?src=pr&el=tree&filepath=junifer%2Fdata%2Fmasks%2F_fsl_mask_warper.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml#diff-anVuaWZlci9kYXRhL21hc2tzL19mc2xfbWFza193YXJwZXIucHk=) | `50.00% <ø> (ø)` | | | [junifer/data/masks/\_masks.py](https://app.codecov.io/gh/juaml/junifer/pull/390?src=pr&el=tree&filepath=junifer%2Fdata%2Fmasks%2F_masks.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml#diff-anVuaWZlci9kYXRhL21hc2tzL19tYXNrcy5weQ==) | `83.33% <ø> (+2.95%)` | :arrow_up: | | [...er/data/parcellations/\_ants\_parcellation\_warper.py](https://app.codecov.io/gh/juaml/junifer/pull/390?src=pr&el=tree&filepath=junifer%2Fdata%2Fparcellations%2F_ants_parcellation_warper.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml#diff-anVuaWZlci9kYXRhL3BhcmNlbGxhdGlvbnMvX2FudHNfcGFyY2VsbGF0aW9uX3dhcnBlci5weQ==) | `77.77% <ø> (+1.91%)` | :arrow_up: | | [...fer/data/parcellations/\_fsl\_parcellation\_warper.py](https://app.codecov.io/gh/juaml/junifer/pull/390?src=pr&el=tree&filepath=junifer%2Fdata%2Fparcellations%2F_fsl_parcellation_warper.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml#diff-anVuaWZlci9kYXRhL3BhcmNlbGxhdGlvbnMvX2ZzbF9wYXJjZWxsYXRpb25fd2FycGVyLnB5) | `50.00% <ø> (ø)` | | | [junifer/data/parcellations/\_parcellations.py](https://app.codecov.io/gh/juaml/junifer/pull/390?src=pr&el=tree&filepath=junifer%2Fdata%2Fparcellations%2F_parcellations.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml#diff-anVuaWZlci9kYXRhL3BhcmNlbGxhdGlvbnMvX3BhcmNlbGxhdGlvbnMucHk=) | `89.53% <ø> (+0.87%)` | :arrow_up: | | [junifer/data/utils.py](https://app.codecov.io/gh/juaml/junifer/pull/390?src=pr&el=tree&filepath=junifer%2Fdata%2Futils.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml#diff-anVuaWZlci9kYXRhL3V0aWxzLnB5) | `100.00% <ø> (ø)` | | | ... and [14 more](https://app.codecov.io/gh/juaml/junifer/pull/390?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml) | | </details>
github-actions[bot] commented 2024-11-08 11:17:17 +00:00 (Migrated from github.com)
PR Preview Action v1.4.8
Preview removed because the pull request was closed.
2024-11-13 12:09 UTC
[PR Preview Action](https://github.com/rossjrw/pr-preview-action) v1.4.8 :---: Preview removed because the pull request was closed. 2024-11-13 12:09 UTC <!-- Sticky Pull Request Commentpr-preview -->
fraimondo (Migrated from github.com) requested changes 2024-11-08 16:09:16 +00:00
fraimondo (Migrated from github.com) left a comment

I did not repeat my comment several times, but the logic to find the right warper/transform needs to consider both the src and dst spaces everywhere.

Maybe a util function can sever as such, also checking if more than one transform is available and raising an error in such cases.

The only special case is coordinates, as long as one MNI space is available, it should work.

I did not repeat my comment several times, but the logic to find the right warper/transform needs to consider both the src and dst spaces everywhere. Maybe a util function can sever as such, also checking if more than one transform is available and raising an error in such cases. The only special case is coordinates, as long as one MNI space is available, it should work.
fraimondo (Migrated from github.com) commented 2024-11-08 15:51:59 +00:00

can we define a type to avoid this mess here?

can we define a type to avoid this mess here?
@ -146,14 +146,23 @@ Warping to subject's native space
To warp to subject's native space, the dataset needs to provide ``T1w`` and
fraimondo (Migrated from github.com) commented 2024-11-08 15:52:55 +00:00

There's no "preferred", but the one that suits the datagrabber's warp file format.

There's no "preferred", but the one that suits the datagrabber's warp file format.
@ -37,10 +37,8 @@ class ANTsCoordinatesWarper:
target_data : dict
The corresponding item of the data object to which the coordinates
will be applied.
fraimondo (Migrated from github.com) commented 2024-11-08 15:55:14 +00:00

Don't we also need to check that the "warper" is "ants"?

Don't we also need to check that the "warper" is "ants"?
fraimondo (Migrated from github.com) commented 2024-11-08 15:59:48 +00:00

This logic needs to be redefined.

  1. Get which is the transform that we need to apply based on src and dst spaces.
  2. Check which warper is the right one.

We could have fMRIPrep output in which we have both MNI6thGen and 2009c to T1w... which one are we choosing?

This logic needs to be redefined. 1) Get which is the transform that we need to apply based on src and dst spaces. 2) Check which warper is the right one. We could have fMRIPrep output in which we have both MNI6thGen and 2009c to T1w... which one are we choosing?
@ -37,10 +37,8 @@ class FSLCoordinatesWarper:
target_data : dict
The corresponding item of the data object to which the coordinates
will be applied.
fraimondo (Migrated from github.com) commented 2024-11-08 16:00:25 +00:00

Same as before.

Same as before.
@ -83,1 +86,3 @@
if dst == "T1w":
if dst == "native":
# Warp data check
if warp_data is None:
fraimondo (Migrated from github.com) commented 2024-11-08 16:00:34 +00:00

Same as before

Same as before
fraimondo (Migrated from github.com) commented 2024-11-08 16:01:18 +00:00

why is this T1w here?

why is this T1w here?
@ -203,0 +200,4 @@
if not isinstance(depends_on, list):
depends_on = [depends_on]
for entry in depends_on:
# Check dependencies
fraimondo (Migrated from github.com) commented 2024-11-08 16:06:02 +00:00
if not isinstance(depends_on, list):
    depends_on = [depends_on]
for entry in depends_on:
    # Check dependencies
    _check_dependencies(entry)
    # Check external dependencies
    _check_ext_dependencies(entry)
```python if not isinstance(depends_on, list): depends_on = [depends_on] for entry in depends_on: # Check dependencies _check_dependencies(entry) # Check external dependencies _check_ext_dependencies(entry) ```
@ -36,17 +36,21 @@ class UpdateMetaMixin:
for k, v in vars(self).items():
fraimondo (Migrated from github.com) commented 2024-11-08 16:07:20 +00:00

same here, if it's not a list, make it a single element list and avoid duplicated code.

same here, if it's not a list, make it a single element list and avoid duplicated code.
synchon (Migrated from github.com) reviewed 2024-11-11 10:37:29 +00:00
synchon (Migrated from github.com) commented 2024-11-11 10:37:29 +00:00

That's a fair point. What about having a junifer.typing module to take care of this and a lot of other type hinting mess across the codebase?

That's a fair point. What about having a `junifer.typing` module to take care of this and a lot of other type hinting mess across the codebase?
fraimondo (Migrated from github.com) reviewed 2024-11-11 17:29:56 +00:00
fraimondo (Migrated from github.com) commented 2024-11-11 17:29:56 +00:00

+1

+1
synchon (Migrated from github.com) reviewed 2024-11-12 14:49:51 +00:00
synchon (Migrated from github.com) commented 2024-11-12 14:49:51 +00:00

Resolving this as it's now being tracked in #392 .

Resolving this as it's now being tracked in #392 .
synchon (Migrated from github.com) reviewed 2024-11-12 15:19:42 +00:00
@ -83,1 +86,3 @@
if dst == "T1w":
if dst == "native":
# Warp data check
if warp_data is None:
synchon (Migrated from github.com) commented 2024-11-12 15:19:42 +00:00
Because that's provided here: https://github.com/juaml/junifer/blob/2d974e1f1c9c569e5f55f2d7e61460f526d931f0/junifer/data/masks/_masks.py#L536
synchon commented 2024-11-13 10:02:27 +00:00 (Migrated from github.com)

@fraimondo Updated the logic now, would appreciate your feedback.

@fraimondo Updated the logic now, would appreciate your feedback.
fraimondo (Migrated from github.com) requested changes 2024-11-13 10:50:25 +00:00
fraimondo (Migrated from github.com) left a comment

The logic is now good.

There are some comments I made on some issues that are not related to this refactor, but might be worth addressing here.

Like assuming that dst == "T1w" means warp to native space.

The logic is now good. There are some comments I made on some issues that are not related to this refactor, but might be worth addressing here. Like assuming that `dst == "T1w"` means warp to native space.
@ -83,1 +86,3 @@
if dst == "T1w":
if dst == "native":
# Warp data check
if warp_data is None:
fraimondo (Migrated from github.com) commented 2024-11-13 10:37:32 +00:00

I meant that it does not make any sense to asume that T1w is native space.

The comment says "native space warping" and then check if dst = "T1w". This would be true if the space of the T1w is native.

I meant that it does not make any sense to asume that T1w is native space. The comment says "native space warping" and then check if `dst = "T1w"`. This would be true if the space of the T1w is native.
@ -69,3 +83,4 @@
# Create element-specific tempdir for storing post-warping assets
element_tempdir = WorkDirManager().get_element_tempdir(
prefix="fsl_warper"
fraimondo (Migrated from github.com) commented 2024-11-13 10:48:59 +00:00

Based on this logic and the get_native_warper function I asume that it's not possible for a datagrabber to provide transforms for the same space pair using two different warpers. Is this assumption correct?

Based on this logic and the `get_native_warper` function I asume that it's not possible for a datagrabber to provide transforms for the same space pair using two different warpers. Is this assumption correct?
fraimondo (Migrated from github.com) commented 2024-11-13 10:49:23 +00:00

If so, can we add the check in the "validate" section of the datagrabber to make it more explicit?

If so, can we add the check in the "validate" section of the datagrabber to make it more explicit?
synchon (Migrated from github.com) reviewed 2024-11-13 11:04:23 +00:00
synchon (Migrated from github.com) reviewed 2024-11-13 11:06:02 +00:00
@ -69,3 +83,4 @@
# Create element-specific tempdir for storing post-warping assets
element_tempdir = WorkDirManager().get_element_tempdir(
prefix="fsl_warper"
synchon (Migrated from github.com) commented 2024-11-13 11:06:02 +00:00

Based on this logic and the get_native_warper function I asume that it's not possible for a datagrabber to provide transforms for the same space pair using two different warpers. Is this assumption correct?

Yes that's why we raise an error now just as we discussed. We can update the logic to take care of that when we actually have the transform files for two different warpers.

> Based on this logic and the `get_native_warper` function I asume that it's not possible for a datagrabber to provide transforms for the same space pair using two different warpers. Is this assumption correct? Yes that's why we raise an error now just as we discussed. We can update the logic to take care of that when we actually have the transform files for two different warpers.
synchon (Migrated from github.com) reviewed 2024-11-13 11:06:34 +00:00
@ -69,3 +83,4 @@
# Create element-specific tempdir for storing post-warping assets
element_tempdir = WorkDirManager().get_element_tempdir(
prefix="fsl_warper"
synchon (Migrated from github.com) commented 2024-11-13 11:06:34 +00:00

If so, can we add the check in the "validate" section of the datagrabber to make it more explicit?

Not sure I get you, can you please explain?

> If so, can we add the check in the "validate" section of the datagrabber to make it more explicit? Not sure I get you, can you please explain?
synchon (Migrated from github.com) reviewed 2024-11-13 11:06:57 +00:00
@ -83,1 +86,3 @@
if dst == "T1w":
if dst == "native":
# Warp data check
if warp_data is None:
synchon (Migrated from github.com) commented 2024-11-13 11:06:57 +00:00

Fair enough.

Fair enough.
synchon (Migrated from github.com) reviewed 2024-11-13 11:07:13 +00:00
synchon (Migrated from github.com) commented 2024-11-13 11:07:13 +00:00

Addressed.

Addressed.
fraimondo (Migrated from github.com) reviewed 2024-11-13 11:16:04 +00:00
fraimondo (Migrated from github.com) reviewed 2024-11-13 11:17:17 +00:00
@ -69,3 +83,4 @@
# Create element-specific tempdir for storing post-warping assets
element_tempdir = WorkDirManager().get_element_tempdir(
prefix="fsl_warper"
fraimondo (Migrated from github.com) commented 2024-11-13 11:17:17 +00:00

I got mixed up. We don't really have a "validator" for post-datagrabbing consistency.

What I wanted to prevent is that someone creates a datagrabber which gives multiple warping options... not for now.

I got mixed up. We don't really have a "validator" for post-datagrabbing consistency. What I wanted to prevent is that someone creates a datagrabber which gives multiple warping options... not for now.
synchon (Migrated from github.com) reviewed 2024-11-13 11:20:10 +00:00
synchon (Migrated from github.com) reviewed 2024-11-13 11:20:42 +00:00
@ -69,3 +83,4 @@
# Create element-specific tempdir for storing post-warping assets
element_tempdir = WorkDirManager().get_element_tempdir(
prefix="fsl_warper"
synchon (Migrated from github.com) commented 2024-11-13 11:20:42 +00:00

Resolving then.

Resolving then.
fraimondo (Migrated from github.com) approved these changes 2024-11-13 11:45:21 +00:00
Sign in to join this conversation.
No reviewers
No milestone
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
juaml/junifer!390
No description provided.