[ENH]: Use junifer-data datalad dataset for fetching data #418

2025-01-08T12:13:44Z

synchon commented

2025-01-08 12:13:44 +00:00

(Migrated from github.com)

description of feature/fix
tests added/passed
add an entry for the latest changes

This PR adapts junifer-data datalad dataset at https://github.com/juaml/junifer-data to provide data for junifer.

* [x] description of feature/fix * [x] tests added/passed * [x] add an entry for the latest changes This PR adapts `junifer-data` datalad dataset at https://github.com/juaml/junifer-data to provide data for junifer.

github-actions[bot] commented

2025-01-08 13:33:07 +00:00

(Migrated from github.com)

PR Preview Action v1.6.0
🚀 View preview at https://juaml.github.io/junifer/pr-preview/pr-418/
Built to branch `gh-pages` at 2025-01-27 11:19 UTC. Preview will be ready when the GitHub Pages deployment is complete.

[PR Preview Action](https://github.com/rossjrw/pr-preview-action) v1.6.0 :---: | <p></p> :rocket: View preview at <br> https://juaml.github.io/junifer/pr-preview/pr-418/ <br><br> | <h6>Built to branch [`gh-pages`](https://github.com/juaml/junifer/tree/gh-pages) at 2025-01-27 11:19 UTC. <br> Preview will be ready when the [GitHub Pages deployment](https://github.com/juaml/junifer/deployments) is complete. <br><br> </h6>

codecov[bot] commented

2025-01-08 17:18:50 +00:00

(Migrated from github.com)

Attention: Patch coverage is 0% with 123 lines in your changes missing coverage. Please review.

Project coverage is 0.01%. Comparing base (5135251) to head (a8582b6).

Files with missing lines	Patch %	Lines
junifer/data/parcellations/_parcellations.py	0.00%	60 Missing ⚠️
junifer/data/utils.py	0.00%	34 Missing ⚠️
junifer/data/masks/_masks.py	0.00%	16 Missing ⚠️
junifer/data/coordinates/_coordinates.py	0.00%	11 Missing ⚠️
junifer/data/template_spaces.py	0.00%	2 Missing ⚠️

Additional details and impacted files

@@          Coverage Diff           @@
##            main    #418    +/-   ##
======================================
  Coverage   0.01%   0.01%            
======================================
  Files        133     133            
  Lines       5795    5652   -143     
======================================
  Hits           1       1            
+ Misses      5794    5651   -143

Flag	Coverage Δ
docs	`100.00% <ø> (ø)`
junifer	`0.00% <0.00%> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
junifer/data/template_spaces.py	`0.00% <0.00%> (ø)`
junifer/data/coordinates/_coordinates.py	`0.00% <0.00%> (ø)`
junifer/data/masks/_masks.py	`0.00% <0.00%> (ø)`
junifer/data/utils.py	`0.00% <0.00%> (ø)`
junifer/data/parcellations/_parcellations.py	`0.00% <0.00%> (ø)`

## [Codecov](https://app.codecov.io/gh/juaml/junifer/pull/418?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml) Report Attention: Patch coverage is `0%` with `123 lines` in your changes missing coverage. Please review. > Project coverage is 0.01%. Comparing base [(`5135251`)](https://app.codecov.io/gh/juaml/junifer/commit/51352511e47f84c5873d2758055f5047bf790041?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml) to head [(`a8582b6`)](https://app.codecov.io/gh/juaml/junifer/commit/a8582b6545556b989052203f7f996050e74b99fe?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml). | [Files with missing lines](https://app.codecov.io/gh/juaml/junifer/pull/418?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml) | Patch % | Lines | |---|---|---| | [junifer/data/parcellations/\_parcellations.py](https://app.codecov.io/gh/juaml/junifer/pull/418?src=pr&el=tree&filepath=junifer%2Fdata%2Fparcellations%2F_parcellations.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml#diff-anVuaWZlci9kYXRhL3BhcmNlbGxhdGlvbnMvX3BhcmNlbGxhdGlvbnMucHk=) | 0.00% | [60 Missing :warning: ](https://app.codecov.io/gh/juaml/junifer/pull/418?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml) | | [junifer/data/utils.py](https://app.codecov.io/gh/juaml/junifer/pull/418?src=pr&el=tree&filepath=junifer%2Fdata%2Futils.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml#diff-anVuaWZlci9kYXRhL3V0aWxzLnB5) | 0.00% | [34 Missing :warning: ](https://app.codecov.io/gh/juaml/junifer/pull/418?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml) | | [junifer/data/masks/\_masks.py](https://app.codecov.io/gh/juaml/junifer/pull/418?src=pr&el=tree&filepath=junifer%2Fdata%2Fmasks%2F_masks.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml#diff-anVuaWZlci9kYXRhL21hc2tzL19tYXNrcy5weQ==) | 0.00% | [16 Missing :warning: ](https://app.codecov.io/gh/juaml/junifer/pull/418?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml) | | [junifer/data/coordinates/\_coordinates.py](https://app.codecov.io/gh/juaml/junifer/pull/418?src=pr&el=tree&filepath=junifer%2Fdata%2Fcoordinates%2F_coordinates.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml#diff-anVuaWZlci9kYXRhL2Nvb3JkaW5hdGVzL19jb29yZGluYXRlcy5weQ==) | 0.00% | [11 Missing :warning: ](https://app.codecov.io/gh/juaml/junifer/pull/418?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml) | | [junifer/data/template\_spaces.py](https://app.codecov.io/gh/juaml/junifer/pull/418?src=pr&el=tree&filepath=junifer%2Fdata%2Ftemplate_spaces.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml#diff-anVuaWZlci9kYXRhL3RlbXBsYXRlX3NwYWNlcy5weQ==) | 0.00% | [2 Missing :warning: ](https://app.codecov.io/gh/juaml/junifer/pull/418?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml) | <details><summary>Additional details and impacted files</summary> [![Impacted file tree graph](https://app.codecov.io/gh/juaml/junifer/pull/418/graphs/tree.svg?width=650&height=150&src=pr&token=5H21JuZXMw&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml)](https://app.codecov.io/gh/juaml/junifer/pull/418?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml) ```diff @@ Coverage Diff @@ ## main #418 +/- ## ====================================== Coverage 0.01% 0.01% ====================================== Files 133 133 Lines 5795 5652 -143 ====================================== Hits 1 1 + Misses 5794 5651 -143 ``` | [Flag](https://app.codecov.io/gh/juaml/junifer/pull/418/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml) | Coverage Δ | | |---|---|---| | [docs](https://app.codecov.io/gh/juaml/junifer/pull/418/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml) | `100.00% <ø> (ø)` | | | [junifer](https://app.codecov.io/gh/juaml/junifer/pull/418/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml) | `0.00% <0.00%> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml#carryforward-flags-in-the-pull-request-comment) to find out more. | [Files with missing lines](https://app.codecov.io/gh/juaml/junifer/pull/418?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml) | Coverage Δ | | |---|---|---| | [junifer/data/template\_spaces.py](https://app.codecov.io/gh/juaml/junifer/pull/418?src=pr&el=tree&filepath=junifer%2Fdata%2Ftemplate_spaces.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml#diff-anVuaWZlci9kYXRhL3RlbXBsYXRlX3NwYWNlcy5weQ==) | `0.00% <0.00%> (ø)` | | | [junifer/data/coordinates/\_coordinates.py](https://app.codecov.io/gh/juaml/junifer/pull/418?src=pr&el=tree&filepath=junifer%2Fdata%2Fcoordinates%2F_coordinates.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml#diff-anVuaWZlci9kYXRhL2Nvb3JkaW5hdGVzL19jb29yZGluYXRlcy5weQ==) | `0.00% <0.00%> (ø)` | | | [junifer/data/masks/\_masks.py](https://app.codecov.io/gh/juaml/junifer/pull/418?src=pr&el=tree&filepath=junifer%2Fdata%2Fmasks%2F_masks.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml#diff-anVuaWZlci9kYXRhL21hc2tzL19tYXNrcy5weQ==) | `0.00% <0.00%> (ø)` | | | [junifer/data/utils.py](https://app.codecov.io/gh/juaml/junifer/pull/418?src=pr&el=tree&filepath=junifer%2Fdata%2Futils.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml#diff-anVuaWZlci9kYXRhL3V0aWxzLnB5) | `0.00% <0.00%> (ø)` | | | [junifer/data/parcellations/\_parcellations.py](https://app.codecov.io/gh/juaml/junifer/pull/418?src=pr&el=tree&filepath=junifer%2Fdata%2Fparcellations%2F_parcellations.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml#diff-anVuaWZlci9kYXRhL3BhcmNlbGxhdGlvbnMvX3BhcmNlbGxhdGlvbnMucHk=) | `0.00% <0.00%> (ø)` | | </details>

fraimondo (Migrated from github.com) requested changes

2025-01-10 06:29:08 +00:00

junifer/data/utils.py

					
				@ -114,3 +122,96 @@ def get_native_warper(

				        )

This should be DEBUG info.

junifer/data/utils.py

					
				@ -117,0 +149,4 @@

				    # else clone a fresh copy

				    if dl.Dataset(data_dir).is_installed():

				        logger.debug(f"Found existing junifer-data at: {data_dir.resolve()}")

				        return dl.Dataset(data_dir)

This function needs to check that we are also using/cloning the exact junifer data version that we want.

In our last discussion, we said we will have a subdirectory inside the junifer_data with the corresponding version to be used with the currently running junifer version.

So somewhere in junifer code, we will have a JUNIFER_DATA_VERSION variable with a string/number/id.

Then we will look if the directory junifer_data/{JUNIFER_DATA_VERSION} exists, that its a datalad dataset, it's clean and its tag corresponds to JUNIFER_DATA_VERSION.

This way we can keep junifer data for several installed junifer versions, each one consistently using the data.

This function needs to check that we are also using/cloning the exact junifer data version that we want. In our last discussion, we said we will have a subdirectory inside the `junifer_data` with the corresponding version to be used with the currently running junifer version. So somewhere in junifer code, we will have a JUNIFER_DATA_VERSION variable with a string/number/id. Then we will look if the directory `junifer_data/{JUNIFER_DATA_VERSION}` exists, that its a datalad dataset, it's clean and its tag corresponds to JUNIFER_DATA_VERSION. This way we can keep junifer data for several installed junifer versions, each one consistently using the data.

synchon (Migrated from github.com) reviewed

2025-01-10 13:01:24 +00:00

junifer/data/utils.py

					
				@ -117,0 +149,4 @@

				    # else clone a fresh copy

				    if dl.Dataset(data_dir).is_installed():

				        logger.debug(f"Found existing junifer-data at: {data_dir.resolve()}")

				        return dl.Dataset(data_dir)

This PR is part of that migration and serves to move manual inconsistent asset downloads to using junifer-data via datalad. The idea is to move all dataset validating and data fetching machinery to https://github.com/juaml/junifer-data-client and junifer using its API to fetch data as required using the technique you described. v1.0.0 image of junifer-data is being built and will be published soon.

The next PR will adapt the usage of junifer-data-client API while taking care of versions.

This PR is part of that migration and serves to move manual inconsistent asset downloads to using `junifer-data` via datalad. The idea is to move all dataset validating and data fetching machinery to https://github.com/juaml/junifer-data-client and `junifer` using its API to fetch data as required using the technique you described. v1.0.0 image of `junifer-data` is being built and will be published soon. The next PR will adapt the usage of `junifer-data-client` API while taking care of versions.