juaml/junifer

Watch

Fork 0

[ENH]: Move data downloading/handling to `junifer-data` package. #363

Merged

synchon merged 13 commits from refactor/junifer-data-api into main

2025-02-13 14:11:39 +00:00

synchon commented

2025-01-21 15:17:43 +00:00

(Migrated from github.com)

Are you requiring a new dataset or marker?

I understand this is not a marker or dataset request

Which feature do you want to include?

Currently, Junifer is handling the data, downloading from various sources. Ideally, this should be centralized and versiones.

How do you imagine this integrated in junifer?

Junifer uses the junifer-data API.
junifer-data keeps a datalad dataset with all data objects, including externals. Each commit to this datasets has a tag.
junifer knows exactly which is the tag that we are dealing with, so each junifer version will always work with the same data files.
We have a command download or similar that makes junifer download all (or the requested) files to be used later on with the run command.

Do you have a sample code that implements this outside of junifer?

No response

Anything else to say?

No response

### Are you requiring a new dataset or marker? - [X] I understand this is not a marker or dataset request ### Which feature do you want to include? Currently, Junifer is handling the data, downloading from various sources. Ideally, this should be centralized and versiones. ### How do you imagine this integrated in junifer? 1. Junifer uses the `junifer-data` API. 2. `junifer-data` keeps a datalad dataset with all data objects, including externals. Each commit to this datasets has a tag. 3. `junifer` knows exactly which is the tag that we are dealing with, so each junifer version will always work with the same data files. 4. We have a command `download` or similar that makes junifer download all (or the requested) files to be used later on with the `run` command. ### Do you have a sample code that implements this outside of junifer? _No response_ ### Anything else to say? _No response_

codecov[bot] commented

2025-01-21 15:18:12 +00:00

(Migrated from github.com)

All modified and coverable lines are covered by tests ✅

Project coverage is 85.67%. Comparing base (b39d56b) to head (700aa5f).
Report is 14 commits behind head on main.

❌ Your project status has failed because the head coverage (85.67%) is below the target coverage (90.00%). You can increase the head coverage or adjust the target coverage.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #363      +/-   ##
==========================================
+ Coverage   85.58%   85.67%   +0.09%     
==========================================
  Files         133      133              
  Lines        5652     5633      -19     
  Branches      958      954       -4     
==========================================
- Hits         4837     4826      -11     
+ Misses        624      618       -6     
+ Partials      191      189       -2

Flag	Coverage Δ
junifer	`85.67% <100.00%> (+0.09%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
junifer/data/coordinates/_coordinates.py	`79.71% <100.00%> (+0.29%)`	⬆️
junifer/data/masks/_masks.py	`68.57% <100.00%> (ø)`
junifer/data/parcellations/_parcellations.py	`92.12% <100.00%> (+0.17%)`	⬆️
junifer/data/template_spaces.py	`81.81% <100.00%> (+0.42%)`	⬆️
junifer/data/utils.py	`62.50% <100.00%> (-4.17%)`	⬇️
junifer/typing/_typing.py	`100.00% <100.00%> (ø)`

## [Codecov](https://app.codecov.io/gh/juaml/junifer/pull/363?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml) Report All modified and coverable lines are covered by tests :white_check_mark: > Project coverage is 85.67%. Comparing base [(`b39d56b`)](https://app.codecov.io/gh/juaml/junifer/commit/b39d56b080410b3df915b293e568e477d6c95866?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml) to head [(`700aa5f`)](https://app.codecov.io/gh/juaml/junifer/commit/700aa5f09c813c4b12d27258d21f7abadb211c7e?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml). > Report is 14 commits behind head on main. :x: Your project status has failed because the head coverage (85.67%) is below the target coverage (90.00%). You can increase the head coverage or adjust the [target](https://docs.codecov.com/docs/commit-status#target) coverage. <details><summary>Additional details and impacted files</summary> [![Impacted file tree graph](https://app.codecov.io/gh/juaml/junifer/pull/363/graphs/tree.svg?width=650&height=150&src=pr&token=5H21JuZXMw&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml)](https://app.codecov.io/gh/juaml/junifer/pull/363?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml) ```diff @@ Coverage Diff @@ ## main #363 +/- ## ========================================== + Coverage 85.58% 85.67% +0.09% ========================================== Files 133 133 Lines 5652 5633 -19 Branches 958 954 -4 ========================================== - Hits 4837 4826 -11 + Misses 624 618 -6 + Partials 191 189 -2 ``` | [Flag](https://app.codecov.io/gh/juaml/junifer/pull/363/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml) | Coverage Δ | | |---|---|---| | [junifer](https://app.codecov.io/gh/juaml/junifer/pull/363/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml) | `85.67% <100.00%> (+0.09%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml#carryforward-flags-in-the-pull-request-comment) to find out more. | [Files with missing lines](https://app.codecov.io/gh/juaml/junifer/pull/363?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml) | Coverage Δ | | |---|---|---| | [junifer/data/coordinates/\_coordinates.py](https://app.codecov.io/gh/juaml/junifer/pull/363?src=pr&el=tree&filepath=junifer%2Fdata%2Fcoordinates%2F_coordinates.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml#diff-anVuaWZlci9kYXRhL2Nvb3JkaW5hdGVzL19jb29yZGluYXRlcy5weQ==) | `79.71% <100.00%> (+0.29%)` | :arrow_up: | | [junifer/data/masks/\_masks.py](https://app.codecov.io/gh/juaml/junifer/pull/363?src=pr&el=tree&filepath=junifer%2Fdata%2Fmasks%2F_masks.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml#diff-anVuaWZlci9kYXRhL21hc2tzL19tYXNrcy5weQ==) | `68.57% <100.00%> (ø)` | | | [junifer/data/parcellations/\_parcellations.py](https://app.codecov.io/gh/juaml/junifer/pull/363?src=pr&el=tree&filepath=junifer%2Fdata%2Fparcellations%2F_parcellations.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml#diff-anVuaWZlci9kYXRhL3BhcmNlbGxhdGlvbnMvX3BhcmNlbGxhdGlvbnMucHk=) | `92.12% <100.00%> (+0.17%)` | :arrow_up: | | [junifer/data/template\_spaces.py](https://app.codecov.io/gh/juaml/junifer/pull/363?src=pr&el=tree&filepath=junifer%2Fdata%2Ftemplate_spaces.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml#diff-anVuaWZlci9kYXRhL3RlbXBsYXRlX3NwYWNlcy5weQ==) | `81.81% <100.00%> (+0.42%)` | :arrow_up: | | [junifer/data/utils.py](https://app.codecov.io/gh/juaml/junifer/pull/363?src=pr&el=tree&filepath=junifer%2Fdata%2Futils.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml#diff-anVuaWZlci9kYXRhL3V0aWxzLnB5) | `62.50% <100.00%> (-4.17%)` | :arrow_down: | | [junifer/typing/\_typing.py](https://app.codecov.io/gh/juaml/junifer/pull/363?src=pr&el=tree&filepath=junifer%2Ftyping%2F_typing.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml#diff-anVuaWZlci90eXBpbmcvX3R5cGluZy5weQ==) | `100.00% <100.00%> (ø)` | | </details>

synchon commented

2025-01-24 16:30:26 +00:00

(Migrated from github.com)

We have a command download or similar that makes junifer download all (or the requested) files to be used later on with the run command.

This is not yet implemented and would depend on junifer-data API. Will implement after the necessary API is available from junifer-data.

> 4. We have a command download or similar that makes junifer download all (or the requested) files to be used later on with the run command. This is not yet implemented and would depend on `junifer-data` API. Will implement after the necessary API is available from `junifer-data`.

github-actions[bot] commented

2025-01-24 17:27:04 +00:00

(Migrated from github.com)

PR Preview Action v1.6.0
🚀 View preview at https://juaml.github.io/junifer/pr-preview/pr-363/
Built to branch `gh-pages` at 2025-01-27 11:23 UTC. Preview will be ready when the GitHub Pages deployment is complete.

[PR Preview Action](https://github.com/rossjrw/pr-preview-action) v1.6.0 :---: | <p></p> :rocket: View preview at <br> https://juaml.github.io/junifer/pr-preview/pr-363/ <br><br> | <h6>Built to branch [`gh-pages`](https://github.com/juaml/junifer/tree/gh-pages) at 2025-01-27 11:23 UTC. <br> Preview will be ready when the [GitHub Pages deployment](https://github.com/juaml/junifer/deployments) is complete. <br><br> </h6>

synchon commented

2025-02-10 08:57:13 +00:00

(Migrated from github.com)

We have a command download or similar that makes junifer download all (or the requested) files to be used later on with the run command.

This is not yet implemented and would depend on junifer-data API. Will implement after the necessary API is available from junifer-data.

junifer-data-client v1.1.0 now has a download command to do this via CLI only.

> > 4. We have a command download or similar that makes junifer download all (or the requested) files to be used later on with the run command. > > This is not yet implemented and would depend on `junifer-data` API. Will implement after the necessary API is available from `junifer-data`. `junifer-data`-client v1.1.0 now has a `download` command to do this via CLI only.

fraimondo (Migrated from github.com) requested changes

2025-02-12 08:26:17 +00:00

junifer/data/utils.py

I would keep this not semver. Just incremental tags. Also easier to handle in case we want to "clean" the directory. Any new modification to the junifer-data package is a new version.

pyproject.toml

Indeed here is confusing. The junifer_data package can be updated, but the data can be the same, no? I mean, datalad dataset version is the same.

So in this case, to avoid confusion, i'll have the junifer-data following semver and the dataset version tag as incremental numbers.

Indeed here is confusing. The `junifer_data` package can be updated, but the data can be the same, no? I mean, datalad dataset version is the same. So in this case, to avoid confusion, i'll have the junifer-data following semver and the dataset version tag as incremental numbers.

synchon (Migrated from github.com) reviewed

2025-02-12 10:56:41 +00:00

junifer/data/utils.py

No problem with no semver and just int. Not sure about the rationale of easier "clean".

synchon (Migrated from github.com) reviewed

2025-02-12 11:00:21 +00:00

pyproject.toml

junifer_data is the client API which adds and improves the API we use to interact with the junifer-data datalad dataset.

junifer_data in pyproject.toml refers to: https://github.com/juaml/junifer-data-client
JUNIFER_DATA_VERSION in junifer/data/utils.py refers to https://github.com/juaml/junifer-data

`junifer_data` is the client API which adds and improves the API we use to interact with the `junifer-data` datalad dataset. - `junifer_data` in `pyproject.toml` refers to: https://github.com/juaml/junifer-data-client - `JUNIFER_DATA_VERSION` in `junifer/data/utils.py` refers to https://github.com/juaml/junifer-data

fraimondo (Migrated from github.com) approved these changes

2025-02-13 08:36:44 +00:00

pyproject.toml

perfect

fraimondo commented

2025-02-13 08:36:57 +00:00

(Migrated from github.com)

CI is not green though

synchon commented

2025-02-13 09:35:08 +00:00

(Migrated from github.com)

CI is not green though

It's a new problem with gh-action where numpy is not getting built for the docs. Will take care of it after getting the important stuff in.

> CI is not green though It's a new problem with gh-action where numpy is not getting built for the docs. Will take care of it after getting the important stuff in.