juaml/junifer
Watch
3
0
Fork
You've already forked junifer
 0
Code Issues 14 Pull requests 2 Packages Activity Actions

[ENH]: Introduce get_xfm for fetching transformation files #297

Merged
synchon merged 14 commits from feat/xfm-download into main 2024-02-01 08:32:57 +00:00
Conversation 17 Commits 14 Files changed 29 +180 -13
synchon commented 2024-01-30 13:37:06 +00:00 (Migrated from github.com)
Copy link
  • description of feature/fix
  • tests added/passed
  • add an entry for the latest changes

This PR introduces a new function junifer.data.get_xfm() to fetch transformation files for moving between template spaces.

* [x] description of feature/fix * [x] tests added/passed * [x] add an entry for the latest changes This PR introduces a new function `junifer.data.get_xfm()` to fetch transformation files for moving between template spaces.
codecov[bot] commented 2024-01-30 13:38:03 +00:00 (Migrated from github.com)
Copy link

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (4acf8b2) 89.23% compared to head (4d9c5cf) 89.24%.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main     #297      +/-   ##
==========================================
+ Coverage   89.23%   89.24%   +0.01%     
==========================================
  Files         100      101       +1     
  Lines        4449     4454       +5     
  Branches      853      853              
==========================================
+ Hits         3970     3975       +5     
  Misses        341      341              
  Partials      138      138
Flag Coverage Δ
docs 100.00% <ø> (ø)
junifer 89.24% <100.00%> (+0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Coverage Δ
junifer/api/cli.py 70.47% <ø> (ø)
junifer/api/functions.py 96.35% <ø> (ø)
junifer/data/__init__.py 100.00% <100.00%> (ø)
junifer/data/template_spaces.py 100.00% <100.00%> (ø)
junifer/data/utils.py 100.00% <ø> (ø)
...r/external/nilearn/junifer_nifti_spheres_masker.py 92.39% <ø> (ø)
junifer/markers/falff/falff_base.py 76.19% <ø> (ø)
junifer/markers/falff/falff_parcels.py 100.00% <ø> (ø)
junifer/markers/falff/falff_spheres.py 100.00% <ø> (ø)
...ivity/crossparcellation_functional_connectivity.py 100.00% <ø> (ø)
... and 9 more
## [Codecov](https://app.codecov.io/gh/juaml/junifer/pull/297?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml) Report All modified and coverable lines are covered by tests :white_check_mark: > Comparison is base [(`4acf8b2`)](https://app.codecov.io/gh/juaml/junifer/commit/4acf8b2133dc33b3d79b1c41a80f63592ed1f280?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml) 89.23% compared to head [(`4d9c5cf`)](https://app.codecov.io/gh/juaml/junifer/pull/297?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml) 89.24%. <details><summary>Additional details and impacted files</summary> [![Impacted file tree graph](https://app.codecov.io/gh/juaml/junifer/pull/297/graphs/tree.svg?width=650&height=150&src=pr&token=5H21JuZXMw&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml)](https://app.codecov.io/gh/juaml/junifer/pull/297?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml) ```diff @@ Coverage Diff @@ ## main #297 +/- ## ========================================== + Coverage 89.23% 89.24% +0.01% ========================================== Files 100 101 +1 Lines 4449 4454 +5 Branches 853 853 ========================================== + Hits 3970 3975 +5 Misses 341 341 Partials 138 138 ``` | [Flag](https://app.codecov.io/gh/juaml/junifer/pull/297/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml) | Coverage Δ | | |---|---|---| | [docs](https://app.codecov.io/gh/juaml/junifer/pull/297/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml) | `100.00% <ø> (ø)` | | | [junifer](https://app.codecov.io/gh/juaml/junifer/pull/297/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml) | `89.24% <100.00%> (+0.01%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml#carryforward-flags-in-the-pull-request-comment) to find out more. | [Files](https://app.codecov.io/gh/juaml/junifer/pull/297?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml) | Coverage Δ | | |---|---|---| | [junifer/api/cli.py](https://app.codecov.io/gh/juaml/junifer/pull/297?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml#diff-anVuaWZlci9hcGkvY2xpLnB5) | `70.47% <ø> (ø)` | | | [junifer/api/functions.py](https://app.codecov.io/gh/juaml/junifer/pull/297?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml#diff-anVuaWZlci9hcGkvZnVuY3Rpb25zLnB5) | `96.35% <ø> (ø)` | | | [junifer/data/\_\_init\_\_.py](https://app.codecov.io/gh/juaml/junifer/pull/297?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml#diff-anVuaWZlci9kYXRhL19faW5pdF9fLnB5) | `100.00% <100.00%> (ø)` | | | [junifer/data/template\_spaces.py](https://app.codecov.io/gh/juaml/junifer/pull/297?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml#diff-anVuaWZlci9kYXRhL3RlbXBsYXRlX3NwYWNlcy5weQ==) | `100.00% <100.00%> (ø)` | | | [junifer/data/utils.py](https://app.codecov.io/gh/juaml/junifer/pull/297?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml#diff-anVuaWZlci9kYXRhL3V0aWxzLnB5) | `100.00% <ø> (ø)` | | | [...r/external/nilearn/junifer\_nifti\_spheres\_masker.py](https://app.codecov.io/gh/juaml/junifer/pull/297?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml#diff-anVuaWZlci9leHRlcm5hbC9uaWxlYXJuL2p1bmlmZXJfbmlmdGlfc3BoZXJlc19tYXNrZXIucHk=) | `92.39% <ø> (ø)` | | | [junifer/markers/falff/falff\_base.py](https://app.codecov.io/gh/juaml/junifer/pull/297?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml#diff-anVuaWZlci9tYXJrZXJzL2ZhbGZmL2ZhbGZmX2Jhc2UucHk=) | `76.19% <ø> (ø)` | | | [junifer/markers/falff/falff\_parcels.py](https://app.codecov.io/gh/juaml/junifer/pull/297?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml#diff-anVuaWZlci9tYXJrZXJzL2ZhbGZmL2ZhbGZmX3BhcmNlbHMucHk=) | `100.00% <ø> (ø)` | | | [junifer/markers/falff/falff\_spheres.py](https://app.codecov.io/gh/juaml/junifer/pull/297?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml#diff-anVuaWZlci9tYXJrZXJzL2ZhbGZmL2ZhbGZmX3NwaGVyZXMucHk=) | `100.00% <ø> (ø)` | | | [...ivity/crossparcellation\_functional\_connectivity.py](https://app.codecov.io/gh/juaml/junifer/pull/297?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml#diff-anVuaWZlci9tYXJrZXJzL2Z1bmN0aW9uYWxfY29ubmVjdGl2aXR5L2Nyb3NzcGFyY2VsbGF0aW9uX2Z1bmN0aW9uYWxfY29ubmVjdGl2aXR5LnB5) | `100.00% <ø> (ø)` | | | ... and [9 more](https://app.codecov.io/gh/juaml/junifer/pull/297?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml) | | </details>
github-actions[bot] commented 2024-01-30 14:05:43 +00:00 (Migrated from github.com)
Copy link
PR Preview Action v1.4.6
Preview removed because the pull request was closed.
2024-02-01 08:37 UTC
[PR Preview Action](https://github.com/rossjrw/pr-preview-action) v1.4.6 :---: Preview removed because the pull request was closed. 2024-02-01 08:37 UTC <!-- Sticky Pull Request Commentpr-preview -->
fraimondo (Migrated from github.com) reviewed 2024-01-31 10:27:09 +00:00
junifer/data/template_spaces.py
@ -0,0 +1,91 @@
"""Provide functions for template spaces."""
fraimondo (Migrated from github.com) commented 2024-01-31 10:21:13 +00:00
Copy link

Can we make it the juaml organization in g-node.org?

Can we make it the juaml organization in g-node.org?
junifer/data/template_spaces.py
@ -0,0 +85,4 @@
)
else:
with open(xfm_file_path, "ab") as f:
for chunk in resp.iter_bytes():
fraimondo (Migrated from github.com) commented 2024-01-31 10:25:19 +00:00
Copy link

Is iter_bytes still streaming? or the content is already in memory? If it is still streaming, an exception might occur here an it will not be catch.

Is `iter_bytes` still streaming? or the content is already in memory? If it is still streaming, an exception might occur here an it will not be catch.
junifer/data/tests/test_template_spaces.py
@ -0,0 +12,4 @@
@pytest.mark.skipif(
socket.gethostname() != "juseless",
fraimondo (Migrated from github.com) commented 2024-01-31 10:25:46 +00:00
Copy link

Why only for juseless?

Why only for juseless?
pyproject.toml
@ -47,3 +47,4 @@
"importlib_metadata; python_version<'3.10'",
"h5py>=3.8.0,<3.10",
"httpx[http2]==0.26.0",
]
fraimondo (Migrated from github.com) commented 2024-01-31 10:27:01 +00:00
Copy link

Is it that strict the dependency?

Is it that strict the dependency?
synchon (Migrated from github.com) reviewed 2024-01-31 10:30:29 +00:00
junifer/data/template_spaces.py
@ -0,0 +1,91 @@
"""Provide functions for template spaces."""
synchon (Migrated from github.com) commented 2024-01-31 10:30:28 +00:00
Copy link

Yeah we can, since I don't have permission to make a repo in the org, I made it in my account.

Yeah we can, since I don't have permission to make a repo in the org, I made it in my account.
synchon (Migrated from github.com) reviewed 2024-01-31 10:34:12 +00:00
junifer/data/template_spaces.py
@ -0,0 +85,4 @@
)
else:
with open(xfm_file_path, "ab") as f:
for chunk in resp.iter_bytes():
synchon (Migrated from github.com) commented 2024-01-31 10:34:12 +00:00
Copy link

It's a generator basically thus it's still streaming. I don't find any exceptions being raised there in httpx's source code.

It's a generator basically thus it's still streaming. I don't find any exceptions being raised there in `httpx`'s source code.
synchon (Migrated from github.com) reviewed 2024-01-31 10:34:39 +00:00
junifer/data/tests/test_template_spaces.py
@ -0,0 +12,4 @@
@pytest.mark.skipif(
socket.gethostname() != "juseless",
synchon (Migrated from github.com) commented 2024-01-31 10:34:39 +00:00
Copy link

The download takes well around 10 mins from Gin.

The download takes well around 10 mins from Gin.
synchon (Migrated from github.com) reviewed 2024-01-31 10:35:42 +00:00
pyproject.toml
@ -47,3 +47,4 @@
"importlib_metadata; python_version<'3.10'",
"h5py>=3.8.0,<3.10",
"httpx[http2]==0.26.0",
]
synchon (Migrated from github.com) commented 2024-01-31 10:35:42 +00:00
Copy link

It's always recommended to keep it like so when dealing with libraries which aren't 1.0 yet as the APIs can change very often.

It's always recommended to keep it like so when dealing with libraries which aren't 1.0 yet as the APIs can change very often.
fraimondo (Migrated from github.com) reviewed 2024-01-31 10:45:31 +00:00
junifer/data/template_spaces.py
@ -0,0 +1,91 @@
"""Provide functions for template spaces."""
fraimondo (Migrated from github.com) commented 2024-01-31 10:45:31 +00:00
Copy link

Now you have !

Now you have !
fraimondo (Migrated from github.com) reviewed 2024-01-31 10:46:00 +00:00
junifer/data/template_spaces.py
@ -0,0 +85,4 @@
)
else:
with open(xfm_file_path, "ab") as f:
for chunk in resp.iter_bytes():
fraimondo (Migrated from github.com) commented 2024-01-31 10:46:00 +00:00
Copy link

Not from httpx, but mostly from an underlying connection issue if the connection is cut

Not from httpx, but mostly from an underlying connection issue if the connection is cut
fraimondo (Migrated from github.com) reviewed 2024-01-31 10:46:51 +00:00
junifer/data/tests/test_template_spaces.py
@ -0,0 +12,4 @@
@pytest.mark.skipif(
socket.gethostname() != "juseless",
fraimondo (Migrated from github.com) commented 2024-01-31 10:46:51 +00:00
Copy link

damn, we need to create a "slow test" suite that we use it to check only when about to merge or something like that. For the moment we keep it like this.

damn, we need to create a "slow test" suite that we use it to check only when about to merge or something like that. For the moment we keep it like this.
fraimondo (Migrated from github.com) reviewed 2024-01-31 10:46:59 +00:00
pyproject.toml
@ -47,3 +47,4 @@
"importlib_metadata; python_version<'3.10'",
"h5py>=3.8.0,<3.10",
"httpx[http2]==0.26.0",
]
fraimondo (Migrated from github.com) commented 2024-01-31 10:46:59 +00:00
Copy link

perfect

perfect
synchon (Migrated from github.com) reviewed 2024-01-31 10:47:22 +00:00
junifer/data/template_spaces.py
@ -0,0 +85,4 @@
)
else:
with open(xfm_file_path, "ab") as f:
for chunk in resp.iter_bytes():
synchon (Migrated from github.com) commented 2024-01-31 10:47:22 +00:00
Copy link

You will get error from httpcore, which is the underlying HTTP library, like you get from urllib3 when using requests.

You will get error from `httpcore`, which is the underlying HTTP library, like you get from `urllib3` when using `requests`.
synchon (Migrated from github.com) reviewed 2024-01-31 10:48:49 +00:00
junifer/data/tests/test_template_spaces.py
@ -0,0 +12,4 @@
@pytest.mark.skipif(
socket.gethostname() != "juseless",
synchon (Migrated from github.com) commented 2024-01-31 10:48:49 +00:00
Copy link

Yeah that's a good idea. Well the essential issue is that Gin is not really as powerful as GitHub when you consider infrastructure, so the octet-stream downloads are not cached from what I understand.

Yeah that's a good idea. Well the essential issue is that Gin is not really as powerful as GitHub when you consider infrastructure, so the octet-stream downloads are not cached from what I understand.
synchon (Migrated from github.com) reviewed 2024-01-31 10:52:14 +00:00
junifer/data/template_spaces.py
@ -0,0 +1,91 @@
"""Provide functions for template spaces."""
synchon (Migrated from github.com) commented 2024-01-31 10:52:14 +00:00
Copy link

Transferred.

Transferred.
fraimondo (Migrated from github.com) reviewed 2024-01-31 10:54:32 +00:00
junifer/data/template_spaces.py
@ -0,0 +85,4 @@
)
else:
with open(xfm_file_path, "ab") as f:
for chunk in resp.iter_bytes():
fraimondo (Migrated from github.com) commented 2024-01-31 10:54:32 +00:00
Copy link

Exactly, I thought of catching any exception and giving a more user-friendly error message of what's going

Exactly, I thought of catching any exception and giving a more user-friendly error message of what's going
synchon (Migrated from github.com) reviewed 2024-01-31 10:57:52 +00:00
junifer/data/template_spaces.py
@ -0,0 +85,4 @@
)
else:
with open(xfm_file_path, "ab") as f:
for chunk in resp.iter_bytes():
synchon (Migrated from github.com) commented 2024-01-31 10:57:52 +00:00
Copy link

Ideally it should be dealt by the underlying http lib. You won't get an exception to catch when doing iter_bytes(), having an exception block there would not serve the purpose imo.

Ideally it should be dealt by the underlying http lib. You won't get an exception to catch when doing `iter_bytes()`, having an exception block there would not serve the purpose imo.
fraimondo (Migrated from github.com) approved these changes 2024-02-01 08:31:46 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
fraimondo
Labels
Clear labels   
CRITICAL
High Priority Issue

  
Stale

   
WIP
Work in progress

  
bug
Something isn't working

  
concept
Design/Implementation concepts

  
coordinate
Issues or pull requests related to coordinates (ROIs)

  
dataset
Issues or pull requests related to datagrabbers or datasets

  
dependencies
Pull requests that update a dependency file

  
documentation
Improvements or additions to documentation

  
duplicate
This issue or pull request already exists

  
enhancement
New feature or request

  
github_actions
Pull requests that update GitHub Actions code

  
good first issue
Good for newcomers

  
help wanted
Extra attention is needed

  
invalid
This doesn't seem right

  
maintenance
Issues or pull requests related to maintenance

  
maps
Issues or pull requests related to maps (probabilistic atlases)

  
marker
Issues or pull requests related to markers

  
mask
Issues or pull requests related to masks

  
on hold
Issue waiting on other issue(s)

  
parcellation
Issues or pull requests related to parcellations (deterministic atlases)

  
preprocess
Issues or pull requests related to preprocessors

  
question
Further information is requested

  
ready
Pull request is ready for review and merge

  
storage
Issues or pull requests related to marker storage or retrieval

  
template-space
Issues or pull requests related to template spaces

  
triage
New issues waiting to be reviewed

  
wontfix
This will not be worked on

No labels
CRITICAL
Stale
WIP
bug
concept
coordinate
dataset
dependencies
documentation
duplicate
enhancement
github_actions
good first issue
help wanted
invalid
maintenance
maps
marker
mask
on hold
parcellation
preprocess
question
ready
storage
template-space
triage
wontfix
Milestone
Clear milestone
No items
No milestone
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
juaml/junifer!297
No description provided.