WIP: Preprocess #111

Merged
fraimondo merged 20 commits from preprocess into main 2022-11-04 09:51:43 +00:00
fraimondo commented 2022-10-29 17:40:32 +00:00 (Migrated from github.com)
  • description of feature/fix
  • tests added/passed
  • add an entry to the latest changes
* [ ] description of feature/fix * [x] tests added/passed * [ ] add an entry to the [latest changes](../docs/changes/latest.inc)
fraimondo commented 2022-10-29 17:41:13 +00:00 (Migrated from github.com)

@synchon: Code is ready, need to work on the documentation. Want to start reviewing the code? It's a big PR.

@synchon: Code is ready, need to work on the documentation. Want to start reviewing the code? It's a big PR.
github-actions[bot] commented 2022-10-29 17:44:36 +00:00 (Migrated from github.com)
PR Preview Action v1.2.0
🚀 Deployed preview to https://juaml.github.io/junifer/pr-preview/pr-111/
on branch gh-pages at 2022-11-04 09:06 UTC
[PR Preview Action](https://github.com/rossjrw/pr-preview-action) v1.2.0 :---: :rocket: Deployed preview to https://juaml.github.io/junifer/pr-preview/pr-111/ on branch [`gh-pages`](https://github.com/juaml/junifer/tree/gh-pages) at 2022-11-04 09:06 UTC <!-- Sticky Pull Request Commentpr-preview -->
codecov[bot] commented 2022-10-29 19:47:15 +00:00 (Migrated from github.com)

Codecov Report

Merging #111 (31d3402) into main (1da27cd) will increase coverage by 2.06%.
The diff coverage is 96.26%.

Impacted file tree graph

@@            Coverage Diff             @@
##             main     #111      +/-   ##
==========================================
+ Coverage   89.95%   92.02%   +2.06%     
==========================================
  Files          50       54       +4     
  Lines        1951     2081     +130     
  Branches      373      395      +22     
==========================================
+ Hits         1755     1915     +160     
+ Misses        150      132      -18     
+ Partials       46       34      -12     
Flag Coverage Δ
docs 100.00% <ø> (∅)
junifer 92.01% <96.26%> (+2.06%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
junifer/testing/utils.py 66.66% <66.66%> (ø)
junifer/preprocess/base.py 88.37% <88.37%> (ø)
.../preprocess/confounds/fmriprep_confound_remover.py 98.77% <98.77%> (ø)
junifer/api/decorators.py 100.00% <100.00%> (ø)
junifer/preprocess/__init__.py 100.00% <100.00%> (ø)
junifer/preprocess/confounds/__init__.py 100.00% <100.00%> (ø)
junifer/testing/__init__.py 100.00% <100.00%> (ø)
junifer/testing/datagrabbers.py 100.00% <100.00%> (ø)
docs/conf.py 100.00% <0.00%> (ø)
# [Codecov](https://codecov.io/gh/juaml/junifer/pull/111?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml) Report > Merging [#111](https://codecov.io/gh/juaml/junifer/pull/111?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml) (31d3402) into [main](https://codecov.io/gh/juaml/junifer/commit/1da27cd4ace92899983d742387af24a88b64966b?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml) (1da27cd) will **increase** coverage by `2.06%`. > The diff coverage is `96.26%`. [![Impacted file tree graph](https://codecov.io/gh/juaml/junifer/pull/111/graphs/tree.svg?width=650&height=150&src=pr&token=5H21JuZXMw&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml)](https://codecov.io/gh/juaml/junifer/pull/111?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml) ```diff @@ Coverage Diff @@ ## main #111 +/- ## ========================================== + Coverage 89.95% 92.02% +2.06% ========================================== Files 50 54 +4 Lines 1951 2081 +130 Branches 373 395 +22 ========================================== + Hits 1755 1915 +160 + Misses 150 132 -18 + Partials 46 34 -12 ``` | Flag | Coverage Δ | | |---|---|---| | docs | `100.00% <ø> (∅)` | | | junifer | `92.01% <96.26%> (+2.06%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/juaml/junifer/pull/111?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml) | Coverage Δ | | |---|---|---| | [junifer/testing/utils.py](https://codecov.io/gh/juaml/junifer/pull/111/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml#diff-anVuaWZlci90ZXN0aW5nL3V0aWxzLnB5) | `66.66% <66.66%> (ø)` | | | [junifer/preprocess/base.py](https://codecov.io/gh/juaml/junifer/pull/111/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml#diff-anVuaWZlci9wcmVwcm9jZXNzL2Jhc2UucHk=) | `88.37% <88.37%> (ø)` | | | [.../preprocess/confounds/fmriprep\_confound\_remover.py](https://codecov.io/gh/juaml/junifer/pull/111/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml#diff-anVuaWZlci9wcmVwcm9jZXNzL2NvbmZvdW5kcy9mbXJpcHJlcF9jb25mb3VuZF9yZW1vdmVyLnB5) | `98.77% <98.77%> (ø)` | | | [junifer/api/decorators.py](https://codecov.io/gh/juaml/junifer/pull/111/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml#diff-anVuaWZlci9hcGkvZGVjb3JhdG9ycy5weQ==) | `100.00% <100.00%> (ø)` | | | [junifer/preprocess/\_\_init\_\_.py](https://codecov.io/gh/juaml/junifer/pull/111/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml#diff-anVuaWZlci9wcmVwcm9jZXNzL19faW5pdF9fLnB5) | `100.00% <100.00%> (ø)` | | | [junifer/preprocess/confounds/\_\_init\_\_.py](https://codecov.io/gh/juaml/junifer/pull/111/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml#diff-anVuaWZlci9wcmVwcm9jZXNzL2NvbmZvdW5kcy9fX2luaXRfXy5weQ==) | `100.00% <100.00%> (ø)` | | | [junifer/testing/\_\_init\_\_.py](https://codecov.io/gh/juaml/junifer/pull/111/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml#diff-anVuaWZlci90ZXN0aW5nL19faW5pdF9fLnB5) | `100.00% <100.00%> (ø)` | | | [junifer/testing/datagrabbers.py](https://codecov.io/gh/juaml/junifer/pull/111/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml#diff-anVuaWZlci90ZXN0aW5nL2RhdGFncmFiYmVycy5weQ==) | `100.00% <100.00%> (ø)` | | | [docs/conf.py](https://codecov.io/gh/juaml/junifer/pull/111/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml#diff-ZG9jcy9jb25mLnB5) | `100.00% <0.00%> (ø)` | |
synchon commented 2022-10-30 15:05:14 +00:00 (Migrated from github.com)

@synchon: Code is ready, need to work on the documentation. Want to start reviewing the code? It's a big PR.

Yeah sure, I'll start reviewing the PR.

> @synchon: Code is ready, need to work on the documentation. Want to start reviewing the code? It's a big PR. Yeah sure, I'll start reviewing the PR.
synchon (Migrated from github.com) requested changes 2022-11-02 08:46:50 +00:00
@ -5,3 +5,3 @@
# License: AGPL
from .confounds import BaseConfoundRemover
from .confounds import fMRIPrepConfoundRemover
synchon (Migrated from github.com) commented 2022-11-02 06:50:51 +00:00

We should also do: from .base import BasePreprocessor

We should also do: `from .base import BasePreprocessor`
@ -0,0 +1,175 @@
"""Provide abstract base class for preprocessor."""
synchon (Migrated from github.com) commented 2022-11-02 06:52:03 +00:00

For clarity: """Provide abstract base class for preprocessors."""

For clarity: `"""Provide abstract base class for preprocessors."""`
synchon (Migrated from github.com) commented 2022-11-02 06:53:17 +00:00

... (default None).

`... (default None).`
synchon (Migrated from github.com) commented 2022-11-02 06:54:08 +00:00

... list of str, optional

`... list of str, optional`
synchon (Migrated from github.com) commented 2022-11-02 07:04:44 +00:00

... for this preprocessor.

`... for this preprocessor.`
synchon (Migrated from github.com) commented 2022-11-02 07:07:30 +00:00

... only key "preprocess".

`... only key "preprocess".`
synchon (Migrated from github.com) commented 2022-11-02 07:09:44 +00:00

This comment needs to be adjusted based on whether we want to have s_meta["name"] = .... If so, we would also want to have name argument in __init__.

This comment needs to be adjusted based on whether we want to have `s_meta["name"] = ...`. If so, we would also want to have `name` argument in `__init__`.
synchon (Migrated from github.com) commented 2022-11-02 07:17:41 +00:00
input : dict
    A single input from the pipeline data object on which
    to preprocess.
``` input : dict A single input from the pipeline data object on which to preprocess. ```
synchon (Migrated from github.com) commented 2022-11-02 07:19:50 +00:00
extra_input : dict, optional
    The other fields in the pipeline data object. Useful for accessing
    other data kind that needs to be used during preprocessing.
``` extra_input : dict, optional The other fields in the pipeline data object. Useful for accessing other data kind that needs to be used during preprocessing. ```
synchon (Migrated from github.com) commented 2022-11-02 07:24:21 +00:00

The module docstring needs to be adjusted. Also, I think the module name should be changed for clarity.

The module docstring needs to be adjusted. Also, I think the module name should be changed for clarity.
synchon (Migrated from github.com) commented 2022-11-02 07:32:54 +00:00

"""Class for confound removal from fMRIPrep-ed data."""

`"""Class for confound removal from fMRIPrep-ed data."""`
synchon (Migrated from github.com) commented 2022-11-02 07:33:40 +00:00

Maybe rename it to fMRIPrepConfoundRemover to maintain consistency with the tool's name?

Maybe rename it to `fMRIPrepConfoundRemover` to maintain consistency with the tool's name?
synchon (Migrated from github.com) commented 2022-11-02 07:34:49 +00:00

... bool, optional

`... bool, optional`
synchon (Migrated from github.com) commented 2022-11-02 07:41:37 +00:00

I think it's better to do this at the end of __init__ as the other checks for this class can happen before the checks of the base class.

I think it's better to do this at the end of `__init__` as the other checks for this class can happen before the checks of the base class.
synchon (Migrated from github.com) commented 2022-11-02 08:20:41 +00:00

Is there a way we don't have to override the method?

Is there a way we don't have to override the method?
synchon (Migrated from github.com) commented 2022-11-02 08:22:36 +00:00

... Any]) -> None:

`... Any]) -> None:`
synchon (Migrated from github.com) commented 2022-11-02 08:25:12 +00:00

) -> Tuple[List[str], Dict[str, str], Dict[str, str], str]:

`) -> Tuple[List[str], Dict[str, str], Dict[str, str], str]:`
synchon (Migrated from github.com) commented 2022-11-02 08:25:44 +00:00

squares_to_compute : dict

`squares_to_compute : dict`
synchon (Migrated from github.com) commented 2022-11-02 08:26:04 +00:00

derivatives_to_compute : dict

`derivatives_to_compute : dict`
synchon (Migrated from github.com) commented 2022-11-02 08:28:21 +00:00

Docstring needs Parameters and Returns sections.

Docstring needs `Parameters` and `Returns` sections.
synchon (Migrated from github.com) commented 2022-11-02 08:28:59 +00:00

Docstring needs Parameters section.

Docstring needs `Parameters` section.
synchon (Migrated from github.com) commented 2022-11-02 08:31:31 +00:00
input : dict
    A single input from the pipeline data object on which
    to preprocess.
``` input : dict A single input from the pipeline data object on which to preprocess. ```
synchon (Migrated from github.com) commented 2022-11-02 08:32:03 +00:00
extra_input : dict, optional
    The other fields in the pipeline data object. Useful for accessing
    other data kind that needs to be used during preprocessing.
``` extra_input : dict, optional The other fields in the pipeline data object. Useful for accessing other data kind that needs to be used during preprocessing. ```
synchon (Migrated from github.com) commented 2022-11-02 08:37:30 +00:00

The module docstring and module name need to be adjusted.

The module docstring and module name need to be adjusted.
@ -0,0 +1,65 @@
"""Provide tests for BasePreprocessor."""
synchon (Migrated from github.com) commented 2022-11-02 08:37:53 +00:00

```Provide tests for base preprocessor."""`

```Provide tests for base preprocessor."""`
@ -120,3 +121,93 @@ class SPMAuditoryTestingDatagrabber(BaseDataGrabber):
# Set the element accordingly
out["meta"]["element"] = {"subject": element}
return out
synchon (Migrated from github.com) commented 2022-11-02 08:42:08 +00:00

(default True).

`(default True).`
synchon (Migrated from github.com) commented 2022-11-02 08:42:35 +00:00

(default "both").

`(default "both").`
synchon (Migrated from github.com) commented 2022-11-02 08:43:10 +00:00

age_group : {"adults", "child", "both"}, optional

`age_group : {"adults", "child", "both"}, optional`
synchon (Migrated from github.com) commented 2022-11-02 08:43:32 +00:00

Which age group to fetch

`Which age group to fetch`
synchon (Migrated from github.com) commented 2022-11-02 08:44:12 +00:00

The arguments need type annotations.

The arguments need type annotations.
@ -0,0 +1,29 @@
"""Provide tests for Oasis VBM Testing datagrabber."""
synchon (Migrated from github.com) commented 2022-11-02 08:46:25 +00:00

The docstring does not match the datagrabber being tested.

The docstring does not match the datagrabber being tested.
fraimondo (Migrated from github.com) reviewed 2022-11-02 11:54:58 +00:00
fraimondo (Migrated from github.com) commented 2022-11-02 11:54:57 +00:00

No, this requires "all of" and not "any of" like the super method.

No, this requires "all of" and not "any of" like the super method.
synchon (Migrated from github.com) reviewed 2022-11-02 12:14:57 +00:00
synchon (Migrated from github.com) commented 2022-11-02 12:14:57 +00:00

I see.

I see.
LeSasse (Migrated from github.com) reviewed 2022-11-03 12:52:36 +00:00
LeSasse (Migrated from github.com) commented 2022-11-03 12:39:55 +00:00

above spike_name is a parameter in the function, but here it is defined/overwritten anyhow, or maybe am I missing something?

above spike_name is a parameter in the function, but here it is defined/overwritten anyhow, or maybe am I missing something?
fraimondo (Migrated from github.com) reviewed 2022-11-04 06:57:11 +00:00
fraimondo (Migrated from github.com) commented 2022-11-04 06:57:11 +00:00

it's not a parameter of the function. This function is to give the parameters to the confound prepare function. Basically, needs to find the variables to use, which squares/derivatives to compute and what is the name of the variable to use for searching "spikes".

it's not a parameter of the function. This function is to give the parameters to the confound prepare function. Basically, needs to find the variables to use, which squares/derivatives to compute and what is the name of the variable to use for searching "spikes".
fraimondo (Migrated from github.com) reviewed 2022-11-04 07:03:43 +00:00
fraimondo (Migrated from github.com) left a comment

Done? I don't understand github UI

Done? I don't understand github UI
LeSasse (Migrated from github.com) reviewed 2022-11-04 07:16:50 +00:00
LeSasse commented 2022-11-04 07:26:41 +00:00 (Migrated from github.com)

Done? I don't understand github UI

It looks good to me. I have one more question: Can users also control the other parameters of the nilearn interface (https://nilearn.github.io/stable/modules/generated/nilearn.interfaces.fmriprep.load_confounds.html#nilearn.interfaces.fmriprep.load_confounds), i.e. "compcor", "n_compcor", and "ica_aroma"?

> Done? I don't understand github UI It looks good to me. I have one more question: Can users also control the other parameters of the nilearn interface (https://nilearn.github.io/stable/modules/generated/nilearn.interfaces.fmriprep.load_confounds.html#nilearn.interfaces.fmriprep.load_confounds), i.e. "compcor", "n_compcor", and "ica_aroma"?
fraimondo commented 2022-11-04 07:32:49 +00:00 (Migrated from github.com)

Done? I don't understand github UI

It looks good to me. I have one more question: Can users also control the other parameters of the nilearn interface (https://nilearn.github.io/stable/modules/generated/nilearn.interfaces.fmriprep.load_confounds.html#nilearn.interfaces.fmriprep.load_confounds), i.e. "compcor", "n_compcor", and "ica_aroma"?

Not yet, this can be added later on. The problem with the nilearn interface is that it requires both the CSV and the JSON, so it's not that straightforward to make it work for non-fmriPrep data. We might need to think it through.

> > Done? I don't understand github UI > > It looks good to me. I have one more question: Can users also control the other parameters of the nilearn interface (https://nilearn.github.io/stable/modules/generated/nilearn.interfaces.fmriprep.load_confounds.html#nilearn.interfaces.fmriprep.load_confounds), i.e. "compcor", "n_compcor", and "ica_aroma"? Not yet, this can be added later on. The problem with the nilearn interface is that it requires both the CSV and the JSON, so it's not that straightforward to make it work for non-fmriPrep data. We might need to think it through.
synchon (Migrated from github.com) requested changes 2022-11-04 07:44:12 +00:00
synchon (Migrated from github.com) left a comment
  • I suggest renaming confounds.py to fmriprep_confound_remover.py and test_confounds.py to test_fmriprep_confound_remover.py.
  • A few comments, and should be good to go.
- I suggest renaming `confounds.py` to `fmriprep_confound_remover.py` and `test_confounds.py` to `test_fmriprep_confound_remover.py`. - A few comments, and should be good to go.
@ -16,0 +88,4 @@
.. list-table::
:widths: 10, 30, 5
:header-rows: 1
synchon (Migrated from github.com) commented 2022-11-04 07:23:34 +00:00

I think setting :widths: to auto is cleaner.

I think setting `:widths:` to `auto` is cleaner.
@ -0,0 +1,175 @@
"""Provide abstract base class for preprocessor."""
synchon (Migrated from github.com) commented 2022-11-04 07:30:28 +00:00
            The updated list of available Junifer Data object keys after
            the pipeline step.
``` The updated list of available Junifer Data object keys after the pipeline step. ```
@ -0,0 +24,4 @@
def __init__(
self,
on: Optional[Union[List[str], str]] = None,
synchon (Migrated from github.com) commented 2022-11-04 07:27:13 +00:00

Having a name argument here would allow us to save it to the meta. What do you think?

Having a `name` argument here would allow us to save it to the `meta`. What do you think?
@ -0,0 +108,4 @@
The metadata as a dictionary with the only key 'preprocess'.
"""
s_meta = super().get_meta()
return {"preprocess": s_meta}
synchon (Migrated from github.com) commented 2022-11-04 07:33:16 +00:00

Should we have a name key in s_meta with the preprocess's name (self.name)?

Should we have a `name` key in `s_meta` with the preprocess's name (`self.name`)?
synchon (Migrated from github.com) commented 2022-11-04 07:39:03 +00:00

Being a bit nitpicky here: ... fMRIPrep format.

Being a bit nitpicky here: `... fMRIPrep format.`
@ -0,0 +1,25 @@
"""Testing utils."""
synchon (Migrated from github.com) commented 2022-11-04 07:41:36 +00:00

Missing type annotations.

Missing type annotations.
fraimondo (Migrated from github.com) reviewed 2022-11-04 08:52:34 +00:00
@ -16,0 +88,4 @@
.. list-table::
:widths: 10, 30, 5
:header-rows: 1
fraimondo (Migrated from github.com) commented 2022-11-04 08:52:34 +00:00

:widths: auto will create a table without line-breaks. (custom.css to allow having wide tables)

`:widths: auto` will create a table without line-breaks. (custom.css to allow having wide tables)
fraimondo commented 2022-11-04 08:53:29 +00:00 (Migrated from github.com)
  • I suggest renaming confounds.py to fmriprep_confound_remover.py and test_confounds.py to test_fmriprep_confound_remover.py.
  • A few comments, and should be good to go.

confounds should be a sub-package for all confound removers. So far we only have fMRIPrep. Later on confounds will become a diretctory with all those files in there.

> * I suggest renaming `confounds.py` to `fmriprep_confound_remover.py` and `test_confounds.py` to `test_fmriprep_confound_remover.py`. > * A few comments, and should be good to go. `confounds` should be a sub-package for all confound removers. So far we only have fMRIPrep. Later on `confounds` will become a diretctory with all those files in there.
fraimondo (Migrated from github.com) reviewed 2022-11-04 08:55:50 +00:00
@ -0,0 +24,4 @@
def __init__(
self,
on: Optional[Union[List[str], str]] = None,
fraimondo (Migrated from github.com) commented 2022-11-04 08:55:50 +00:00

Name is only for markers, as there can be many markers on the "pipeline". The "name" in this case is "preprocess"

Name is only for markers, as there can be many markers on the "pipeline". The "name" in this case is "preprocess"
fraimondo (Migrated from github.com) reviewed 2022-11-04 08:56:58 +00:00
@ -0,0 +108,4 @@
The metadata as a dictionary with the only key 'preprocess'.
"""
s_meta = super().get_meta()
return {"preprocess": s_meta}
fraimondo (Migrated from github.com) commented 2022-11-04 08:56:58 +00:00

As explained before, the name is "preprocess".

As explained before, the name is "preprocess".
fraimondo (Migrated from github.com) reviewed 2022-11-04 08:57:03 +00:00
@ -0,0 +1,175 @@
"""Provide abstract base class for preprocessor."""
fraimondo (Migrated from github.com) commented 2022-11-04 08:57:03 +00:00

done

done
fraimondo (Migrated from github.com) reviewed 2022-11-04 08:57:22 +00:00
fraimondo (Migrated from github.com) commented 2022-11-04 08:57:21 +00:00

done

done
fraimondo (Migrated from github.com) reviewed 2022-11-04 09:01:18 +00:00
@ -0,0 +1,25 @@
"""Testing utils."""
fraimondo (Migrated from github.com) commented 2022-11-04 09:01:17 +00:00

done

done
synchon (Migrated from github.com) reviewed 2022-11-04 09:23:54 +00:00
@ -0,0 +24,4 @@
def __init__(
self,
on: Optional[Union[List[str], str]] = None,
synchon (Migrated from github.com) commented 2022-11-04 09:23:54 +00:00

Fair enough.

Fair enough.
synchon (Migrated from github.com) approved these changes 2022-11-04 09:27:04 +00:00
synchon (Migrated from github.com) left a comment

LGTM 🚀

LGTM 🚀
Sign in to join this conversation.
No reviewers
No milestone
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
juaml/junifer!111
No description provided.