juaml/junifer
Watch
3
0
Fork
You've already forked junifer
 0
Code Issues 14 Pull requests 2 Packages Activity Actions

[ENH]: Improving DataGrabber patterns #308

Merged
synchon merged 27 commits from refactor/dg-patterns into main 2024-04-04 13:38:56 +00:00
Conversation 4 Commits 27 Files changed 26 +1357 -739
26 changed files with 1357 additions and 739 deletions
Download patch file Download diff file Expand all files Collapse all files

1
docs/changes/newsfragments/308.enh Normal file
View file

@ -0,0 +1 @@
Improve :class:`.PatternDataGrabber` and :class:`.PatternDataladDataGrabber`'s ``patterns`` to enable ``space``, ``format``, ``mask_item`` and other metadata description handling via YAML by `Synchon Mandal`_

105
docs/extending/datagrabber.rst
View file

@ -61,7 +61,7 @@ Now that we have our element defined, we need to think about the structure of
the dataset. Mainly, because the structure of the dataset will determine how the dataset. Mainly, because the structure of the dataset will determine how
the DataGrabber needs to be implemented. the DataGrabber needs to be implemented.
``junifer`` provides an abstract class to deal with datasets that can be thought ``junifer`` provides a concrete class to deal with datasets that can be thought
in terms of *patterns*. A *pattern* is a string that contains placeholders that in terms of *patterns*. A *pattern* is a string that contains placeholders that
are replaced by the actual values of the element. In our BIDS example, the path are replaced by the actual values of the element. In our BIDS example, the path
to the T1w image of subject ``sub-01`` and session ``ses-01``, relative to the to the T1w image of subject ``sub-01`` and session ``ses-01``, relative to the
@ -98,13 +98,14 @@ Step 3: Create a Data Grabber
Option A: Extending from PatternDataGrabber Option A: Extending from PatternDataGrabber
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The :class:`.PatternDataGrabber` class is an abstract class that has the The :class:`.PatternDataGrabber` class is a concrete class that has the
functionality of understanding patterns embedded in it. functionality of understanding patterns embedded in it.
Before creating the DataGrabber, we need to define 3 variables: Before creating the DataGrabber, we need to define 3 variables:
* ``types``: A list with the available :ref:`data_types` in our dataset. * ``types``: A list with the available :ref:`data_types` in our dataset.
* ``patterns``: A dictionary that specifies the pattern for each data type. * ``patterns``: A dictionary that specifies the pattern and some additional
information for each data type.
* ``replacements``: A list indicating which of the elements in the patterns * ``replacements``: A list indicating which of the elements in the patterns
should be replaced by the values of the element. should be replaced by the values of the element.
@ -114,8 +115,14 @@ For example, in our BIDS example, the variables will be:
types = ["T1w", "BOLD"] types = ["T1w", "BOLD"]
patterns = { patterns = {
"T1w": "{subject}/{session}/anat/{subject}_{session}_T1w.nii.gz", "T1w": {
"BOLD": "{subject}/{session}/func/{subject}_{session}_task-rest_bold.nii.gz", "pattern": "{subject}/{session}/anat/{subject}_{session}_T1w.nii.gz",
"space": "native",
},
"BOLD": {
"pattern": "{subject}/{session}/func/{subject}_{session}_task-rest_bold.nii.gz",
"space": "MNI152NLin6Asym",
},
} }
replacements = ["subject", "session"] replacements = ["subject", "session"]
@ -141,8 +148,14 @@ With the variables defined above, we can create our DataGrabber and name it
def __init__(self, datadir: str | Path) -> None: def __init__(self, datadir: str | Path) -> None:
types = ["T1w", "BOLD"] types = ["T1w", "BOLD"]
patterns = { patterns = {
"T1w": "{subject}/{session}/anat/{subject}_{session}_T1w.nii.gz", "T1w": {
"BOLD": "{subject}/{session}/func/{subject}_{session}_task-rest_bold.nii.gz", "pattern": "{subject}/{session}/anat/{subject}_{session}_T1w.nii.gz",
"space": "native",
},
"BOLD": {
"pattern": "{subject}/{session}/func/{subject}_{session}_task-rest_bold.nii.gz",
"space": "MNI152NLin6Asym",
},
} }
replacements = ["subject", "session"] replacements = ["subject", "session"]
super().__init__( super().__init__(
@ -171,8 +184,14 @@ use the :func:`.register_datagrabber` decorator.
def __init__(self, datadir: str | Path) -> None: def __init__(self, datadir: str | Path) -> None:
types = ["T1w", "BOLD"] types = ["T1w", "BOLD"]
patterns = { patterns = {
"T1w": "{subject}/{session}/anat/{subject}_{session}_T1w.nii.gz", "T1w": {
"BOLD": "{subject}/{session}/func/{subject}_{session}_task-rest_bold.nii.gz", "pattern": "{subject}/{session}/anat/{subject}_{session}_T1w.nii.gz",
"space": "native",
},
"BOLD": {
"pattern": "{subject}/{session}/func/{subject}_{session}_task-rest_bold.nii.gz",
"space": "MNI152NLin6Asym",
},
} }
replacements = ["subject", "session"] replacements = ["subject", "session"]
super().__init__( super().__init__(
@ -252,8 +271,14 @@ And we can create our DataGrabber:
def __init__(self) -> None: def __init__(self) -> None:
types = ["T1w", "BOLD"] types = ["T1w", "BOLD"]
patterns = { patterns = {
"T1w": "{subject}/{session}/anat/{subject}_{session}_T1w.nii.gz", "T1w": {
"BOLD": "{subject}/{session}/func/{subject}_{session}_task-rest_bold.nii.gz", "pattern": "{subject}/{session}/anat/{subject}_{session}_T1w.nii.gz",
"space": "native",
},
"BOLD": {
"pattern": "{subject}/{session}/func/{subject}_{session}_task-rest_bold.nii.gz",
"space": "MNI152NLin6Asym",
},
} }
replacements = ["subject", "session"] replacements = ["subject", "session"]
uri = "https://gin.g-node.org/juaml/datalad-example-bids" uri = "https://gin.g-node.org/juaml/datalad-example-bids"
@ -277,13 +302,17 @@ This approach can be used directly from the YAML, like so:
- BOLD - BOLD
- T1w - T1w
patterns: patterns:
BOLD: "{subject}/{session}/func/{subject}_{session}_task-rest_bold.nii.gz" BOLD:
T1w: "{subject}/{session}/anat/{subject}_{session}_T1w.nii.gz" pattern: "{subject}/{session}/func/{subject}_{session}_task-rest_bold.nii.gz"
space: MNI152NLin6Asym
T1w:
pattern: "{subject}/{session}/anat/{subject}_{session}_T1w.nii.gz"
space: native
replacements: replacements:
- subject - subject
- session - session
uri: "https://gin.g-node.org/juaml/datalad-example-bids" uri: "https://gin.g-node.org/juaml/datalad-example-bids"
rootdir: "example_bids_ses" rootdir: example_bids_ses
.. _extending_datagrabbers_base: .. _extending_datagrabbers_base:
@ -314,10 +343,16 @@ and ``session``, we will use them as parameters of ``get_item``:
.. code-block:: python .. code-block:: python
def get_item(self, subject: str, session: str) -> dict[str, str]: def get_item(self, subject: str, session: str) -> dict[str, dict[str, str]]:
out = { out = {
"T1w": f"{subject}/{session}/anat/{subject}_{session}_T1w.nii.gz", "T1w": {
"BOLD": f"{subject}/{session}/func/{subject}_{session}_task-rest_bold.nii.gz", "path": f"{subject}/{session}/anat/{subject}_{session}_T1w.nii.gz",
"space": "native",
},
"BOLD": {
"path": f"{subject}/{session}/func/{subject}_{session}_task-rest_bold.nii.gz",
"space": "MNI152NLin6Asym",
},
} }
return out return out
@ -367,12 +402,18 @@ So, to summarise, our DataGrabber will look like this:
@register_datagrabber @register_datagrabber
class ExampleBIDSDataGrabber(BaseDataGrabber): class ExampleBIDSDataGrabber(BaseDataGrabber):
def get_item(self, subject: str, session: str) -> dict[str, str]: def get_item(self, subject: str, session: str) -> dict[str, dict[str, str]]:
out = { out = {
"T1w": f"{subject}/{session}/anat/{subject}_{session}_T1w.nii.gz", "T1w": {
"BOLD": f"{subject}/{session}/func/{subject}_{session}_task-rest_bold.nii.gz", "path": f"{subject}/{session}/anat/{subject}_{session}_T1w.nii.gz",
"space": "native",
},
"BOLD": {
"path": f"{subject}/{session}/func/{subject}_{session}_task-rest_bold.nii.gz",
"space": "MNI152NLin6Asym",
},
} }
return out return out
def get_elements(self) -> list[str]: def get_elements(self) -> list[str]:
subjects = ["sub-01", "sub-02", "sub-03"] subjects = ["sub-01", "sub-02", "sub-03"]
@ -438,16 +479,20 @@ this:
self, subject: str, session: str self, subject: str, session: str
) -> dict: ) -> dict:
out = { out = {
"BOLD": f"{subject}/{session}/func/{subject}_{session}_task-rest_bold.nii.gz", "BOLD": {
"path": f"{subject}/{session}/func/{subject}_{session}_task-rest_bold.nii.gz",
"space": "MNI152NLin6Asym",
},
"BOLD_confounds": { "BOLD_confounds": {
"path": f"{subject}/{session}/func/{subject}_{session}_confounds.tsv", "path": f"{subject}/{session}/func/{subject}_{session}_confounds.tsv",
"format": "adhoc", "format": "adhoc",
"mappings": { "mappings": {
"fmriprep": { "fmriprep": {
"variable1": "rot_x", "variable1": "rot_x",
"variable2": "rot_z", "variable2": "rot_z",
"variable3": "rot_y", "variable3": "rot_y",
} },
},
}, },
} }

3
docs/understanding/data.rst
View file

@ -103,6 +103,9 @@ Data Types
* - ``T1w`` * - ``T1w``
- T1w image (3D) - T1w image (3D)
- Preprocessed or Raw T1w image - Preprocessed or Raw T1w image
* - ``T2w``
- T2w image (3D)
- Preprocessed or Raw T2w image
* - ``BOLD`` * - ``BOLD``
- BOLD image (4D) - BOLD image (4D)
- Preprocessed or Denoised BOLD image (fMRIPrep output) - Preprocessed or Denoised BOLD image (fMRIPrep output)

10
examples/run_datagrabber_bids_datalad.py
View file

@ -25,8 +25,14 @@ configure_logging(level="INFO")
# replaced in the patterns. # replaced in the patterns.
types = ["T1w", "BOLD"] types = ["T1w", "BOLD"]
patterns = { patterns = {
"T1w": "{subject}/anat/{subject}_T1w.nii.gz", "T1w": {
"BOLD": "{subject}/func/{subject}_task-rest_bold.nii.gz", "pattern": "{subject}/anat/{subject}_T1w.nii.gz",
"space": "native",
},
"BOLD": {
"pattern": "{subject}/func/{subject}_task-rest_bold.nii.gz",
"space": "MNI152NLin6Asym",
},
} }
replacements = ["subject"] replacements = ["subject"]
############################################################################### ###############################################################################

7
junifer/configs/juseless/datagrabbers/aomic_id1000_vbm.py
View file

@ -31,7 +31,12 @@ class JuselessDataladAOMICID1000VBM(PatternDataladDataGrabber):
types = ["VBM_GM"] types = ["VBM_GM"]
replacements = ["subject"] replacements = ["subject"]
patterns = { patterns = {
"VBM_GM": "sub-{subject}/mri/mwp1sub-{subject}_run-2_T1w.nii.gz", "VBM_GM": {
"pattern": (
"sub-{subject}/mri/mwp1sub-{subject}_run-2_T1w.nii.gz"
),
"space": "IXI549Space",
},
} }
super().__init__( super().__init__(
types=types, types=types,

7
junifer/configs/juseless/datagrabbers/camcan_vbm.py
View file

@ -34,7 +34,12 @@ class JuselessDataladCamCANVBM(PatternDataladDataGrabber):
) )
types = ["VBM_GM"] types = ["VBM_GM"]
replacements = ["subject"] replacements = ["subject"]
patterns = {"VBM_GM": "sub-{subject}/mri/m0wp1sub-{subject}.nii.gz"} patterns = {
"VBM_GM": {
"pattern": "sub-{subject}/mri/m0wp1sub-{subject}.nii.gz",
"space": "IXI549Space",
},
}
super().__init__( super().__init__(
types=types, types=types,
datadir=datadir, datadir=datadir,

7
junifer/configs/juseless/datagrabbers/ixi_vbm.py
View file

@ -43,7 +43,12 @@ class JuselessDataladIXIVBM(PatternDataladDataGrabber):
types = ["VBM_GM"] types = ["VBM_GM"]
replacements = ["site", "subject"] replacements = ["site", "subject"]
patterns = { patterns = {
"VBM_GM": "{site}/sub-{subject}/mri/m0wp1sub-{subject}.nii.gz" "VBM_GM": {
"pattern": (
"{site}/sub-{subject}/mri/m0wp1sub-{subject}.nii.gz"
),
"space": "IXI549Space",
},
} }
# validate and/or transform 'site' input # validate and/or transform 'site' input

66
junifer/configs/juseless/datagrabbers/ucla.py
View file

@ -70,30 +70,48 @@ class JuselessUCLA(PatternDataGrabber):
self.tasks = tasks self.tasks = tasks
# The patterns # The patterns
patterns = { patterns = {
"BOLD": ( "BOLD": {
"sub-{subject}/func/sub-{subject}_task-{task}_bold_space-" "pattern": (
"MNI152NLin2009cAsym_preproc.nii.gz" "sub-{subject}/func/sub-{subject}_task-{task}_bold_space-"
), "MNI152NLin2009cAsym_preproc.nii.gz"
"BOLD_confounds": ( ),
"sub-{subject}/func/sub-{subject}_" "space": "MNI152NLin2009cAsym",
"task-{task}_bold_confounds.tsv" },
), "BOLD_confounds": {
"T1w": ( "pattern": (
"sub-{subject}/anat/sub-{subject}_" "sub-{subject}/func/sub-{subject}_"
"T1w_space-MNI152NLin2009cAsym_preproc.nii.gz" "task-{task}_bold_confounds.tsv"
), ),
"probseg_CSF": ( "space": "fmriprep",
"sub-{subject}/anat/sub-{subject}_T1w_space-" },
"MNI152NLin2009cAsym_class-CSF_probtissue.nii.gz" "T1w": {
), "pattern": (
"probseg_GM": ( "sub-{subject}/anat/sub-{subject}_"
"sub-{subject}/anat/sub-{subject}_T1w_space-" "T1w_space-MNI152NLin2009cAsym_preproc.nii.gz"
"MNI152NLin2009cAsym_class-GM_probtissue.nii.gz" ),
), "space": "MNI152NLin2009cAsym",
"probseg_WM": ( },
"sub-{subject}/anat/sub-{subject}_T1w_space" "probseg_CSF": {
"-MNI152NLin2009cAsym_class-WM_probtissue.nii.gz" "pattern": (
), "sub-{subject}/anat/sub-{subject}_T1w_space-"
"MNI152NLin2009cAsym_class-CSF_probtissue.nii.gz"
),
"space": "MNI152NLin2009cAsym",
},
"probseg_GM": {
"pattern": (
"sub-{subject}/anat/sub-{subject}_T1w_space-"
"MNI152NLin2009cAsym_class-GM_probtissue.nii.gz"
),
"space": "MNI152NLin2009cAsym",
},
"probseg_WM": {
"pattern": (
"sub-{subject}/anat/sub-{subject}_T1w_space"
"-MNI152NLin2009cAsym_class-WM_probtissue.nii.gz"
),
"space": "MNI152NLin2009cAsym",
},
} }
# Set default types # Set default types
if types is None: if types is None:

7
junifer/configs/juseless/datagrabbers/ukb_vbm.py
View file

@ -32,7 +32,12 @@ class JuselessDataladUKBVBM(PatternDataladDataGrabber):
rootdir = "m0wp1" rootdir = "m0wp1"
types = ["VBM_GM"] types = ["VBM_GM"]
replacements = ["subject", "session"] replacements = ["subject", "session"]
patterns = {"VBM_GM": "m0wp1sub-{subject}_ses-{session}_T1w.nii.gz"} patterns = {
"VBM_GM": {
"pattern": "m0wp1sub-{subject}_ses-{session}_T1w.nii.gz",
"space": "IXI549Space",
},
}
super().__init__( super().__init__(
types=types, types=types,
datadir=datadir, datadir=datadir,

189
junifer/datagrabber/aomic/id1000.py
View file

@ -8,7 +8,7 @@
# License: AGPL # License: AGPL
from pathlib import Path from pathlib import Path
from typing import Dict, List, Union from typing import List, Union
from ...api.decorators import register_datagrabber from ...api.decorators import register_datagrabber
from ..pattern_datalad import PatternDataladDataGrabber from ..pattern_datalad import PatternDataladDataGrabber
@ -41,51 +41,79 @@ class DataladAOMICID1000(PatternDataladDataGrabber):
) -> None: ) -> None:
# The patterns # The patterns
patterns = { patterns = {
"BOLD": ( "BOLD": {
"derivatives/fmriprep/sub-{subject}/func/" "pattern": (
"sub-{subject}_task-moviewatching_" "derivatives/fmriprep/sub-{subject}/func/"
"space-MNI152NLin2009cAsym_desc-preproc_bold.nii.gz" "sub-{subject}_task-moviewatching_"
), "space-MNI152NLin2009cAsym_desc-preproc_bold.nii.gz"
"BOLD_confounds": ( ),
"derivatives/fmriprep/sub-{subject}/func/" "space": "MNI152NLin2009cAsym",
"sub-{subject}_task-moviewatching_" "mask_item": "BOLD_mask",
"desc-confounds_regressors.tsv" },
), "BOLD_confounds": {
"BOLD_mask": ( "pattern": (
"derivatives/fmriprep/sub-{subject}/func/" "derivatives/fmriprep/sub-{subject}/func/"
"sub-{subject}_task-moviewatching_" "sub-{subject}_task-moviewatching_"
"space-MNI152NLin2009cAsym_" "desc-confounds_regressors.tsv"
"desc-brain_mask.nii.gz" ),
), "format": "fmriprep",
"T1w": ( },
"derivatives/fmriprep/sub-{subject}/anat/" "BOLD_mask": {
"sub-{subject}_space-MNI152NLin2009cAsym_" "pattern": (
"desc-preproc_T1w.nii.gz" "derivatives/fmriprep/sub-{subject}/func/"
), "sub-{subject}_task-moviewatching_"
"T1w_mask": ( "space-MNI152NLin2009cAsym_"
"derivatives/fmriprep/sub-{subject}/anat/" "desc-brain_mask.nii.gz"
"sub-{subject}_space-MNI152NLin2009cAsym_" ),
"desc-brain_mask.nii.gz" "space": "MNI152NLin2009cAsym",
), },
"probseg_CSF": ( "T1w": {
"derivatives/fmriprep/sub-{subject}/anat/" "pattern": (
"sub-{subject}_space-MNI152NLin2009cAsym_label-" "derivatives/fmriprep/sub-{subject}/anat/"
"CSF_probseg.nii.gz" "sub-{subject}_space-MNI152NLin2009cAsym_"
), "desc-preproc_T1w.nii.gz"
"probseg_GM": ( ),
"derivatives/fmriprep/sub-{subject}/anat/" "space": "MNI152NLin2009cAsym",
"sub-{subject}_space-MNI152NLin2009cAsym_label-" "mask_item": "T1w_mask",
"GM_probseg.nii.gz" },
), "T1w_mask": {
"probseg_WM": ( "pattern": (
"derivatives/fmriprep/sub-{subject}/anat/" "derivatives/fmriprep/sub-{subject}/anat/"
"sub-{subject}_space-MNI152NLin2009cAsym_label-" "sub-{subject}_space-MNI152NLin2009cAsym_"
"WM_probseg.nii.gz" "desc-brain_mask.nii.gz"
), ),
"DWI": ( "space": "MNI152NLin2009cAsym",
"derivatives/dwipreproc/sub-{subject}/dwi/" },
"sub-{subject}_desc-preproc_dwi.nii.gz" "probseg_CSF": {
), "pattern": (
"derivatives/fmriprep/sub-{subject}/anat/"
"sub-{subject}_space-MNI152NLin2009cAsym_label-"
"CSF_probseg.nii.gz"
),
"space": "MNI152NLin2009cAsym",
},
"probseg_GM": {
"pattern": (
"derivatives/fmriprep/sub-{subject}/anat/"
"sub-{subject}_space-MNI152NLin2009cAsym_label-"
"GM_probseg.nii.gz"
),
"space": "MNI152NLin2009cAsym",
},
"probseg_WM": {
"pattern": (
"derivatives/fmriprep/sub-{subject}/anat/"
"sub-{subject}_space-MNI152NLin2009cAsym_label-"
"WM_probseg.nii.gz"
),
"space": "MNI152NLin2009cAsym",
},
"DWI": {
"pattern": (
"derivatives/dwipreproc/sub-{subject}/dwi/"
"sub-{subject}_desc-preproc_dwi.nii.gz"
),
},
} }
# Use native T1w assets # Use native T1w assets
self.native_t1w = False self.native_t1w = False
@ -93,19 +121,30 @@ class DataladAOMICID1000(PatternDataladDataGrabber):
self.native_t1w = True self.native_t1w = True
patterns.update( patterns.update(
{ {
"T1w": ( "T1w": {
"derivatives/fmriprep/sub-{subject}/anat/" "pattern": (
"sub-{subject}_desc-preproc_T1w.nii.gz" "derivatives/fmriprep/sub-{subject}/anat/"
), "sub-{subject}_desc-preproc_T1w.nii.gz"
"T1w_mask": ( ),
"derivatives/fmriprep/sub-{subject}/anat/" "space": "native",
"sub-{subject}_desc-brain_mask.nii.gz" "mask_item": "T1w_mask",
), },
"Warp": ( "T1w_mask": {
"derivatives/fmriprep/sub-{subject}/anat/" "pattern": (
"sub-{subject}_from-MNI152NLin2009cAsym_to-T1w_" "derivatives/fmriprep/sub-{subject}/anat/"
"mode-image_xfm.h5" "sub-{subject}_desc-brain_mask.nii.gz"
), ),
"space": "native",
},
"Warp": {
"pattern": (
"derivatives/fmriprep/sub-{subject}/anat/"
"sub-{subject}_from-MNI152NLin2009cAsym_to-T1w_"
"mode-image_xfm.h5"
),
"src": "MNI152NLin2009cAsym",
"dst": "native",
},
} }
) )
# Set default types # Set default types
@ -126,35 +165,3 @@ class DataladAOMICID1000(PatternDataladDataGrabber):
replacements=replacements, replacements=replacements,
confounds_format="fmriprep", confounds_format="fmriprep",
) )
def get_item(self, subject: str) -> Dict:
"""Index one element in the dataset.
Parameters
----------
subject : str
The subject ID.
Returns
-------
out : dict
Dictionary of paths for each type of data required for the
specified element.
"""
out = super().get_item(subject=subject)
if out.get("BOLD"):
out["BOLD"]["mask_item"] = "BOLD_mask"
# Add space information
out["BOLD"].update({"space": "MNI152NLin2009cAsym"})
if out.get("T1w"):
out["T1w"]["mask_item"] = "T1w_mask"
# Add space information
if self.native_t1w:
out["T1w"].update({"space": "native"})
else:
out["T1w"].update({"space": "MNI152NLin2009cAsym"})
if out.get("Warp"):
# Add source space information
out["Warp"].update({"src": "MNI152NLin2009cAsym"})
return out

170
junifer/datagrabber/aomic/piop1.py
View file

@ -77,50 +77,78 @@ class DataladAOMICPIOP1(PatternDataladDataGrabber):
self.tasks = tasks self.tasks = tasks
# The patterns # The patterns
patterns = { patterns = {
"BOLD": ( "BOLD": {
"derivatives/fmriprep/sub-{subject}/func/" "pattern": (
"sub-{subject}_task-{task}_" "derivatives/fmriprep/sub-{subject}/func/"
"space-MNI152NLin2009cAsym_desc-preproc_bold.nii.gz" "sub-{subject}_task-{task}_"
), "space-MNI152NLin2009cAsym_desc-preproc_bold.nii.gz"
"BOLD_confounds": ( ),
"derivatives/fmriprep/sub-{subject}/func/" "space": "MNI152NLin2009cAsym",
"sub-{subject}_task-{task}_" "mask_item": "BOLD_mask",
"desc-confounds_regressors.tsv" },
), "BOLD_confounds": {
"BOLD_mask": ( "pattern": (
"derivatives/fmriprep/sub-{subject}/func/" "derivatives/fmriprep/sub-{subject}/func/"
"sub-{subject}_task-{task}_" "sub-{subject}_task-{task}_"
"space-MNI152NLin2009cAsym_desc-brain_mask.nii.gz" "desc-confounds_regressors.tsv"
), ),
"T1w": ( "format": "fmriprep",
"derivatives/fmriprep/sub-{subject}/anat/" },
"sub-{subject}_space-MNI152NLin2009cAsym_" "BOLD_mask": {
"desc-preproc_T1w.nii.gz" "pattern": (
), "derivatives/fmriprep/sub-{subject}/func/"
"T1w_mask": ( "sub-{subject}_task-{task}_"
"derivatives/fmriprep/sub-{subject}/anat/" "space-MNI152NLin2009cAsym_desc-brain_mask.nii.gz"
"sub-{subject}_space-MNI152NLin2009cAsym_" ),
"desc-brain_mask.nii.gz" "space": "MNI152NLin2009cAsym",
), },
"probseg_CSF": ( "T1w": {
"derivatives/fmriprep/sub-{subject}/anat/" "pattern": (
"sub-{subject}_space-MNI152NLin2009cAsym_label-" "derivatives/fmriprep/sub-{subject}/anat/"
"CSF_probseg.nii.gz" "sub-{subject}_space-MNI152NLin2009cAsym_"
), "desc-preproc_T1w.nii.gz"
"probseg_GM": ( ),
"derivatives/fmriprep/sub-{subject}/anat/" "space": "MNI152NLin2009cAsym",
"sub-{subject}_space-MNI152NLin2009cAsym_label-" "mask_item": "T1w_mask",
"GM_probseg.nii.gz" },
), "T1w_mask": {
"probseg_WM": ( "pattern": (
"derivatives/fmriprep/sub-{subject}/anat/" "derivatives/fmriprep/sub-{subject}/anat/"
"sub-{subject}_space-MNI152NLin2009cAsym_label-" "sub-{subject}_space-MNI152NLin2009cAsym_"
"WM_probseg.nii.gz" "desc-brain_mask.nii.gz"
), ),
"DWI": ( "space": "MNI152NLin2009cAsym",
"derivatives/dwipreproc/sub-{subject}/dwi/" },
"sub-{subject}_desc-preproc_dwi.nii.gz" "probseg_CSF": {
), "pattern": (
"derivatives/fmriprep/sub-{subject}/anat/"
"sub-{subject}_space-MNI152NLin2009cAsym_label-"
"CSF_probseg.nii.gz"
),
"space": "MNI152NLin2009cAsym",
},
"probseg_GM": {
"pattern": (
"derivatives/fmriprep/sub-{subject}/anat/"
"sub-{subject}_space-MNI152NLin2009cAsym_label-"
"GM_probseg.nii.gz"
),
"space": "MNI152NLin2009cAsym",
},
"probseg_WM": {
"pattern": (
"derivatives/fmriprep/sub-{subject}/anat/"
"sub-{subject}_space-MNI152NLin2009cAsym_label-"
"WM_probseg.nii.gz"
),
"space": "MNI152NLin2009cAsym",
},
"DWI": {
"pattern": (
"derivatives/dwipreproc/sub-{subject}/dwi/"
"sub-{subject}_desc-preproc_dwi.nii.gz"
),
},
} }
# Use native T1w assets # Use native T1w assets
self.native_t1w = False self.native_t1w = False
@ -128,19 +156,30 @@ class DataladAOMICPIOP1(PatternDataladDataGrabber):
self.native_t1w = True self.native_t1w = True
patterns.update( patterns.update(
{ {
"T1w": ( "T1w": {
"derivatives/fmriprep/sub-{subject}/anat/" "pattern": (
"sub-{subject}_desc-preproc_T1w.nii.gz" "derivatives/fmriprep/sub-{subject}/anat/"
), "sub-{subject}_desc-preproc_T1w.nii.gz"
"T1w_mask": ( ),
"derivatives/fmriprep/sub-{subject}/anat/" "space": "native",
"sub-{subject}_desc-brain_mask.nii.gz" "mask_item": "T1w_mask",
), },
"Warp": ( "T1w_mask": {
"derivatives/fmriprep/sub-{subject}/anat/" "pattern": (
"sub-{subject}_from-MNI152NLin2009cAsym_to-T1w_" "derivatives/fmriprep/sub-{subject}/anat/"
"mode-image_xfm.h5" "sub-{subject}_desc-brain_mask.nii.gz"
), ),
"space": "native",
},
"Warp": {
"pattern": (
"derivatives/fmriprep/sub-{subject}/anat/"
"sub-{subject}_from-MNI152NLin2009cAsym_to-T1w_"
"mode-image_xfm.h5"
),
"src": "MNI152NLin2009cAsym",
"dst": "native",
},
} }
) )
# Set default types # Set default types
@ -192,22 +231,7 @@ class DataladAOMICPIOP1(PatternDataladDataGrabber):
acq = task_acqs[task] acq = task_acqs[task]
new_task = f"{task}_acq-{acq}" new_task = f"{task}_acq-{acq}"
out = super().get_item(subject=subject, task=new_task) return super().get_item(subject=subject, task=new_task)
if out.get("BOLD"):
out["BOLD"]["mask_item"] = "BOLD_mask"
# Add space information
out["BOLD"].update({"space": "MNI152NLin2009cAsym"})
if out.get("T1w"):
out["T1w"]["mask_item"] = "T1w_mask"
# Add space information
if self.native_t1w:
out["T1w"].update({"space": "native"})
else:
out["T1w"].update({"space": "MNI152NLin2009cAsym"})
if out.get("Warp"):
# Add source space information
out["Warp"].update({"src": "MNI152NLin2009cAsym"})
return out
def get_elements(self) -> List: def get_elements(self) -> List:
"""Implement fetching list of subjects in the dataset. """Implement fetching list of subjects in the dataset.

170
junifer/datagrabber/aomic/piop2.py
View file

@ -74,50 +74,78 @@ class DataladAOMICPIOP2(PatternDataladDataGrabber):
self.tasks = tasks self.tasks = tasks
# The patterns # The patterns
patterns = { patterns = {
"BOLD": ( "BOLD": {
"derivatives/fmriprep/sub-{subject}/func/" "pattern": (
"sub-{subject}_task-{task}_" "derivatives/fmriprep/sub-{subject}/func/"
"space-MNI152NLin2009cAsym_desc-preproc_bold.nii.gz" "sub-{subject}_task-{task}_"
), "space-MNI152NLin2009cAsym_desc-preproc_bold.nii.gz"
"BOLD_confounds": ( ),
"derivatives/fmriprep/sub-{subject}/func/" "space": "MNI152NLin2009cAsym",
"sub-{subject}_task-{task}_" "mask_item": "BOLD_mask",
"desc-confounds_regressors.tsv" },
), "BOLD_confounds": {
"BOLD_mask": ( "pattern": (
"derivatives/fmriprep/sub-{subject}/func/" "derivatives/fmriprep/sub-{subject}/func/"
"sub-{subject}_task-{task}_" "sub-{subject}_task-{task}_"
"space-MNI152NLin2009cAsym_desc-brain_mask.nii.gz" "desc-confounds_regressors.tsv"
), ),
"T1w": ( "format": "fmriprep",
"derivatives/fmriprep/sub-{subject}/anat/" },
"sub-{subject}_space-MNI152NLin2009cAsym_" "BOLD_mask": {
"desc-preproc_T1w.nii.gz" "pattern": (
), "derivatives/fmriprep/sub-{subject}/func/"
"T1w_mask": ( "sub-{subject}_task-{task}_"
"derivatives/fmriprep/sub-{subject}/anat/" "space-MNI152NLin2009cAsym_desc-brain_mask.nii.gz"
"sub-{subject}_space-MNI152NLin2009cAsym_" ),
"desc-brain_mask.nii.gz" "space": "MNI152NLin2009cAsym",
), },
"probseg_CSF": ( "T1w": {
"derivatives/fmriprep/sub-{subject}/anat/" "pattern": (
"sub-{subject}_space-MNI152NLin2009cAsym_label-" "derivatives/fmriprep/sub-{subject}/anat/"
"CSF_probseg.nii.gz" "sub-{subject}_space-MNI152NLin2009cAsym_"
), "desc-preproc_T1w.nii.gz"
"probseg_GM": ( ),
"derivatives/fmriprep/sub-{subject}/anat/" "space": "MNI152NLin2009cAsym",
"sub-{subject}_space-MNI152NLin2009cAsym_label-" "mask_item": "T1w_mask",
"GM_probseg.nii.gz" },
), "T1w_mask": {
"probseg_WM": ( "pattern": (
"derivatives/fmriprep/sub-{subject}/anat/" "derivatives/fmriprep/sub-{subject}/anat/"
"sub-{subject}_space-MNI152NLin2009cAsym_label-" "sub-{subject}_space-MNI152NLin2009cAsym_"
"WM_probseg.nii.gz" "desc-brain_mask.nii.gz"
), ),
"DWI": ( "space": "MNI152NLin2009cAsym",
"derivatives/dwipreproc/sub-{subject}/dwi/" },
"sub-{subject}_desc-preproc_dwi.nii.gz" "probseg_CSF": {
), "pattern": (
"derivatives/fmriprep/sub-{subject}/anat/"
"sub-{subject}_space-MNI152NLin2009cAsym_label-"
"CSF_probseg.nii.gz"
),
"space": "MNI152NLin2009cAsym",
},
"probseg_GM": {
"pattern": (
"derivatives/fmriprep/sub-{subject}/anat/"
"sub-{subject}_space-MNI152NLin2009cAsym_label-"
"GM_probseg.nii.gz"
),
"space": "MNI152NLin2009cAsym",
},
"probseg_WM": {
"pattern": (
"derivatives/fmriprep/sub-{subject}/anat/"
"sub-{subject}_space-MNI152NLin2009cAsym_label-"
"WM_probseg.nii.gz"
),
"space": "MNI152NLin2009cAsym",
},
"DWI": {
"pattern": (
"derivatives/dwipreproc/sub-{subject}/dwi/"
"sub-{subject}_desc-preproc_dwi.nii.gz"
),
},
} }
# Use native T1w assets # Use native T1w assets
self.native_t1w = False self.native_t1w = False
@ -125,19 +153,30 @@ class DataladAOMICPIOP2(PatternDataladDataGrabber):
self.native_t1w = True self.native_t1w = True
patterns.update( patterns.update(
{ {
"T1w": ( "T1w": {
"derivatives/fmriprep/sub-{subject}/anat/" "pattern": (
"sub-{subject}_desc-preproc_T1w.nii.gz" "derivatives/fmriprep/sub-{subject}/anat/"
), "sub-{subject}_desc-preproc_T1w.nii.gz"
"T1w_mask": ( ),
"derivatives/fmriprep/sub-{subject}/anat/" "space": "native",
"sub-{subject}_desc-brain_mask.nii.gz" "mask_item": "T1w_mask",
), },
"Warp": ( "T1w_mask": {
"derivatives/fmriprep/sub-{subject}/anat/" "pattern": (
"sub-{subject}_from-MNI152NLin2009cAsym_to-T1w_" "derivatives/fmriprep/sub-{subject}/anat/"
"mode-image_xfm.h5" "sub-{subject}_desc-brain_mask.nii.gz"
), ),
"space": "native",
},
"Warp": {
"pattern": (
"derivatives/fmriprep/sub-{subject}/anat/"
"sub-{subject}_from-MNI152NLin2009cAsym_to-T1w_"
"mode-image_xfm.h5"
),
"src": "MNI152NLin2009cAsym",
"dst": "native",
},
} }
) )
# Set default types # Set default types
@ -192,19 +231,4 @@ class DataladAOMICPIOP2(PatternDataladDataGrabber):
specified element. specified element.
""" """
out = super().get_item(subject=subject, task=f"{task}_acq-seq") return super().get_item(subject=subject, task=f"{task}_acq-seq")
if out.get("BOLD"):
out["BOLD"]["mask_item"] = "BOLD_mask"
# Add space information
out["BOLD"].update({"space": "MNI152NLin2009cAsym"})
if out.get("T1w"):
out["T1w"]["mask_item"] = "T1w_mask"
# Add space information
if self.native_t1w:
out["T1w"].update({"space": "native"})
else:
out["T1w"].update({"space": "MNI152NLin2009cAsym"})
if out.get("Warp"):
# Add source space information
out["Warp"].update({"src": "MNI152NLin2009cAsym"})
return out

10
junifer/datagrabber/base.py
View file

@ -59,7 +59,9 @@ class BaseDataGrabber(ABC, UpdateMetaMixin):
""" """
yield from self.get_elements() yield from self.get_elements()
def __getitem__(self, element: Union[str, Tuple[str]]) -> Dict[str, Dict]: def __getitem__(
self, element: Union[str, Tuple[str, ...]]
) -> Dict[str, Dict]:
"""Enable indexing support. """Enable indexing support.
Parameters Parameters
@ -183,7 +185,7 @@ class BaseDataGrabber(ABC, UpdateMetaMixin):
raise_error( raise_error(
msg="Concrete classes need to implement get_element_keys().", msg="Concrete classes need to implement get_element_keys().",
klass=NotImplementedError, klass=NotImplementedError,
) ) # pragma: no cover
@abstractmethod @abstractmethod
def get_elements(self) -> List[Union[str, Tuple[str]]]: def get_elements(self) -> List[Union[str, Tuple[str]]]:
@ -200,7 +202,7 @@ class BaseDataGrabber(ABC, UpdateMetaMixin):
raise_error( raise_error(
msg="Concrete classes need to implement get_elements().", msg="Concrete classes need to implement get_elements().",
klass=NotImplementedError, klass=NotImplementedError,
) ) # pragma: no cover
@abstractmethod @abstractmethod
def get_item(self, **element: Dict) -> Dict[str, Dict]: def get_item(self, **element: Dict) -> Dict[str, Dict]:
@ -221,4 +223,4 @@ class BaseDataGrabber(ABC, UpdateMetaMixin):
raise_error( raise_error(
msg="Concrete classes need to implement get_item().", msg="Concrete classes need to implement get_item().",
klass=NotImplementedError, klass=NotImplementedError,
) ) # pragma: no cover

2
junifer/datagrabber/datalad_base.py
View file

@ -17,12 +17,10 @@ import datalad.api as dl
from datalad.support.exceptions import IncompleteResultsError from datalad.support.exceptions import IncompleteResultsError
from datalad.support.gitrepo import GitRepo from datalad.support.gitrepo import GitRepo
from ..api.decorators import register_datagrabber
from ..utils import logger, raise_error, warn_with_log from ..utils import logger, raise_error, warn_with_log
from .base import BaseDataGrabber from .base import BaseDataGrabber
@register_datagrabber
class DataladDataGrabber(BaseDataGrabber): class DataladDataGrabber(BaseDataGrabber):
"""Abstract base class for datalad-based data fetching. """Abstract base class for datalad-based data fetching.

137
junifer/datagrabber/dmcc13_benchmark.py
View file

@ -139,43 +139,69 @@ class DMCC13Benchmark(PatternDataladDataGrabber):
self.runs = runs self.runs = runs
# The patterns # The patterns
patterns = { patterns = {
"BOLD": ( "BOLD": {
"derivatives/fmriprep-1.3.2/sub-{subject}/ses-{session}/" "pattern": (
"func/sub-{subject}_ses-{session}_task-{task}_acq-mb4" "derivatives/fmriprep-1.3.2/sub-{subject}/ses-{session}/"
"{phase_encoding}_run-{run}_" "func/sub-{subject}_ses-{session}_task-{task}_acq-mb4"
"space-MNI152NLin2009cAsym_desc-preproc_bold.nii.gz" "{phase_encoding}_run-{run}_"
), "space-MNI152NLin2009cAsym_desc-preproc_bold.nii.gz"
"BOLD_confounds": ( ),
"derivatives/fmriprep-1.3.2/sub-{subject}/ses-{session}/" "space": "MNI152NLin2009cAsym",
"func/sub-{subject}_ses-{session}_task-{task}_acq-mb4" "mask_item": "BOLD_mask",
"{phase_encoding}_run-{run}_desc-confounds_regressors.tsv" },
), "BOLD_confounds": {
"BOLD_mask": ( "pattern": (
"derivatives/fmriprep-1.3.2/sub-{subject}/ses-{session}/" "derivatives/fmriprep-1.3.2/sub-{subject}/ses-{session}/"
"/func/sub-{subject}_ses-{session}_task-{task}_acq-mb4" "func/sub-{subject}_ses-{session}_task-{task}_acq-mb4"
"{phase_encoding}_run-{run}_" "{phase_encoding}_run-{run}_desc-confounds_regressors.tsv"
"space-MNI152NLin2009cAsym_desc-brain_mask.nii.gz" ),
), "format": "fmriprep",
"T1w": ( },
"derivatives/fmriprep-1.3.2/sub-{subject}/anat/" "BOLD_mask": {
"sub-{subject}_space-MNI152NLin2009cAsym_desc-preproc_T1w.nii.gz" "pattern": (
), "derivatives/fmriprep-1.3.2/sub-{subject}/ses-{session}/"
"T1w_mask": ( "/func/sub-{subject}_ses-{session}_task-{task}_acq-mb4"
"derivatives/fmriprep-1.3.2/sub-{subject}/anat/" "{phase_encoding}_run-{run}_"
"sub-{subject}_space-MNI152NLin2009cAsym_desc-brain_mask.nii.gz" "space-MNI152NLin2009cAsym_desc-brain_mask.nii.gz"
), ),
"probseg_CSF": ( "space": "MNI152NLin2009cAsym",
"derivatives/fmriprep-1.3.2/sub-{subject}/anat/" },
"sub-{subject}_space-MNI152NLin2009cAsym_label-CSF_probseg.nii.gz" "T1w": {
), "pattern": (
"probseg_GM": ( "derivatives/fmriprep-1.3.2/sub-{subject}/anat/"
"derivatives/fmriprep-1.3.2/sub-{subject}/anat/" "sub-{subject}_space-MNI152NLin2009cAsym_desc-preproc_T1w.nii.gz"
"sub-{subject}_space-MNI152NLin2009cAsym_label-GM_probseg.nii.gz" ),
), "space": "MNI152NLin2009cAsym",
"probseg_WM": ( "mask_item": "T1w_mask",
"derivatives/fmriprep-1.3.2/sub-{subject}/anat/" },
"sub-{subject}_space-MNI152NLin2009cAsym_label-WM_probseg.nii.gz" "T1w_mask": {
), "pattern": (
"derivatives/fmriprep-1.3.2/sub-{subject}/anat/"
"sub-{subject}_space-MNI152NLin2009cAsym_desc-brain_mask.nii.gz"
),
"space": "MNI152NLin2009cAsym",
},
"probseg_CSF": {
"pattern": (
"derivatives/fmriprep-1.3.2/sub-{subject}/anat/"
"sub-{subject}_space-MNI152NLin2009cAsym_label-CSF_probseg.nii.gz"
),
"space": "MNI152NLin2009cAsym",
},
"probseg_GM": {
"pattern": (
"derivatives/fmriprep-1.3.2/sub-{subject}/anat/"
"sub-{subject}_space-MNI152NLin2009cAsym_label-GM_probseg.nii.gz"
),
"space": "MNI152NLin2009cAsym",
},
"probseg_WM": {
"pattern": (
"derivatives/fmriprep-1.3.2/sub-{subject}/anat/"
"sub-{subject}_space-MNI152NLin2009cAsym_label-WM_probseg.nii.gz"
),
"space": "MNI152NLin2009cAsym",
},
} }
# Use native T1w assets # Use native T1w assets
self.native_t1w = False self.native_t1w = False
@ -183,19 +209,30 @@ class DMCC13Benchmark(PatternDataladDataGrabber):
self.native_t1w = True self.native_t1w = True
patterns.update( patterns.update(
{ {
"T1w": ( "T1w": {
"derivatives/fmriprep-1.3.2/sub-{subject}/anat/" "pattern": (
"sub-{subject}_desc-preproc_T1w.nii.gz" "derivatives/fmriprep-1.3.2/sub-{subject}/anat/"
), "sub-{subject}_desc-preproc_T1w.nii.gz"
"T1w_mask": ( ),
"derivatives/fmriprep-1.3.2/sub-{subject}/anat/" "space": "native",
"sub-{subject}_desc-brain_mask.nii.gz" "mask_item": "T1w_mask",
), },
"Warp": ( "T1w_mask": {
"derivatives/fmriprep-1.3.2/sub-{subject}/anat/" "pattern": (
"sub-{subject}_from-MNI152NLin2009cAsym_to-T1w_" "derivatives/fmriprep-1.3.2/sub-{subject}/anat/"
"mode-image_xfm.h5" "sub-{subject}_desc-brain_mask.nii.gz"
), ),
"space": "native",
},
"Warp": {
"pattern": (
"derivatives/fmriprep-1.3.2/sub-{subject}/anat/"
"sub-{subject}_from-MNI152NLin2009cAsym_to-T1w_"
"mode-image_xfm.h5"
),
"src": "MNI152NLin2009cAsym",
"dst": "native",
},
} }
) )
# Set default types # Set default types

40
junifer/datagrabber/hcp1200/hcp1200.py
View file

@ -106,14 +106,26 @@ class HCP1200(PatternDataGrabber):
types = ["BOLD", "T1w", "Warp"] types = ["BOLD", "T1w", "Warp"]
# The patterns # The patterns
patterns = { patterns = {
"BOLD": ( "BOLD": {
"{subject}/MNINonLinear/Results/" "pattern": (
"{task}_{phase_encoding}/" "{subject}/MNINonLinear/Results/"
"{task}_{phase_encoding}" "{task}_{phase_encoding}/"
f"{suffix}.nii.gz" "{task}_{phase_encoding}"
), f"{suffix}.nii.gz"
"T1w": "{subject}/T1w/T1w_acpc_dc_restore.nii.gz", ),
"Warp": "{subject}/MNINonLinear/xfms/standard2acpc_dc.nii.gz", "space": "MNI152NLin6Asym",
},
"T1w": {
"pattern": "{subject}/T1w/T1w_acpc_dc_restore.nii.gz",
"space": "native",
},
"Warp": {
"pattern": (
"{subject}/MNINonLinear/xfms/standard2acpc_dc.nii.gz"
),
"src": "MNI152NLin6Asym",
"dst": "native",
},
} }
# The replacements # The replacements
replacements = ["subject", "task", "phase_encoding"] replacements = ["subject", "task", "phase_encoding"]
@ -150,19 +162,9 @@ class HCP1200(PatternDataGrabber):
else: else:
new_task = f"tfMRI_{task}" new_task = f"tfMRI_{task}"
out = super().get_item( return super().get_item(
subject=subject, task=new_task, phase_encoding=phase_encoding subject=subject, task=new_task, phase_encoding=phase_encoding
) )
# Add space for BOLD data type
if "BOLD" in out:
out["BOLD"].update({"space": "MNI152NLin6Asym"})
# Add space for T1w data type
if "T1w" in out:
out["T1w"].update({"space": "native"})
# Add source space for Warp data type
if "Warp" in out:
out["Warp"].update({"src": "MNI152NLin6Asym"})
return out
def get_elements(self) -> List: def get_elements(self) -> List:
"""Implement fetching list of elements in the dataset. """Implement fetching list of elements in the dataset.

149
junifer/datagrabber/pattern.py
View file

@ -32,11 +32,98 @@ class PatternDataGrabber(BaseDataGrabber):
types : list of str types : list of str
The types of data to be grabbed. The types of data to be grabbed.
patterns : dict patterns : dict
Patterns for each type of data as a dictionary. The keys are the types Data type patterns as a dictionary. It has the following schema:
and the values are the patterns. Each occurrence of the string
``{subject}`` in the pattern will be replaced by the indexed element. * ``"T1w"`` :
.. code-block:: none
{
"mandatory": ["pattern", "space"],
"optional": []
}
* ``"T2w"`` :
.. code-block:: none
{
"mandatory": ["pattern", "space"],
"optional": []
}
* ``"BOLD"`` :
.. code-block:: none
{
"mandatory": ["pattern", "space"],
"optional": ["mask_item"]
}
* ``"Warp"`` :
.. code-block:: none
{
"mandatory": ["pattern", "src", "dst"],
"optional": []
}
* ``"BOLD_confounds"`` :
.. code-block:: none
{
"mandatory": ["pattern", "format"],
"optional": []
}
* ``"VBM_GM"`` :
.. code-block:: none
{
"mandatory": ["pattern", "space"],
"optional": []
}
* ``"VBM_WM"`` :
.. code-block:: none
{
"mandatory": ["pattern", "space"],
"optional": []
}
Basically, for each data type, one needs to provide ``mandatory`` keys
and can choose to also provide ``optional`` keys. The value for each
key is a string. So, one needs to provide necessary data types as a
dictionary, for example:
.. code-block:: none
{
"BOLD": {
"pattern": "...",
"space": "...",
},
"T1w": {
"pattern": "...",
"space": "...",
},
"Warp": {
"pattern": "...",
"src": "...",
"dst": "...",
}
}
taken from :class:`.HCP1200`.
replacements : str or list of str replacements : str or list of str
Replacements in the patterns for each item in the "element" tuple. Replacements in the ``pattern`` key of each data type. The value needs
to be a list of all possible replacements.
datadir : str or pathlib.Path datadir : str or pathlib.Path
The directory where the data is / will be stored. The directory where the data is / will be stored.
confounds_format : {"fmriprep", "adhoc"} or None, optional confounds_format : {"fmriprep", "adhoc"} or None, optional
@ -52,7 +139,7 @@ class PatternDataGrabber(BaseDataGrabber):
def __init__( def __init__(
self, self,
types: List[str], types: List[str],
patterns: Dict[str, str], patterns: Dict[str, Dict[str, str]],
replacements: Union[List[str], str], replacements: Union[List[str], str],
datadir: Union[str, Path], datadir: Union[str, Path],
confounds_format: Optional[str] = None, confounds_format: Optional[str] = None,
@ -69,7 +156,10 @@ class PatternDataGrabber(BaseDataGrabber):
self.replacements = replacements self.replacements = replacements
# Validate confounds format # Validate confounds format
if confounds_format and confounds_format not in _CONFOUNDS_FORMATS: if (
confounds_format is not None
and confounds_format not in _CONFOUNDS_FORMATS
):
raise_error( raise_error(
"Invalid value for `confounds_format`, should be one of " "Invalid value for `confounds_format`, should be one of "
f"{_CONFOUNDS_FORMATS}." f"{_CONFOUNDS_FORMATS}."
@ -143,6 +233,11 @@ class PatternDataGrabber(BaseDataGrabber):
str str
The pattern with the element replaced. The pattern with the element replaced.
Raises
------
ValueError
If element keys do not match with replacements.
""" """
if list(element.keys()) != self.replacements: if list(element.keys()) != self.replacements:
raise_error( raise_error(
@ -167,7 +262,7 @@ class PatternDataGrabber(BaseDataGrabber):
return self.replacements return self.replacements
def get_item(self, **element: str) -> Dict[str, Dict]: def get_item(self, **element: str) -> Dict[str, Dict]:
"""Implement single element indexing in the database. """Implement single element indexing for the datagrabber.
This method constructs a real path to the requested item's data, by This method constructs a real path to the requested item's data, by
replacing the ``patterns`` with actual values passed via ``**element``. replacing the ``patterns`` with actual values passed via ``**element``.
@ -184,20 +279,33 @@ class PatternDataGrabber(BaseDataGrabber):
Dictionary of dictionaries for each type of data required for the Dictionary of dictionaries for each type of data required for the
specified element. specified element.
Raises
------
RuntimeError
If more than one file matches for a data type's pattern or
if no file matches for a data type's pattern or
if file cannot be accessed for an element.
""" """
out = {} out = {}
for t_type in self.types: for t_type in self.types:
t_pattern = self.patterns[t_type] t_pattern = self.patterns[t_type]
t_replace = self._replace_patterns_glob(element, t_pattern) t_replace = self._replace_patterns_glob(
element, t_pattern["pattern"]
)
if "*" in t_replace: if "*" in t_replace:
t_matches = list(self.datadir.absolute().glob(t_replace)) t_matches = list(self.datadir.absolute().glob(t_replace))
if len(t_matches) > 1: if len(t_matches) > 1:
raise_error( raise_error(
f"More than one file matches for {element} / {t_type}:" f"More than one file matches for {element} / {t_type}:"
f" {t_matches}" f" {t_matches}",
klass=RuntimeError,
) )
elif len(t_matches) == 0: elif len(t_matches) == 0:
raise_error(f"No file matches for {element} / {t_type}") raise_error(
f"No file matches for {element} / {t_type}",
klass=RuntimeError,
)
t_out = t_matches[0] t_out = t_matches[0]
else: else:
t_out = self.datadir / t_replace t_out = self.datadir / t_replace
@ -205,22 +313,13 @@ class PatternDataGrabber(BaseDataGrabber):
if not t_out.exists() and not t_out.is_symlink(): if not t_out.exists() and not t_out.is_symlink():
raise_error( raise_error(
f"Cannot access {t_type} for {element}: " f"Cannot access {t_type} for {element}: "
f"File {t_out} does not exist" f"File {t_out} does not exist",
klass=RuntimeError,
) )
# Update path for the element # Update path for the element
out[t_type] = {"path": t_out} out[t_type] = t_pattern.copy() # copy data type dictionary
# Update confounds format for BOLD_confounds out[t_type].pop("pattern") # remove pattern key
# (if found in the datagrabber) out[t_type].update({"path": t_out}) # add path key
if t_type == "BOLD_confounds":
if not self.confounds_format:
raise_error(
"`confounds_format` needs to be one of "
f"{_CONFOUNDS_FORMATS}, None provided. "
"As the DataGrabber used specifies "
"'BOLD_confounds', None is invalid."
)
# Set the format
out[t_type].update({"format": self.confounds_format})
return out return out
@ -259,7 +358,7 @@ class PatternDataGrabber(BaseDataGrabber):
re_pattern, re_pattern,
glob_pattern, glob_pattern,
t_replacements, t_replacements,
) = self._replace_patterns_regex(t_pattern) ) = self._replace_patterns_regex(t_pattern["pattern"])
for fname in self.datadir.glob(glob_pattern): for fname in self.datadir.glob(glob_pattern):
suffix = fname.relative_to(self.datadir).as_posix() suffix = fname.relative_to(self.datadir).as_posix()
m = re.match(re_pattern, suffix) m = re.match(re_pattern, suffix)

124
junifer/datagrabber/pattern_datalad.py
View file

@ -5,12 +5,11 @@
# Synchon Mandal <s.mandal@fz-juelich.de> # Synchon Mandal <s.mandal@fz-juelich.de>
# License: AGPL # License: AGPL
from typing import Dict, List
from ..api.decorators import register_datagrabber from ..api.decorators import register_datagrabber
from ..utils import logger
from .datalad_base import DataladDataGrabber from .datalad_base import DataladDataGrabber
from .pattern import PatternDataGrabber from .pattern import PatternDataGrabber
from .utils import validate_patterns
@register_datagrabber @register_datagrabber
@ -25,11 +24,109 @@ class PatternDataladDataGrabber(DataladDataGrabber, PatternDataGrabber):
types : list of str types : list of str
The types of data to be grabbed. The types of data to be grabbed.
patterns : dict patterns : dict
Patterns for each type of data as a dictionary. The keys are the types Data type patterns as a dictionary. It has the following schema:
and the values are the patterns. Each occurrence of the string
``{subject}`` in the pattern will be replaced by the indexed element. * ``"T1w"`` :
**kwargs
Keyword arguments passed to superclass. .. code-block:: none
{
"mandatory": ["pattern", "space"],
"optional": []
}
* ``"T2w"`` :
.. code-block:: none
{
"mandatory": ["pattern", "space"],
"optional": []
}
* ``"BOLD"`` :
.. code-block:: none
{
"mandatory": ["pattern", "space"],
"optional": ["mask_item"]
}
* ``"Warp"`` :
.. code-block:: none
{
"mandatory": ["pattern", "src", "dst"],
"optional": []
}
* ``"BOLD_confounds"`` :
.. code-block:: none
{
"mandatory": ["pattern", "format"],
"optional": []
}
* ``"VBM_GM"`` :
.. code-block:: none
{
"mandatory": ["pattern", "space"],
"optional": []
}
* ``"VBM_WM"`` :
.. code-block:: none
{
"mandatory": ["pattern", "space"],
"optional": []
}
Basically, for each data type, one needs to provide ``mandatory`` keys
and can choose to also provide ``optional`` keys. The value for each
key is a string. So, one needs to provide necessary data types as a
dictionary, for example:
.. code-block:: none
{
"BOLD": {
"pattern": "...",
"space": "...",
},
"T1w": {
"pattern": "...",
"space": "...",
},
"Warp": {
"pattern": "...",
"src": "...",
"dst": "...",
}
}
taken from :class:`.HCP1200`.
replacements : str or list of str
Replacements in the ``pattern`` key of each data type. The value needs
to be a list of all possible replacements.
confounds_format : {"fmriprep", "adhoc"} or None, optional
The format of the confounds for the dataset (default None).
datadir : str or pathlib.Path or None, optional
That directory where the datalad dataset will be cloned. If None,
the datalad dataset will be cloned into a temporary directory
(default None).
rootdir : str or pathlib.Path, optional
The path within the datalad dataset to the root directory
(default ".").
uri : str or None, optional
URI of the datalad sibling (default None).
See Also See Also
-------- --------
@ -42,12 +139,13 @@ class PatternDataladDataGrabber(DataladDataGrabber, PatternDataGrabber):
def __init__( def __init__(
self, self,
types: List[str],
patterns: Dict[str, str],
**kwargs, **kwargs,
) -> None: ) -> None:
# Validate patterns # TODO(synchon): needs to be reworked, DataladDataGrabber needs to be
validate_patterns(types=types, patterns=patterns) # a mixin to avoid multiple inheritance wherever possible.
super().__init__(types=types, patterns=patterns, **kwargs) logger.debug("Initializing PatternDataladDataGrabber")
self.patterns = patterns for key, val in kwargs.items():
logger.debug(f"\t{key} = {val}")
super().__init__(**kwargs)

6
junifer/datagrabber/tests/test_base.py
View file

@ -12,12 +12,6 @@ import pytest
from junifer.datagrabber import BaseDataGrabber from junifer.datagrabber import BaseDataGrabber
def test_BaseDataGrabber_abstractness() -> None:
"""Test BaseDataGrabber is abstract base class."""
with pytest.raises(TypeError, match=r"abstract"):
BaseDataGrabber(datadir="/tmp", types=["func"]) # type: ignore
def test_BaseDataGrabber() -> None: def test_BaseDataGrabber() -> None:
"""Test BaseDataGrabber.""" """Test BaseDataGrabber."""

262
junifer/datagrabber/tests/test_datagrabber_utils.py
View file

@ -3,6 +3,9 @@
# Authors: Federico Raimondo <f.raimondo@fz-juelich.de> # Authors: Federico Raimondo <f.raimondo@fz-juelich.de>
# License: AGPL # License: AGPL
from contextlib import nullcontext
from typing import ContextManager, Dict, List, Union
import pytest import pytest
from junifer.datagrabber.utils import ( from junifer.datagrabber.utils import (
@ -12,79 +15,204 @@ from junifer.datagrabber.utils import (
) )
def test_validate_types() -> None: @pytest.mark.parametrize(
"""Test validation of types.""" "types, expect",
with pytest.raises(TypeError, match="must be a list"): [
validate_types("wrong") # type: ignore ("wrong", pytest.raises(TypeError, match="must be a list")),
with pytest.raises(TypeError, match="must be a list of strings"): ([1], pytest.raises(TypeError, match="must be a list of strings")),
validate_types([1]) # type: ignore (["T1w", "BOLD"], nullcontext()),
],
)
def test_validate_types(
types: Union[str, List[str], List[int]],
expect: ContextManager,
) -> None:
"""Test validation of types.
validate_types(["T1w", "BOLD"]) Parameters
----------
types : str, list of int or str
The parametrized data types to validate.
expect : typing.ContextManager
The parametrized ContextManager object.
"""
with expect:
validate_types(types) # type: ignore
def test_validate_replacements() -> None: @pytest.mark.parametrize(
"""Test validation of replacements.""" "replacements, patterns, expect",
with pytest.raises(TypeError, match="must be a list"): [
validate_replacements("wrong", "also wrong") # type: ignore (
with pytest.raises(TypeError, match="must be a dict"): "wrong",
validate_replacements(["correct"], "wrong") # type: ignore "also wrong",
pytest.raises(TypeError, match="must be a list"),
),
(
[1],
{
"T1w": {"pattern": "{subject}/anat/{subject}_T1w.nii.gz"},
"BOLD": {
"pattern": "{subject}/func/{subject}_task-rest_bold.nii.gz"
},
},
pytest.raises(TypeError, match="must be a list of strings"),
),
(
["session"],
{
"T1w": {"pattern": "{subject}/anat/{subject}_T1w.nii.gz"},
"BOLD": {
"pattern": "{subject}/func/{subject}_task-rest_bold.nii.gz"
},
},
pytest.raises(ValueError, match="is not part of"),
),
(
["subject", "session"],
{
"T1w": {"pattern": "{subject}/anat/_T1w.nii.gz"},
"BOLD": {"pattern": "{session}/func/_task-rest_bold.nii.gz"},
},
pytest.raises(ValueError, match="At least one pattern"),
),
(
["subject"],
{
"T1w": {"pattern": "{subject}/anat/{subject}_T1w.nii.gz"},
"BOLD": {
"pattern": "{subject}/func/{subject}_task-rest_bold.nii.gz"
},
},
nullcontext(),
),
],
)
def test_validate_replacements(
replacements: Union[str, List[str], List[int]],
patterns: Union[str, Dict[str, Dict[str, str]]],
expect: ContextManager,
) -> None:
"""Test validation of replacements.
patterns = { Parameters
"T1w": "{subject}/anat/{subject}_T1w.nii.gz", ----------
"BOLD": "{subject}/func/{subject}_task-rest_bold.nii.gz", replacements : str, list of str or int
} The parametrized pattern replacements to validate.
patterns : str, dict
The parametrized patterns to validate against.
expect : typing.ContextManager
The parametrized ContextManager object.
with pytest.raises(TypeError, match="must be a list of strings"): """
validate_replacements([1], patterns) # type: ignore with expect:
validate_replacements(replacements=replacements, patterns=patterns) # type: ignore
with pytest.raises(ValueError, match="is not part of"):
validate_replacements(["session"], patterns)
wrong_patterns = {
"T1w": "{subject}/anat/_T1w.nii.gz",
"BOLD": "{session}/func/_task-rest_bold.nii.gz",
}
with pytest.raises(ValueError, match="At least one pattern"):
validate_replacements(["subject", "session"], wrong_patterns)
validate_replacements(["subject"], patterns)
def test_validate_patterns() -> None: @pytest.mark.parametrize(
"""Test validation of patterns.""" "types, patterns, expect",
types = ["T1w", "BOLD"] [
with pytest.raises(TypeError, match="must be a dict"): (
validate_patterns(types, "wrong") # type: ignore ["T1w", "BOLD"],
"wrong",
pytest.raises(TypeError, match="must be a dict"),
),
(
["T1w", "BOLD"],
{
"T1w": {"pattern": "{subject}/anat/{subject}_T1w.nii.gz"},
},
pytest.raises(
ValueError,
match="Length of `types` more than that of `patterns`.",
),
),
(
["T1w", "BOLD"],
{
"T1w": {"pattern": "{subject}/anat/{subject}_T1w.nii.gz"},
"T2w": {"pattern": "{subject}/anat/{subject}_T2w.nii.gz"},
},
pytest.raises(ValueError, match="contain all"),
),
(
["T3w"],
{
"T3w": {"pattern": "{subject}/anat/{subject}_T3w.nii.gz"},
},
pytest.raises(ValueError, match="Unknown data type"),
),
(
["BOLD"],
{
"BOLD": {"patterns": "{subject}/func/{subject}_BOLD.nii.gz"},
},
pytest.raises(KeyError, match="Mandatory key"),
),
(
["BOLD_confounds"],
{
"BOLD_confounds": {
"pattern": "{subject}/func/{subject}_confounds.tsv",
"format": "fmriprep",
"space": "MNINLin6Asym",
},
},
pytest.raises(RuntimeError, match="not accepted"),
),
(
["T1w"],
{
"T1w": {
"pattern": "{subject}/anat/{subject}*.nii",
"space": "native",
},
},
pytest.raises(ValueError, match="following a replacement"),
),
(
["T1w", "T2w", "BOLD", "BOLD_confounds"],
{
"T1w": {
"pattern": "{subject}/anat/{subject}_T1w.nii.gz",
"space": "native",
},
"T2w": {
"pattern": "{subject}/anat/{subject}_T2w.nii.gz",
"space": "native",
},
"BOLD": {
"pattern": (
"{subject}/func/{subject}_task-rest_bold.nii.gz"
),
"space": "MNI152NLin6Asym",
},
"BOLD_confounds": {
"pattern": "{subject}/func/{subject}_confounds.tsv",
"format": "fmriprep",
},
},
nullcontext(),
),
],
)
def test_validate_patterns(
types: List[str],
patterns: Union[str, Dict[str, Dict[str, str]]],
expect: ContextManager,
) -> None:
"""Test validation of patterns.
wrongpatterns = { Parameters
"T1w": "{subject}/anat/{subject}_T1w.nii.gz", ----------
} types : list of str
The parametrized data types.
patterns : str, dict
The patterns to validate.
expect : typing.ContextManager
The parametrized ContextManager object.
with pytest.raises( """
ValueError, match="Length of `types` more than that of `patterns`." with expect:
): validate_patterns(types=types, patterns=patterns) # type: ignore
validate_patterns(types, wrongpatterns) # type: ignore
wrongpatterns = {
"T1w": "{subject}/anat/{subject}_T1w.nii.gz",
"T2": "{subject}/anat/{subject}_T2.nii.gz",
}
with pytest.raises(ValueError, match="contain all"):
validate_patterns(types, wrongpatterns) # type: ignore
patterns = {
"T1w": "{subject}/anat/{subject}_T1w.nii.gz",
"BOLD": "{subject}/func/{subject}_task-rest_bold.nii.gz",
}
wrongpatterns = {
"T1w": "{subject}/anat/{subject}*.nii",
"BOLD": "{subject}/func/{subject}_task-rest_bold.nii.gz",
}
with pytest.raises(ValueError, match="following a replacement"):
validate_patterns(types, wrongpatterns)
validate_patterns(types, patterns)

6
junifer/datagrabber/tests/test_datalad_base.py
View file

@ -26,12 +26,6 @@ _testing_dataset = {
} }
def test_DataladDataGrabber_abstractness() -> None:
"""Test DataladDataGrabber is abstract base class."""
with pytest.raises(TypeError, match=r"abstract"):
DataladDataGrabber() # type: ignore
@pytest.fixture @pytest.fixture
def concrete_datagrabber() -> Type[DataladDataGrabber]: def concrete_datagrabber() -> Type[DataladDataGrabber]:
"""Return a concrete datalad-based DataGrabber. """Return a concrete datalad-based DataGrabber.

53
junifer/datagrabber/tests/test_multiple.py
View file

@ -26,11 +26,21 @@ def test_MultipleDataGrabber() -> None:
rootdir = "example_bids_ses" rootdir = "example_bids_ses"
replacements = ["subject", "session"] replacements = ["subject", "session"]
pattern1 = { pattern1 = {
"T1w": "{subject}/{session}/anat/{subject}_{session}_T1w.nii.gz", "T1w": {
"pattern": (
"{subject}/{session}/anat/{subject}_{session}_T1w.nii.gz"
),
"space": "native",
},
} }
pattern2 = { pattern2 = {
"BOLD": "{subject}/{session}/func/" "BOLD": {
"{subject}_{session}_task-rest_bold.nii.gz", "pattern": (
"{subject}/{session}/func/"
"{subject}_{session}_task-rest_bold.nii.gz"
),
"space": "MNI152NLin6Asym",
},
} }
dg1 = PatternDataladDataGrabber( dg1 = PatternDataladDataGrabber(
rootdir=rootdir, rootdir=rootdir,
@ -84,11 +94,21 @@ def test_MultipleDataGrabber_no_intersection() -> None:
rootdir = "example_bids_ses" rootdir = "example_bids_ses"
replacements = ["subject", "session"] replacements = ["subject", "session"]
pattern1 = { pattern1 = {
"T1w": "{subject}/{session}/anat/{subject}_{session}_T1w.nii.gz", "T1w": {
"pattern": (
"{subject}/{session}/anat/{subject}_{session}_T1w.nii.gz"
),
"space": "native",
},
} }
pattern2 = { pattern2 = {
"BOLD": "{subject}/{session}/func/" "BOLD": {
"{subject}_{session}_task-rest_bold.nii.gz", "pattern": (
"{subject}/{session}/func/"
"{subject}_{session}_task-rest_bold.nii.gz"
),
"space": "MNI152NLin6Asym",
},
} }
dg1 = PatternDataladDataGrabber( dg1 = PatternDataladDataGrabber(
rootdir=rootdir, rootdir=rootdir,
@ -119,7 +139,12 @@ def test_MultipleDataGrabber_get_item() -> None:
rootdir = "example_bids_ses" rootdir = "example_bids_ses"
replacements = ["subject", "session"] replacements = ["subject", "session"]
pattern1 = { pattern1 = {
"T1w": "{subject}/{session}/anat/{subject}_{session}_T1w.nii.gz", "T1w": {
"pattern": (
"{subject}/{session}/anat/{subject}_{session}_T1w.nii.gz"
),
"space": "native",
},
} }
dg1 = PatternDataladDataGrabber( dg1 = PatternDataladDataGrabber(
rootdir=rootdir, rootdir=rootdir,
@ -142,10 +167,18 @@ def test_MultipleDataGrabber_validation() -> None:
replacement1 = ["subject", "session"] replacement1 = ["subject", "session"]
replacement2 = ["subject"] replacement2 = ["subject"]
pattern1 = { pattern1 = {
"T1w": "{subject}/{session}/anat/{subject}_{session}_T1w.nii.gz", "T1w": {
"pattern": (
"{subject}/{session}/anat/{subject}_{session}_T1w.nii.gz"
),
"space": "native",
},
} }
pattern2 = { pattern2 = {
"bold": "{subject}/func/{subject}_task-rest_bold.nii.gz", "BOLD": {
"pattern": "{subject}/func/{subject}_task-rest_bold.nii.gz",
"space": "MNI152NLin6Asym",
},
} }
dg1 = PatternDataladDataGrabber( dg1 = PatternDataladDataGrabber(
rootdir=rootdir, rootdir=rootdir,
@ -158,7 +191,7 @@ def test_MultipleDataGrabber_validation() -> None:
dg2 = PatternDataladDataGrabber( dg2 = PatternDataladDataGrabber(
rootdir=rootdir, rootdir=rootdir,
uri=repo_uri2, uri=repo_uri2,
types=["bold"], types=["BOLD"],
patterns=pattern2, patterns=pattern2,
replacements=replacement2, replacements=replacement2,
) )

303
junifer/datagrabber/tests/test_pattern.py
View file

@ -5,6 +5,7 @@
# Synchon Mandal <s.mandal@fz-juelich.de> # Synchon Mandal <s.mandal@fz-juelich.de>
# License: AGPL # License: AGPL
from itertools import product
from pathlib import Path from pathlib import Path
import pytest import pytest
@ -21,144 +22,88 @@ def test_PatternDataGrabber_errors(tmp_path: Path) -> None:
The path to the test directory. The path to the test directory.
""" """
with pytest.raises(TypeError, match=r"`types` must be a list"):
PatternDataGrabber(
datadir="/tmp",
types="wrong", # type: ignore
patterns={"wrong": "pattern"},
replacements="subject", # type: ignore
)
with pytest.raises(TypeError, match=r"`types` must be a list of strings"):
PatternDataGrabber(
datadir="/tmp", # type: ignore
types=[1, 2, 3], # type: ignore
patterns={"1": "pattern", "2": "pattern", "3": "pattern"},
replacements="subject", # type: ignore
)
with pytest.raises(
ValueError, match=r"`patterns` must contain all `types`"
):
PatternDataGrabber(
datadir="/tmp",
types=["func", "anat"],
patterns={"1": "pattern", "2": "pattern", "3": "pattern"},
replacements=1, # type: ignore
)
with pytest.raises(TypeError, match=r"`patterns` must be a dict"):
PatternDataGrabber(
datadir="/tmp",
types=["func", "anat"],
patterns="wrong", # type: ignore
replacements="subject", # type: ignore
)
with pytest.raises(
ValueError, match=r"Length of `types` more than that of `patterns`"
):
PatternDataGrabber(
datadir="/tmp",
types=["func", "anat"],
patterns={"wrong": "pattern"},
replacements="subject", # type: ignore
)
with pytest.raises(
ValueError, match=r"`patterns` must contain all `types`"
):
PatternDataGrabber(
datadir="/tmp",
types=["func", "anat"],
patterns={"wrong": "pattern", "func": "pattern"},
replacements="subject", # type: ignore
)
with pytest.raises(TypeError, match=r"must be a list of strings"):
PatternDataGrabber(
datadir="/tmp",
types=["func", "anat"],
patterns={"func": "func/test", "anat": "anat/test"},
replacements=1, # type: ignore
)
with pytest.raises(ValueError, match=r"not part of any pattern"):
PatternDataGrabber(
datadir="/tmp",
types=["func", "anat"],
patterns={
"func": "func/{subject}.nii",
"anat": "anat/{subject}.nii",
},
replacements=["subject", "wrong"],
)
tmpdir = tmp_path / "pattern_dg_test_errors" tmpdir = tmp_path / "pattern_dg_test_errors"
datagrabber = PatternDataGrabber( datagrabber_no_access = PatternDataGrabber(
datadir=tmpdir, datadir=tmpdir,
types=["func", "anat"], types=["BOLD", "T1w"],
patterns={ patterns={
"func": "func/{subject}_single.nii", "BOLD": {
"anat": "anat/{subject}_{session}_ses.nii", "pattern": "func/{subject}_single.nii",
"space": "MNI152NLin6Asym",
},
"T1w": {
"pattern": "anat/{subject}_{session}_ses.nii",
"space": "MNI152NLin6Asym",
},
}, },
replacements=["subject", "session"], replacements=["subject", "session"],
) )
with pytest.raises(ValueError, match="element keys must be"): with pytest.raises(ValueError, match="element keys must be"):
datagrabber["sub001"] datagrabber_no_access[("sub001")]
# This should not work, file does not exists # This should not work, file does not exists
with pytest.raises(ValueError, match="Cannot access"): with pytest.raises(RuntimeError, match="Cannot access"):
datagrabber["sub001", "ses001"] datagrabber_no_access[("sub001", "ses001")]
# Create directories and files
(tmpdir / "func").mkdir(exist_ok=True, parents=True) (tmpdir / "func").mkdir(exist_ok=True, parents=True)
(tmpdir / "anat").mkdir(exist_ok=True, parents=True) (tmpdir / "anat").mkdir(exist_ok=True, parents=True)
for t_subject in range(3): for t_subject, t_session in product(range(3), range(2)):
for t_session in range(2): subject = f"sub{t_subject:03d}"
subject = f"sub{t_subject:03d}" session = f"ses{t_session:03d}"
session = f"ses{t_session:03d}" (tmpdir / "func" / f"{subject}_single.nii").touch()
(tmpdir / "func" / f"{subject}_single.nii").touch() if t_subject == 2:
if t_subject == 2: (tmpdir / "func" / f"{subject}_extra.nii").touch()
(tmpdir / "func" / f"{subject}_extra.nii").touch() (tmpdir / "anat" / f"{subject}_{session}_ses.nii").touch()
(tmpdir / "anat" / f"{subject}_{session}_ses.nii").touch()
# This should work, file now exists # This should work, file now exists
datagrabber["sub001", "ses001"] datagrabber_no_access[("sub001", "ses001")]
datagrabber = PatternDataGrabber( datagrabber_multi_access = PatternDataGrabber(
datadir=tmpdir, datadir=tmpdir,
types=["func", "anat"], types=["BOLD", "T1w"],
patterns={ patterns={
"func": "func/{subject}_*.nii", "BOLD": {
"anat": "anat/{subject}_{session}_*.nii", "pattern": "func/{subject}_*.nii",
"space": "MNI152NLin6Asym",
},
"T1w": {
"pattern": "anat/{subject}_{session}_*.nii",
"space": "MNI152NLin6Asym",
},
}, },
replacements=["subject", "session"], replacements=["subject", "session"],
) )
# access a subject with a missing session # Access a subject with a missing session
with pytest.raises(ValueError, match="No file matches"): with pytest.raises(RuntimeError, match="No file matches"):
datagrabber["sub001", "ses004"] datagrabber_multi_access[("sub001", "ses004")]
# access a subject with two matching files # Access a subject with two matching files
with pytest.raises(ValueError, match="More than one"): with pytest.raises(RuntimeError, match="More than one"):
datagrabber["sub002", "ses001"] datagrabber_multi_access[("sub002", "ses001")]
# access the right one # Access the right one
datagrabber["sub001", "ses001"] datagrabber_multi_access[("sub001", "ses001")]
datagrabber = PatternDataGrabber( datagrabber_fake_access = PatternDataGrabber(
datadir=tmpdir, datadir=tmpdir,
types=["func", "anat2"], types=["BOLD", "T1w"],
patterns={ patterns={
"func": "func/{subject}_single.nii", "BOLD": {
"anat2": "anat2/{subject}_{session}_ses.nii", "pattern": "func/{subject}_single.nii",
"space": "MNI152NLin6Asym",
},
"T1w": {
"pattern": "anat2/{subject}_{session}_ses.nii",
"space": "MNI152NLin6Asym",
},
}, },
replacements=["subject", "session"], replacements=["subject", "session"],
) )
assert len(datagrabber.get_elements()) == 0 assert len(datagrabber_fake_access.get_elements()) == 0
def test_PatternDataGrabber(tmp_path: Path) -> None: def test_PatternDataGrabber(tmp_path: Path) -> None:
@ -171,43 +116,59 @@ def test_PatternDataGrabber(tmp_path: Path) -> None:
""" """
datagrabber = PatternDataGrabber( datagrabber_first = PatternDataGrabber(
datadir="/tmp/data", datadir="/tmp/data",
types=["func", "anat"], types=["BOLD", "T1w"],
patterns={"func": "func/{subject}.nii", "anat": "anat/{subject}.nii"}, patterns={
"BOLD": {
"pattern": "func/{subject}.nii",
"space": "MNI152NLin6Asym",
},
"T1w": {
"pattern": "anat/{subject}.nii",
"space": "native",
},
},
replacements="subject", replacements="subject",
) )
assert datagrabber.datadir == Path("/tmp/data") assert datagrabber_first.datadir == Path("/tmp/data")
assert datagrabber.types == ["func", "anat"] assert set(datagrabber_first.types) == {"T1w", "BOLD"}
assert datagrabber.replacements == ["subject"] assert datagrabber_first.replacements == ["subject"]
datagrabber = PatternDataGrabber( datagrabber_second = PatternDataGrabber(
datadir=Path("/tmp/data"), datadir=Path("/tmp/data"),
types=["func", "anat"], types=["BOLD", "T1w"],
patterns={ patterns={
"func": "func/{subject}.nii", "BOLD": {
"anat": "anat/{subject}_{session}.nii", "pattern": "func/{subject}.nii",
"space": "MNI152NLin6Asym",
},
"T1w": {
"pattern": "anat/{subject}_{session}.nii",
"space": "native",
},
}, },
replacements=["subject", "session"], replacements=["subject", "session"],
) )
assert datagrabber.datadir == Path("/tmp/data") assert datagrabber_second.datadir == Path("/tmp/data")
assert datagrabber.types == ["func", "anat"] assert set(datagrabber_second.types) == {"T1w", "BOLD"}
assert datagrabber.replacements == ["subject", "session"] assert datagrabber_second.replacements == ["subject", "session"]
# Create directories and files
tmpdir = tmp_path / "pattern_dg_test" tmpdir = tmp_path / "pattern_dg_test"
(tmpdir / "func").mkdir(exist_ok=True, parents=True) (tmpdir / "func").mkdir(exist_ok=True, parents=True)
(tmpdir / "anat").mkdir(exist_ok=True, parents=True) (tmpdir / "anat").mkdir(exist_ok=True, parents=True)
(tmpdir / "vbm").mkdir(exist_ok=True, parents=True) (tmpdir / "vbm").mkdir(exist_ok=True, parents=True)
for t_subject in range(3): for t_subject, t_session, t_task in product(
for t_session in range(2): range(3), range(2), range(2, 4)
for t_task in range(2, 4): ):
subject = f"sub{t_subject:03d}" subject = f"sub{t_subject:03d}"
session = f"ses{t_session:03d}" session = f"ses{t_session:03d}"
task = f"task{t_task:03d}" task = f"task{t_task:03d}"
if t_subject != 2: if t_subject != 2:
(tmpdir / "func" / f"{subject}.nii").touch() (tmpdir / "func" / f"{subject}.nii").touch()
(tmpdir / "anat" / f"{subject}_{session}.nii").touch() (tmpdir / "anat" / f"{subject}_{session}.nii").touch()
(tmpdir / "vbm" / f"{subject}_{task}_{session}.nii").touch() (tmpdir / "vbm" / f"{subject}_{task}_{session}.nii").touch()
expected_elements = [ expected_elements = [
("sub000", "ses000"), ("sub000", "ses000"),
@ -218,16 +179,19 @@ def test_PatternDataGrabber(tmp_path: Path) -> None:
("sub002", "ses001"), ("sub002", "ses001"),
] ]
datagrabber = PatternDataGrabber( datagrabber_third = PatternDataGrabber(
datadir=tmpdir, datadir=tmpdir,
types=["anat"], types=["T1w"],
patterns={ patterns={
"anat": "anat/{subject}_{session}.nii", "T1w": {
"pattern": "anat/{subject}_{session}.nii",
"space": "native",
},
}, },
replacements=["subject", "session"], replacements=["subject", "session"],
) )
elements = datagrabber.get_elements() elements = datagrabber_third.get_elements()
assert set(elements) == set(expected_elements) assert set(elements) == set(expected_elements)
expected_elements = [ expected_elements = [
@ -241,26 +205,35 @@ def test_PatternDataGrabber(tmp_path: Path) -> None:
("sub001", "ses001", "task003"), ("sub001", "ses001", "task003"),
] ]
datagrabber = PatternDataGrabber( datagrabber_fourth = PatternDataGrabber(
datadir=tmpdir, datadir=tmpdir,
types=["func", "anat", "vbm"], types=["T1w", "BOLD", "VBM_GM"],
patterns={ patterns={
"func": "func/{subject}.nii", "BOLD": {
"anat": "anat/{subject}_{session}.nii", "pattern": "func/{subject}.nii",
"vbm": "vbm/{subject}_{task}_{session}.nii", "space": "MNI152NLin6Asym",
},
"T1w": {
"pattern": "anat/{subject}_{session}.nii",
"space": "native",
},
"VBM_GM": {
"pattern": "vbm/{subject}_{task}_{session}.nii",
"space": "MNI152NLin6Asym",
},
}, },
replacements=["subject", "session", "task"], replacements=["subject", "session", "task"],
) )
elements = datagrabber.get_elements() elements = datagrabber_fourth.get_elements()
assert set(elements) == set(expected_elements) assert set(elements) == set(expected_elements)
out1 = datagrabber[("sub000", "ses000", "task002")] out1 = datagrabber_fourth[("sub000", "ses000", "task002")]
out2 = datagrabber[("sub000", "ses000", "task003")] out2 = datagrabber_fourth[("sub000", "ses000", "task003")]
assert out1["func"]["path"] == out2["func"]["path"] assert out1["BOLD"]["path"] == out2["BOLD"]["path"]
assert out1["anat"]["path"] == out2["anat"]["path"] assert out1["T1w"]["path"] == out2["T1w"]["path"]
assert out1["vbm"]["path"] != out2["vbm"]["path"] assert out1["VBM_GM"]["path"] != out2["VBM_GM"]["path"]
def test_PatternDataGrabber_confounds_format_error_on_init() -> None: def test_PatternDataGrabber_confounds_format_error_on_init() -> None:
@ -269,40 +242,14 @@ def test_PatternDataGrabber_confounds_format_error_on_init() -> None:
ValueError, match="Invalid value for `confounds_format`" ValueError, match="Invalid value for `confounds_format`"
): ):
PatternDataGrabber( PatternDataGrabber(
types=["func"], types=["BOLD"],
patterns={"func": "func/{subject}.nii"}, patterns={
"BOLD": {
"pattern": "func/{subject}.nii",
"space": "MNI152NLin6Asym",
},
},
replacements=["subject"], replacements=["subject"],
datadir="/tmp", datadir="/tmp",
confounds_format="foobar", confounds_format="foobar",
) )
def test_PatternDataGrabber_confounds_format_error_on_fetch(
tmp_path: Path,
) -> None:
"""Test PatterDataGrabber confounds format error on fetching.
Parameters
----------
tmp_path : pathlib.Path
The path to the test directory.
"""
# Create test directory path
tmpdir = tmp_path / "pattern_dg_test"
# Create final test directory
(tmpdir / "func" / "confounds").mkdir(exist_ok=True, parents=True)
# Create test confound file
(tmpdir / "func" / "confounds" / "sub-001.nii").touch()
# Initialize datagrabber
datagrabber = PatternDataGrabber(
types=["BOLD_confounds"],
patterns={"BOLD_confounds": "func/confounds/{subject}.nii"},
replacements=["subject"],
datadir=tmpdir,
)
# Check error on fetch
with pytest.raises(
ValueError, match="As the DataGrabber used specifies 'BOLD_confounds'"
):
datagrabber.get_item(subject="sub-001")

69
junifer/datagrabber/tests/test_pattern_datalad.py
View file

@ -43,8 +43,14 @@ def test_bids_PatternDataladDataGrabber() -> None:
types = ["T1w", "BOLD"] types = ["T1w", "BOLD"]
# Define patterns # Define patterns
patterns = { patterns = {
"T1w": "{subject}/anat/{subject}_T1w.nii.gz", "T1w": {
"BOLD": "{subject}/func/{subject}_task-rest_bold.nii.gz", "pattern": "{subject}/anat/{subject}_T1w.nii.gz",
"space": "MNI152NLin6Asym",
},
"BOLD": {
"pattern": "{subject}/func/{subject}_task-rest_bold.nii.gz",
"space": "MNI152NLin6Asym",
},
} }
# Define replacements # Define replacements
replacements = ["subject"] replacements = ["subject"]
@ -92,31 +98,28 @@ def test_bids_PatternDataladDataGrabber() -> None:
def test_bids_PatternDataladDataGrabber_datadir() -> None: def test_bids_PatternDataladDataGrabber_datadir() -> None:
"""Test PatternDataladDataGrabber with a datadir set to a relative path.""" """Test PatternDataladDataGrabber with a datadir set to a relative path."""
# Define types
types = ["T1w", "BOLD"]
# Define patterns # Define patterns
patterns = { patterns = {
"T1w": "{subject}/anat/{subject}_T1w.nii.gz", "T1w": {
"BOLD": "{subject}/func/{subject}_task-rest_bold.nii.gz", "pattern": "{subject}/anat/{subject}_T*w.nii.gz",
"space": "MNI152NLin6Asym",
},
"BOLD": {
"pattern": "{subject}/func/{subject}_task-rest_*.nii.gz",
"space": "MNI152NLin6Asym",
},
} }
# Define replacements # Define datadir
replacements = ["subject"]
repo_uri = _testing_dataset["example_bids"]["uri"]
datadir = "dataset" # use string and not absolute path datadir = "dataset" # use string and not absolute path
patterns = {
"T1w": "example_bids/{subject}/anat/{subject}_T*w.nii.gz",
"BOLD": "example_bids/{subject}/func/{subject}_task-rest_*.nii.gz",
}
with PatternDataladDataGrabber( with PatternDataladDataGrabber(
uri=repo_uri, uri=_testing_dataset["example_bids"]["uri"],
types=types, types=["T1w", "BOLD"],
patterns=patterns, patterns=patterns,
datadir=datadir, datadir=datadir,
replacements=replacements, rootdir="example_bids",
replacements=["subject"],
) as dg: ) as dg:
assert dg.datadir == Path(datadir) assert dg.datadir == Path(datadir) / "example_bids"
for elem in dg: for elem in dg:
t_sub = dg[elem] t_sub = dg[elem]
assert "path" in t_sub["T1w"] assert "path" in t_sub["T1w"]
@ -133,12 +136,23 @@ def test_bids_PatternDataladDataGrabber_session():
"""Test a subject and session-based BIDS PatternDataladDataGrabber.""" """Test a subject and session-based BIDS PatternDataladDataGrabber."""
types = ["T1w", "BOLD"] types = ["T1w", "BOLD"]
patterns = { patterns = {
"T1w": "{subject}/{session}/anat/{subject}_{session}_T1w.nii.gz", "T1w": {
"BOLD": "{subject}/{session}/func/" "pattern": (
"{subject}_{session}_task-rest_bold.nii.gz", "{subject}/{session}/anat/{subject}_{session}_T1w.nii.gz"
),
"space": "MNI152NLin6Asym",
},
"BOLD": {
"pattern": (
"{subject}/{session}/func/"
"{subject}_{session}_task-rest_bold.nii.gz"
),
"space": "MNI152NLin6Asym",
},
} }
replacements = ["subject", "session"] replacements = ["subject", "session"]
# Check error
with pytest.raises(ValueError, match=r"`uri` must be provided"): with pytest.raises(ValueError, match=r"`uri` must be provided"):
PatternDataladDataGrabber( PatternDataladDataGrabber(
datadir=None, datadir=None,
@ -147,9 +161,9 @@ def test_bids_PatternDataladDataGrabber_session():
replacements=replacements, replacements=replacements,
) )
# Set parameters
repo_uri = _testing_dataset["example_bids_ses"]["uri"] repo_uri = _testing_dataset["example_bids_ses"]["uri"]
rootdir = "example_bids_ses" rootdir = "example_bids_ses"
# repo_commit = _testing_dataset['example_bids_ses']['id']
# With T1W and bold, only 2 sessions are available # With T1W and bold, only 2 sessions are available
with PatternDataladDataGrabber( with PatternDataladDataGrabber(
@ -159,7 +173,7 @@ def test_bids_PatternDataladDataGrabber_session():
patterns=patterns, patterns=patterns,
replacements=replacements, replacements=replacements,
) as dg: ) as dg:
subs = list(dg) subs = list(dg.get_elements())
expected_subs = [ expected_subs = [
(f"sub-{i:02d}", f"ses-{j:02d}") (f"sub-{i:02d}", f"ses-{j:02d}")
for j in range(1, 3) for j in range(1, 3)
@ -170,7 +184,12 @@ def test_bids_PatternDataladDataGrabber_session():
# Test with a different T1w only, it should have 3 sessions # Test with a different T1w only, it should have 3 sessions
types = ["T1w"] types = ["T1w"]
patterns = { patterns = {
"T1w": "{subject}/{session}/anat/{subject}_{session}_T1w.nii.gz", "T1w": {
"pattern": (
"{subject}/{session}/anat/{subject}_{session}_T1w.nii.gz"
),
"space": "MNI152NLin6Asym",
},
} }
with PatternDataladDataGrabber( with PatternDataladDataGrabber(
rootdir=rootdir, rootdir=rootdir,

162
junifer/datagrabber/utils.py
View file

@ -6,7 +6,68 @@
from typing import Dict, List from typing import Dict, List
from ..utils import raise_error from ..utils import logger, raise_error
# Define schema for pattern-based datagrabber's patterns
PATTERNS_SCHEMA = {
"T1w": {
"mandatory": ["pattern", "space"],
"optional": ["mask_item"],
},
"T1w_mask": {
"mandatory": ["pattern", "space"],
"optional": [],
},
"T2w": {
"mandatory": ["pattern", "space"],
"optional": ["mask_item"],
},
"T2w_mask": {
"mandatory": ["pattern", "space"],
"optional": [],
},
"BOLD": {
"mandatory": ["pattern", "space"],
"optional": ["mask_item"],
},
"BOLD_confounds": {
"mandatory": ["pattern", "format"],
"optional": [],
},
"BOLD_mask": {
"mandatory": ["pattern", "space"],
"optional": [],
},
"Warp": {
"mandatory": ["pattern", "src", "dst"],
"optional": [],
},
"VBM_GM": {
"mandatory": ["pattern", "space"],
"optional": [],
},
"VBM_WM": {
"mandatory": ["pattern", "space"],
"optional": [],
},
"probseg_CSF": {
"mandatory": ["pattern", "space"],
"optional": [],
},
"probseg_GM": {
"mandatory": ["pattern", "space"],
"optional": [],
},
"probseg_WM": {
"mandatory": ["pattern", "space"],
"optional": [],
},
"DWI": {
"mandatory": ["pattern"],
"optional": [],
},
}
def validate_types(types: List[str]) -> None: def validate_types(types: List[str]) -> None:
@ -30,7 +91,7 @@ def validate_types(types: List[str]) -> None:
def validate_replacements( def validate_replacements(
replacements: List[str], patterns: Dict[str, str] replacements: List[str], patterns: Dict[str, Dict[str, str]]
) -> None: ) -> None:
"""Validate the replacements. """Validate the replacements.
@ -44,38 +105,41 @@ def validate_replacements(
Raises Raises
------ ------
TypeError TypeError
If ``replacements`` is not a list or if the values are not string or If ``replacements`` is not a list or if the values are not string.
if ``patterns`` is not a dictionary.
ValueError ValueError
If a value in ``replacements`` is not in ``pattern`` or if no value in If a value in ``replacements`` is not part of a data type pattern or
``patterns`` contain all values in ``replacements``. if no data type patterns contain all values in ``replacements``.
""" """
if not isinstance(replacements, list): if not isinstance(replacements, list):
raise_error(msg="`replacements` must be a list.", klass=TypeError) raise_error(msg="`replacements` must be a list.", klass=TypeError)
if not isinstance(patterns, dict):
raise_error(msg="`patterns` must be a dict.", klass=TypeError)
if any(not isinstance(x, str) for x in replacements): if any(not isinstance(x, str) for x in replacements):
raise_error( raise_error(
msg="`replacements` must be a list of strings.", klass=TypeError msg="`replacements` must be a list of strings.", klass=TypeError
) )
for x in replacements: for x in replacements:
if all(x not in y for y in patterns.values()): if all(
raise_error(msg=f"Replacement {x} is not part of any pattern.") x not in y
for y in [
data_type_val["pattern"] for data_type_val in patterns.values()
]
):
raise_error(msg=f"Replacement: {x} is not part of any pattern.")
# Check that at least one pattern has all the replacements # Check that at least one pattern has all the replacements
at_least_one = False at_least_one = False
for _, v in patterns.items(): for data_type_val in patterns.values():
if all(x in v for x in replacements): if all(x in data_type_val["pattern"] for x in replacements):
at_least_one = True at_least_one = True
if at_least_one is False: if at_least_one is False:
raise_error(msg="At least one pattern must contain all replacements.") raise_error(msg="At least one pattern must contain all replacements.")
def validate_patterns(types: List[str], patterns: Dict[str, str]) -> None: def validate_patterns(
types: List[str], patterns: Dict[str, Dict[str, str]]
) -> None:
"""Validate the patterns. """Validate the patterns.
Parameters Parameters
@ -87,12 +151,17 @@ def validate_patterns(types: List[str], patterns: Dict[str, str]) -> None:
Raises Raises
------ ------
KeyError
If any mandatory key is missing for a data type.
RuntimeError
If an unknown key is found for a data type.
TypeError TypeError
If ``patterns`` is not a dictionary. If ``patterns`` is not a dictionary.
ValueError ValueError
If length of ``types`` and ``patterns`` are different or If length of ``types`` and ``patterns`` are different or
if ``patterns`` is missing entries from ``types`` or if ``patterns`` is missing entries from ``types`` or
if ``patterns`` contain '*' as value. if unknown data type is found in ``patterns`` or
if data type pattern key contains '*' as value.
""" """
# Validate the types # Validate the types
@ -110,9 +179,60 @@ def validate_patterns(types: List[str], patterns: Dict[str, str]) -> None:
raise_error( raise_error(
msg="`patterns` must contain all `types`", klass=ValueError msg="`patterns` must contain all `types`", klass=ValueError
) )
# Wildcard check in patterns # Check against schema
if any("}*" in pattern for pattern in patterns.values()): for data_type_key, data_type_val in patterns.items():
raise_error( # Check if valid data type is provided
msg="`patterns` must not contain `*` following a replacement", if data_type_key not in PATTERNS_SCHEMA:
klass=ValueError, raise_error(
) f"Unknown data type: {data_type_key}, "
f"should be one of: {list(PATTERNS_SCHEMA.keys())}"
)
# Check mandatory keys for data type
for mandatory_key in PATTERNS_SCHEMA[data_type_key]["mandatory"]:
if mandatory_key not in data_type_val:
raise_error(
msg=(
f"Mandatory key: `{mandatory_key}` missing for "
f"{data_type_key}"
),
klass=KeyError,
)
else:
logger.debug(
f"Mandatory key: `{mandatory_key}` found for "
f"{data_type_key}"
)
# Check optional keys for data type
for optional_key in PATTERNS_SCHEMA[data_type_key]["optional"]:
if optional_key not in data_type_val:
logger.debug(
f"Optional key: `{optional_key}` missing for "
f"{data_type_key}"
)
else:
logger.debug(
f"Optional key: `{optional_key}` found for "
f"{data_type_key}"
)
# Check stray key for data type
for key in data_type_val.keys():
if key not in (
PATTERNS_SCHEMA[data_type_key]["mandatory"]
+ PATTERNS_SCHEMA[data_type_key]["optional"]
):
raise_error(
msg=(
f"Key: {key} not accepted for {data_type_key} "
"pattern, remove it to proceed"
),
klass=RuntimeError,
)
# Wildcard check in patterns
if "}*" in data_type_val["pattern"]:
raise_error(
msg=(
f"`{data_type_key}.pattern` must not contain `*` "
"following a replacement"
),
klass=ValueError,
)

13
junifer/datareader/default.py
View file

@ -53,23 +53,22 @@ class DefaultDataReader(PipelineStepMixin, UpdateMetaMixin):
# Nothing to validate, any input is fine # Nothing to validate, any input is fine
return input return input
def get_output_type(self, input: List[str]) -> List[str]: def get_output_type(self, input_type: str) -> str:
"""Get output type. """Get output type.
Parameters Parameters
---------- ----------
input : list of str input_type : str
The input to the reader. The list must contain the The data type input to the reader.
available Junifer Data dictionary keys.
Returns Returns
------- -------
list of str str
The updated list of output types, as reading possibilities. The data type output by the reader.
""" """
# It will output the same type of data as the input # It will output the same type of data as the input
return input return input_type
def _fit_transform( def _fit_transform(
self, self,