diff --git a/README.md b/README.md index 5ba79f77c..9f89a61c9 100644 --- a/README.md +++ b/README.md @@ -1,3 +1,5 @@ +![Junifer logo](docs/images/junifer_logo.png) + # junifer - JUelich NeuroImaging FEature extractoR ![PyPI](https://img.shields.io/pypi/v/junifer?style=flat-square) diff --git a/docs/builtin.rst b/docs/builtin.rst index d1e04d007..19ef11d07 100644 --- a/docs/builtin.rst +++ b/docs/builtin.rst @@ -14,7 +14,7 @@ Data Grabbers - Open with registration - Restricted - Type/config: this should mention weather the class is built-in in the + Type/config: this should mention whether the class is built-in in the core of junifer or needs to be imported from a specific configuration in the `junifer.configs` module. @@ -47,7 +47,7 @@ Available - Built-in - In Progress - :gh:`4` - * - :class:`junifer.configs.juseless.JuselessDataladUKBVBM` + * - :class:`junifer.configs.juseless.datagrabbers.JuselessDataladUKBVBM` - UKB VBM dataset preprocessed with CAT. Available for Juseless only. - Restricted - ``junifer.configs.juseless`` @@ -56,7 +56,7 @@ Available * - :class:`junifer.configs.juseless.datagrabbers.JuselessDataladCamCANVBM` - CamCAN VBM dataset preprocessed with CAT. Available for Juseless only. - Restricted - - ``junifer.configs.juseless.datagrabbers`` + - ``junifer.configs.juseless`` - Done - 0.0.1 * - :class:`junifer.datagrabber.DataladAOMICID1000` @@ -77,7 +77,7 @@ Available - Built-in - Done - 0.0.1 - * - :class:`junifer.configs.juseless.JuselessDataladAOMICVBM` + * - :class:`junifer.configs.juseless.datagrabbers.JuselessDataladAOMICVBM` - AOMIC VBM dataset. Available for Juseless only. - Restricted - ``junifer.configs.juseless`` @@ -86,7 +86,7 @@ Available * - :class:`junifer.configs.juseless.datagrabbers.JuselessDataladIXIVBM` - `IXI VBM dataset `_. Available for Juseless only. - Restricted - - ``junifer.configs.juseless.datagrabbers`` + - ``junifer.configs.juseless`` - Done - 0.0.1 @@ -154,6 +154,10 @@ Available - Spherical aggregation using mean - Done - 0.0.1 + * - :class:`junifer.markers.FunctionalConnectivitySpheres` + - Perform spherical aggregation and compute functional connectivity + - Done + - 0.0.1 * - :class:`junifer.markers.RSSETSMarker` - Compute root sum of squares of edgewise timeseries - Done diff --git a/docs/conf.py b/docs/conf.py index a6ba15330..fd12b8dcd 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -64,9 +64,19 @@ exclude_patterns = ["_build", "Thumbs.db", ".DS_Store"] # a list of builtin themes. # html_theme = "sphinx_rtd_theme" - -html_sidebars = {"**": ["globaltoc.html", "sourcelink.html", "searchbox.html"]} - +html_theme_options = { + "display_version": True, + "style_external_links": True, + "logo_only": True, +} +html_sidebars = { + "**": [ + "globaltoc.html", + "sourcelink.html", + "searchbox.html", + ] +} +html_logo = "./images/junifer_logo.png" # Add any paths that contain custom static files (such as style sheets) here, # relative to this directory. They are copied after the builtin static files, diff --git a/docs/images/junifer_logo.png b/docs/images/junifer_logo.png new file mode 100644 index 000000000..3810ebf98 Binary files /dev/null and b/docs/images/junifer_logo.png differ diff --git a/docs/index.rst b/docs/index.rst index cef0f0606..5aaa20d88 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -1,5 +1,9 @@ .. include:: links.inc +.. image:: images/junifer_logo.png + :width: 300px + :alt: junifer logo + Welcome to junifer's documentation! =================================== diff --git a/docs/installation.rst b/docs/installation.rst index 90209b8e5..f52e13d1c 100644 --- a/docs/installation.rst +++ b/docs/installation.rst @@ -9,14 +9,14 @@ Requirements junifer is compatible with `Python`_ >= 3.8 and requires the following packages: -* click>=8.1.3,<8.2 -* numpy>=1.22,<1.23 -* datalad>=0.15.4,<0.18 -* pandas>=1.4.0,<1.5 -* nibabel>=3.2.0,<4.1 -* nilearn>=0.9.0,<1.0 -* sqlalchemy>=1.4.27,<= 1.5.0 -* pyyaml>=5.1.2,<7.0 +* ``click>=8.1.3,<8.2`` +* ``numpy>=1.22,<1.23`` +* ``datalad>=0.15.4,<0.18`` +* ``pandas>=1.4.0,<1.5`` +* ``nibabel>=3.2.0,<4.1`` +* ``nilearn>=0.9.0,<1.0`` +* ``sqlalchemy>=1.4.27,<= 1.5.0`` +* ``pyyaml>=5.1.2,<7.0`` Depending on the installation method, these packages might be installed automatically. diff --git a/docs/understanding/data.rst b/docs/understanding/data.rst index 1010720a8..b119cd6ac 100644 --- a/docs/understanding/data.rst +++ b/docs/understanding/data.rst @@ -1,5 +1,7 @@ .. include:: ../links.inc +.. _data_object: + The Data Object =============== @@ -7,7 +9,7 @@ Description ^^^^^^^^^^^ This is the *object* that traverses the steps of the pipeline. It is indeed a -dictionary of dictionaries. The first level of keys are the :ref:`data_types` +dictionary of dictionaries. The first level of keys are the :ref:`data types ` and a special key named ``meta`` that contains all the information on the data object including source and previous transformation steps. @@ -16,17 +18,19 @@ The second level of keys are the actual data. So far, there are two keys used: - ``path``: path to the file containing the data. - ``data``: the data loaded in memory. -The :ref:`datagrabber` step will only fill the ``path`` value. -The ``data`` value will be filled by the :ref:`datareader` step, if it is one of the possible file types +The :ref:`DataGrabber ` step will only fill the ``path`` value. +The ``data`` value will be filled by the :ref:`DataReader ` step, if it is one of the possible file types that the datareader can read. +A point to note is that you never directly interact with the *data object* but it's important to know where and how the object is being manipulated to reason about your pipeine. + .. _data_types: Data types ^^^^^^^^^^ -.. list-table:: Built-in data types - :widths: 30 80 40 +.. list-table:: + :widths: auto :header-rows: 1 * - Name diff --git a/docs/understanding/datagrabber.rst b/docs/understanding/datagrabber.rst index ed27cc606..aa6cb7df5 100644 --- a/docs/understanding/datagrabber.rst +++ b/docs/understanding/datagrabber.rst @@ -2,50 +2,52 @@ .. _datagrabber: -Data Grabber -============ - +DataGrabber +=========== Description ^^^^^^^^^^^ -A datagrabber is an object that can provide datasets you want to junifer. -For example, a DataladDataGrabber can provide data from a Datalad dataset to junifer. -Of course, datagrabbers are not only possible for Datalad but any origin of a dataset. -It is intended to use them as context managers. -If you are interested in just using already provided datagrabbers please go to :doc:`../builtin`. -If you want to implement your own Data Grabbers you need to inherit from different types of -Data Grabbers we already provide. -Typical Data Grabbers -^^^^^^^^^^^^^^^^^^^^^ -In this section we will showcase different types of datagrabber classes you might want to use -to implement your own datagrabbers for your own data. +The ``DataGrabber`` is an object that can provide an interface to datasets you want to work with in junifer. +Every concrete implementation of a datagrabber is aware of a particular dataset's structure and thus allows +you to fetch specific elements of interest from the dataset. It adds the ``path`` key to each :ref:`data type ` +in the :ref:`Data object `. -.. list-table:: Data Grabbers Type - :widths: 25 35 +Datagrabbers are intended to be used as context managers. When used within a context, a datagrabber takes care +of any pre and post steps for interacting with the dataset, for example, downloading and cleaning up. As the interface +is consistent, you always use the same procedure to interact with the datagrabber. + +For example, a concrete implementation of :class:`junifer.datagrabber.DataladDataGrabber` can provide junifer +with data from a Datalad dataset. Of course, datagrabbers are not only meant to work with Datalad datasets but +any dataset. + +If you are interested in using already provided datagrabbers, please go to :doc:`../builtin`. And, if you want +to implement your own datagrabber, you need to provide concrete implementations of base classes already +provided. + +Base classes +^^^^^^^^^^^^ +In this section, we showcase different abstract base classes you might want to use to implement your own datagrabber. + +.. list-table:: + :widths: auto :header-rows: 1 * - Name - Description - * - :py:class:`~junifer.datagrabber.base.BaseDataGrabber` - - | An abstract class providing you an interface to implement for you own datagrabber. - | This is not intedent to be used in general. - | Instead you should use the DataladDataGrabber or PatternDataGrabber if possible. - | You have to at least implement the ``get_elements`` method, but most of the time - | you should also overwrite other existing methods like ``__enter__`` and ``__exit__``. - * - :py:class:`~junifer.datagrabber.pattern.PatternDataGrabber` - - | Implements some functionality to help you to define the pattern of the dataset you want to get. - | E.g. you know that T1 images are found in a directory following this pattern - | inside of the dataset "{subject}/anat/{subject}_T1w.nii.gz". - | Now you can provide this to the PatternDataGrabber - | and it will be able to get the image to junifer. - * - :py:class:`~junifer.datagrabber.datalad_base.DataladDataGrabber` - - | Implements some functionality specific to basic usage of Datalad datasets. - | This mostly takes care of the ``__enter__`` and ``__exit__`` methods to clone your Datalad dataset - | and remove it on ``__exit__``. - | This means that even if your code fails inside of the context of the - | datagrabber the DataladDataGrabber cleans up the created Datalad directories. - * - :py:class:`~junifer.datagrabber.pattern_datalad.PatternDataladDataGrabber` - - | Combination of PatternDataGrabber and DataladDataGrabber. - | Probably the class you are looking for when using Datalad. - + * - :class:`junifer.datagrabber.BaseDataGrabber` + - | The abstract base class providing you an interface to implement your own datagrabber. + | You should try to avoid using this directly and instead use + | :class:`junifer.datagrabber.PatternDataGrabber` or :class:`junifer.datagrabber.DataladDataGrabber`. + | To build your own custom *low-level* datagrabber, you need to at least implement the ``get_elements`` method, + | but most of the time you should also override other existing methods like ``__enter__`` and ``__exit__``. + * - :class:`junifer.datagrabber.PatternDataGrabber` + - | It implements functionality to help you define the pattern of the dataset you want to get. For example, + | you know that T1 images are found in a directory following this pattern ``{subject}/anat/{subject}_T1w.nii.gz`` + | inside of the dataset. Now you can provide this to the **PatternDataGrabber** and it will be able to get the file. + * - :class:`junifer.datagrabber.DataladDataGrabber` + - | It implements functionality to deal with Datalad datasets. Specifically, the ``__enter__`` and ``__exit__`` methods + | take care of cloning and removing the Datalad dataset. + * - :class:`junifer.datagrabber.PatternDataladDataGrabber` + - | It is a combination of :class:`junifer.datagrabber.PatternPatternDataGrabber` and + | :class:`junifer.datagrabber.DataladDataGrabber`. This is probably the class you are looking for when using Datalad. diff --git a/docs/understanding/datareader.rst b/docs/understanding/datareader.rst index 8a3521efd..774f402db 100644 --- a/docs/understanding/datareader.rst +++ b/docs/understanding/datareader.rst @@ -2,5 +2,45 @@ .. _datareader: -Data Reader -=========== +DataReader +========== + +Description +^^^^^^^^^^^ + +The ``DataReader`` is an object that is responsible for actually reading data files in junifer. +It reads the value of the key ``path`` for each :ref:`data type ` in the :ref:`Data object ` +and loads them to memory. After reading the data into memory, it adds the key ``data`` to the same level as ``path`` +and the value is the actual data in the memory. + +Datareaders are meant to be used inside the datagrabber context but you can operate on them outside the context as long +as the actual data is in the memory and the Python runtime has not garbage-collected it. + +For data formats not supported by junifer yet, you can either make your own ``DataReader`` or open an issue on +`junifer Github`_ and we can help you out. + +Currently supported file-formats +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +We already provide a concrete implementation :class:`junifer.datareader.DefaultDataReader` which knows how to +read the following file formats: + +.. list-table:: + :widths: auto + :header-rows: 1 + + * - File extension + - File type + - Description + * - ``.nii`` + - NIfTI (uncompressed) + - Uncompressed NIfTI + * - ``.nii.gz`` + - NIfTI (compressed) + - Compressed NIfTI + * - ``.csv`` + - CSV + - Comma-separated values file + * - ``.tsv`` + - TSV + - Tab-separated values file diff --git a/docs/understanding/index.rst b/docs/understanding/index.rst index e24c92883..abc1a6e00 100644 --- a/docs/understanding/index.rst +++ b/docs/understanding/index.rst @@ -4,7 +4,7 @@ Understanding junifer ===================== Before you start, you should understand how junifer works. Junifer is a -tool conceived to extract features from neuroimaging data in a easy-to-use +tool conceived to extract features from neuroimaging data in an easy-to-use manner, with minimal coding and minimal user expertise in the internal aspects. Unlike other tools like FSL, SPM, AFNI, etc., junifer is not a toolbox to @@ -24,5 +24,6 @@ julearn_). data datagrabber datareader + preprocess marker storage diff --git a/docs/understanding/marker.rst b/docs/understanding/marker.rst index 3bf018ba7..915f8ed99 100644 --- a/docs/understanding/marker.rst +++ b/docs/understanding/marker.rst @@ -1,4 +1,23 @@ .. include:: ../links.inc +.. _marker: + Marker ====== + +Description +^^^^^^^^^^^ + +The ``Marker`` is an object that is responsible for feature extraction. It primarily operates on data loaded +in memory by :ref:`datareader ` and stored in the ``data`` key of each :ref:`data type ` +in the :ref:`Data object `. In some cases, it can also operate on pre-processed data as obtained +from the :ref:`Preprocess ` step of the pipeline. It is important to note that this pre-process is +not similar to pre-processing done by tools like FSL, SPM, AFNI, etc. . For example, one can perform confound +removal on loaded data and then perform feature extraction. + +Markers are meant to be used inside the datagrabber context but you can operate on them outside the context as long +as the actual data is in the memory and the Python runtime has not garbage-collected it. + +If you are interested in using already provided markers, please go to :doc:`../builtin`. And, if you want to implement +your own marker, you need to provide concrete implementation of :class:`junifer.markers.BaseMarker`. Specifically, you +need to override ``get_output_kind``, ``store`` and ``compute`` methods. diff --git a/docs/understanding/preprocess.rst b/docs/understanding/preprocess.rst new file mode 100644 index 000000000..eea630214 --- /dev/null +++ b/docs/understanding/preprocess.rst @@ -0,0 +1,15 @@ +.. include:: ../links.inc + +.. _preprocess: + +Preprocess +========== + +Description +^^^^^^^^^^^ + +The ``Preprocess`` step of the pipeline is meant for pre-processing before or after :ref:`Marker ` step +depending on the use-case. For example, you might want to perform confound removal on ``BOLD`` data before +feature extraction. + +This step is still under development and is an optional one for the pipeline to work. diff --git a/docs/understanding/storage.rst b/docs/understanding/storage.rst index ef296e1d4..c409809b0 100644 --- a/docs/understanding/storage.rst +++ b/docs/understanding/storage.rst @@ -1,4 +1,41 @@ .. include:: ../links.inc +.. _storage: + Storage ======= + +Description +^^^^^^^^^^^ + +The ``Storage`` is an object that is responsible for storing extracted features as computed from :ref:`Marker ` +step of the pipeline. If the pipeline is provided with a ``storage-like`` object, the extracted features are stored via +that object else they are kept in memory. + +Storage is meant to be used inside the datagrabber context but you can operate on them outside the context as long +as the processed data is in the memory and the Python runtime has not garbage-collected it. + +The :ref:`Markers ` are responsible for defining what *storage kind* (``matrix``, ``table``, ``timeseries``) +they support for which :ref:`data type ` by overriding its ``store`` method. The storage object in turn +declares and provides implementation for specific *storage kind*. For example, :class:`junifer.storage.SQLiteFeatureStorage` +supports saving ``matrix``, ``table`` and ``timeseries`` via ``store_matrix``, ``store_table`` and ``store_timeseries`` +methods respectively. + +For storage interfaces not supported by junifer yet, you can either make your own ``Storage`` by providing a concrete +implementation of :class:`junifer.storage.BaseFeatureStorage` or open an issue on `junifer Github`_ and we can help you out. + +Currently supported storage interfaces +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. list-table:: + :widths: auto + :header-rows: 1 + + * - Storage class + - File extension + - File type + - Storage kinds + * - :class:`junifer.storage.SQLiteFeatureStorage` + - ``.db`` + - SQLite + - ``matrix``, ``table``, ``timeseries``