[ENH]: Allow for dumping data object to disk for debugging purposes #452

Merged

synchon merged 9 commits from feat/data-obj-dumper into main

2025-07-31 14:17:10 +00:00

synchon commented

2025-07-29 17:01:01 +00:00

(Migrated from github.com)

Are you requiring a new dataset or marker?

I understand this is not a marker or dataset request

Which feature do you want to include?

The possibility to set a set of configuration flags that dumps the data object into a directory (configurable) either at the end of preprocessing or minor step.

How do you imagine this integrated in junifer?

I see a few env vars/config flags (ConfigManager) that make junifer dump the data object:

JUNIFER_PREPROCESSING_DUMP_LOCATION: which takes a path. Within the path, a directory for the element will be created and data dumped inside
JUNIFER_PREPROCESSING_DUMP_GRANULARITY: "final" or "full". If "final", only after preprocessing. if "full", it will be dumped before preprocessing (output from data reader) and after each preprocessing step.

I see this implemented in a class DataObjectDumper which also allows to register functions that allows to save each class in a native format (instead of using pickle). Example: if its a nifti image, don't use pickle for that field, but save as .nii.gz. This should be extensible as it might be needed for EEG data too.

Do you have a sample code that implements this outside of junifer?

Anything else to say?

No response

### Are you requiring a new dataset or marker? - [x] I understand this is not a marker or dataset request ### Which feature do you want to include? The possibility to set a set of configuration flags that dumps the data object into a directory (configurable) either at the end of preprocessing or minor step. ### How do you imagine this integrated in junifer? I see a few env vars/config flags (`ConfigManager`) that make junifer dump the data object: - `JUNIFER_PREPROCESSING_DUMP_LOCATION`: which takes a path. Within the path, a directory for the element will be created and data dumped inside - `JUNIFER_PREPROCESSING_DUMP_GRANULARITY`: "final" or "full". If "final", only after preprocessing. if "full", it will be dumped before preprocessing (output from data reader) and after each preprocessing step. I see this implemented in a class `DataObjectDumper` which also allows to register functions that allows to save each class in a native format (instead of using pickle). Example: if its a nifti image, don't use pickle for that field, but save as `.nii.gz`. This should be extensible as it might be needed for EEG data too. ### Do you have a sample code that implements this outside of junifer? ```shell ``` ### Anything else to say? _No response_

codecov[bot] commented

2025-07-29 17:36:55 +00:00

(Migrated from github.com)

❌ Patch coverage is 91.11111% with 16 lines in your changes missing coverage. Please review.
✅ Project coverage is 91.29%. Comparing base (ee99ec6) to head (8540d98).
⚠️ Report is 10 commits behind head on main.

Files with missing lines	Patch %	Lines
junifer/pipeline/_data_object_dumper.py	90.00%	12 Missing and 4 partials ⚠️

❌ Your patch status has failed because the patch coverage (91.11%) is below the target coverage (95.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #452      +/-   ##
==========================================
- Coverage   91.30%   91.29%   -0.02%     
==========================================
  Files         135      136       +1     
  Lines        5588     5764     +176     
  Branches      910      941      +31     
==========================================
+ Hits         5102     5262     +160     
- Misses        313      325      +12     
- Partials      173      177       +4

Flag	Coverage Δ
docs	`100.00% <ø> (ø)`
junifer	`91.28% <91.11%> (-0.02%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
junifer/api/decorators.py	`100.00% <100.00%> (ø)`
junifer/pipeline/marker_collection.py	`100.00% <100.00%> (ø)`
junifer/typing/_typing.py	`100.00% <100.00%> (ø)`
junifer/pipeline/_data_object_dumper.py	`90.00% <90.00%> (ø)`

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

## [Codecov](https://app.codecov.io/gh/juaml/junifer/pull/452?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml) Report :x: Patch coverage is `91.11111%` with `16 lines` in your changes missing coverage. Please review. :white_check_mark: Project coverage is 91.29%. Comparing base ([`ee99ec6`](https://app.codecov.io/gh/juaml/junifer/commit/ee99ec664be8a217da75670cd87d02c9afe88de8?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml)) to head ([`8540d98`](https://app.codecov.io/gh/juaml/junifer/commit/8540d98691518650715b50bf282798f392b9b45f?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml)). :warning: Report is 10 commits behind head on main. | [Files with missing lines](https://app.codecov.io/gh/juaml/junifer/pull/452?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml) | Patch % | Lines | |---|---|---| | [junifer/pipeline/\_data\_object\_dumper.py](https://app.codecov.io/gh/juaml/junifer/pull/452?src=pr&el=tree&filepath=junifer%2Fpipeline%2F_data_object_dumper.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml#diff-anVuaWZlci9waXBlbGluZS9fZGF0YV9vYmplY3RfZHVtcGVyLnB5) | 90.00% | [12 Missing and 4 partials :warning: ](https://app.codecov.io/gh/juaml/junifer/pull/452?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml) | :x: Your patch status has failed because the patch coverage (91.11%) is below the target coverage (95.00%). You can increase the patch coverage or adjust the [target](https://docs.codecov.com/docs/commit-status#target) coverage. <details><summary>Additional details and impacted files</summary> [![Impacted file tree graph](https://app.codecov.io/gh/juaml/junifer/pull/452/graphs/tree.svg?width=650&height=150&src=pr&token=5H21JuZXMw&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml)](https://app.codecov.io/gh/juaml/junifer/pull/452?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml) ```diff @@ Coverage Diff @@ ## main #452 +/- ## ========================================== - Coverage 91.30% 91.29% -0.02% ========================================== Files 135 136 +1 Lines 5588 5764 +176 Branches 910 941 +31 ========================================== + Hits 5102 5262 +160 - Misses 313 325 +12 - Partials 173 177 +4 ``` | [Flag](https://app.codecov.io/gh/juaml/junifer/pull/452/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml) | Coverage Δ | | |---|---|---| | [docs](https://app.codecov.io/gh/juaml/junifer/pull/452/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml) | `100.00% <ø> (ø)` | | | [junifer](https://app.codecov.io/gh/juaml/junifer/pull/452/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml) | `91.28% <91.11%> (-0.02%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml#carryforward-flags-in-the-pull-request-comment) to find out more. | [Files with missing lines](https://app.codecov.io/gh/juaml/junifer/pull/452?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml) | Coverage Δ | | |---|---|---| | [junifer/api/decorators.py](https://app.codecov.io/gh/juaml/junifer/pull/452?src=pr&el=tree&filepath=junifer%2Fapi%2Fdecorators.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml#diff-anVuaWZlci9hcGkvZGVjb3JhdG9ycy5weQ==) | `100.00% <100.00%> (ø)` | | | [junifer/pipeline/marker\_collection.py](https://app.codecov.io/gh/juaml/junifer/pull/452?src=pr&el=tree&filepath=junifer%2Fpipeline%2Fmarker_collection.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml#diff-anVuaWZlci9waXBlbGluZS9tYXJrZXJfY29sbGVjdGlvbi5weQ==) | `100.00% <100.00%> (ø)` | | | [junifer/typing/\_typing.py](https://app.codecov.io/gh/juaml/junifer/pull/452?src=pr&el=tree&filepath=junifer%2Ftyping%2F_typing.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml#diff-anVuaWZlci90eXBpbmcvX3R5cGluZy5weQ==) | `100.00% <100.00%> (ø)` | | | [junifer/pipeline/\_data\_object\_dumper.py](https://app.codecov.io/gh/juaml/junifer/pull/452?src=pr&el=tree&filepath=junifer%2Fpipeline%2F_data_object_dumper.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml#diff-anVuaWZlci9waXBlbGluZS9fZGF0YV9vYmplY3RfZHVtcGVyLnB5) | `90.00% <90.00%> (ø)` | | </details> <details><summary> :rocket: New features to boost your workflow: </summary> - :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems. </details>

github-actions[bot] commented

2025-07-29 17:43:00 +00:00

(Migrated from github.com)

PR Preview Action v1.6.2
Preview removed because the pull request was closed.
2025-07-31 14:55 UTC

[PR Preview Action](https://github.com/rossjrw/pr-preview-action) v1.6.2 :---: Preview removed because the pull request was closed. 2025-07-31 14:55 UTC

fraimondo (Migrated from github.com) approved these changes

2025-07-31 13:35:08 +00:00

fraimondo commented

2025-07-31 13:35:20 +00:00

(Migrated from github.com)

Excellent!