[ENH]: Introduce `ConfigManager` #401

Merged

fraimondo merged 10 commits from enh/config_manager into main

2024-11-26 14:50:30 +00:00

fraimondo commented

2024-11-25 15:14:02 +00:00

(Migrated from github.com)

The idea of this PR is to provide two functions junifer.utils.config.get and junifer.utils.config.set which can be retrieved at any moment, allowing for run-wide configuration options that can be set to control internal behaviour of junifer.

On intialization, the enviroment variables that start with JUNIFER_ will also be set.

E.g.:

In order to test/debug, we might already have a dataset cloned with some files already present. Currently, junifer will try to check if the dataset id matches with the URL and if also if the dataset is dirty.

By setting datagrabber.skipdirtycheck and datagrabber.skipidcheck, we can avoid cloning and checking, which are long operations on big datasets.

The idea of this PR is to provide two functions `junifer.utils.config.get` and `junifer.utils.config.set` which can be retrieved at any moment, allowing for run-wide configuration options that can be set to control internal behaviour of junifer. On intialization, the enviroment variables that start with `JUNIFER_` will also be set. E.g.: In order to test/debug, we might already have a dataset cloned with some files already present. Currently, junifer will try to check if the dataset id matches with the URL and if also if the dataset is dirty. By setting `datagrabber.skipdirtycheck` and `datagrabber.skipidcheck`, we can avoid cloning and checking, which are long operations on big datasets.

github-actions[bot] commented

2024-11-25 15:29:21 +00:00

(Migrated from github.com)

PR Preview Action v1.4.8
🚀 Deployed preview to https://juaml.github.io/junifer/pr-preview/pr-401/
on branch `gh-pages` at 2024-11-26 15:02 UTC

[PR Preview Action](https://github.com/rossjrw/pr-preview-action) v1.4.8 :---: :rocket: Deployed preview to https://juaml.github.io/junifer/pr-preview/pr-401/ on branch [`gh-pages`](https://github.com/juaml/junifer/tree/gh-pages) at 2024-11-26 15:02 UTC

codecov[bot] commented

2024-11-25 21:10:10 +00:00

(Migrated from github.com)

Attention: Patch coverage is 0% with 45 lines in your changes missing coverage. Please review.

Project coverage is 0.01%. Comparing base (1944462) to head (4388cb0).

Files with missing lines	Patch %	Lines
junifer/utils/_config.py	0.00%	37 Missing ⚠️
junifer/api/functions.py	0.00%	1 Missing ⚠️
junifer/cli/parser.py	0.00%	1 Missing ⚠️
junifer/data/masks/_masks.py	0.00%	1 Missing ⚠️
junifer/data/template_spaces.py	0.00%	1 Missing ⚠️
junifer/stats.py	0.00%	1 Missing ⚠️
junifer/typing/_typing.py	0.00%	1 Missing ⚠️
junifer/utils/helpers.py	0.00%	1 Missing ⚠️
junifer/utils/singleton.py	0.00%	1 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff            @@
##            main    #401      +/-   ##
========================================
- Coverage   0.01%   0.01%   -0.01%     
========================================
  Files        132     133       +1     
  Lines       5502    5540      +38     
========================================
  Hits           1       1              
- Misses      5501    5539      +38

Flag	Coverage Δ
junifer	`0.00% <0.00%> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
junifer/api/decorators.py	`0.00% <ø> (ø)`
junifer/cli/cli.py	`0.00% <ø> (ø)`
junifer/data/_dispatch.py	`0.00% <ø> (ø)`
junifer/storage/utils.py	`0.00% <ø> (ø)`
junifer/testing/datagrabbers.py	`0.00% <ø> (ø)`
junifer/utils/logging.py	`0.00% <ø> (ø)`
junifer/api/functions.py	`0.00% <0.00%> (ø)`
junifer/cli/parser.py	`0.00% <0.00%> (ø)`
junifer/data/masks/_masks.py	`0.00% <0.00%> (ø)`
junifer/data/template_spaces.py	`0.00% <0.00%> (ø)`
... and 5 more

## [Codecov](https://app.codecov.io/gh/juaml/junifer/pull/401?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml) Report Attention: Patch coverage is `0%` with `45 lines` in your changes missing coverage. Please review. > Project coverage is 0.01%. Comparing base [(`1944462`)](https://app.codecov.io/gh/juaml/junifer/commit/19444623578dd632f5633141958e7934b44e6d52?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml) to head [(`4388cb0`)](https://app.codecov.io/gh/juaml/junifer/commit/4388cb0071d8a6c7bab5f6547aaf25f995acbda6?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml). | [Files with missing lines](https://app.codecov.io/gh/juaml/junifer/pull/401?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml) | Patch % | Lines | |---|---|---| | [junifer/utils/\_config.py](https://app.codecov.io/gh/juaml/junifer/pull/401?src=pr&el=tree&filepath=junifer%2Futils%2F_config.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml#diff-anVuaWZlci91dGlscy9fY29uZmlnLnB5) | 0.00% | [37 Missing :warning: ](https://app.codecov.io/gh/juaml/junifer/pull/401?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml) | | [junifer/api/functions.py](https://app.codecov.io/gh/juaml/junifer/pull/401?src=pr&el=tree&filepath=junifer%2Fapi%2Ffunctions.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml#diff-anVuaWZlci9hcGkvZnVuY3Rpb25zLnB5) | 0.00% | [1 Missing :warning: ](https://app.codecov.io/gh/juaml/junifer/pull/401?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml) | | [junifer/cli/parser.py](https://app.codecov.io/gh/juaml/junifer/pull/401?src=pr&el=tree&filepath=junifer%2Fcli%2Fparser.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml#diff-anVuaWZlci9jbGkvcGFyc2VyLnB5) | 0.00% | [1 Missing :warning: ](https://app.codecov.io/gh/juaml/junifer/pull/401?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml) | | [junifer/data/masks/\_masks.py](https://app.codecov.io/gh/juaml/junifer/pull/401?src=pr&el=tree&filepath=junifer%2Fdata%2Fmasks%2F_masks.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml#diff-anVuaWZlci9kYXRhL21hc2tzL19tYXNrcy5weQ==) | 0.00% | [1 Missing :warning: ](https://app.codecov.io/gh/juaml/junifer/pull/401?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml) | | [junifer/data/template\_spaces.py](https://app.codecov.io/gh/juaml/junifer/pull/401?src=pr&el=tree&filepath=junifer%2Fdata%2Ftemplate_spaces.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml#diff-anVuaWZlci9kYXRhL3RlbXBsYXRlX3NwYWNlcy5weQ==) | 0.00% | [1 Missing :warning: ](https://app.codecov.io/gh/juaml/junifer/pull/401?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml) | | [junifer/stats.py](https://app.codecov.io/gh/juaml/junifer/pull/401?src=pr&el=tree&filepath=junifer%2Fstats.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml#diff-anVuaWZlci9zdGF0cy5weQ==) | 0.00% | [1 Missing :warning: ](https://app.codecov.io/gh/juaml/junifer/pull/401?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml) | | [junifer/typing/\_typing.py](https://app.codecov.io/gh/juaml/junifer/pull/401?src=pr&el=tree&filepath=junifer%2Ftyping%2F_typing.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml#diff-anVuaWZlci90eXBpbmcvX3R5cGluZy5weQ==) | 0.00% | [1 Missing :warning: ](https://app.codecov.io/gh/juaml/junifer/pull/401?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml) | | [junifer/utils/helpers.py](https://app.codecov.io/gh/juaml/junifer/pull/401?src=pr&el=tree&filepath=junifer%2Futils%2Fhelpers.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml#diff-anVuaWZlci91dGlscy9oZWxwZXJzLnB5) | 0.00% | [1 Missing :warning: ](https://app.codecov.io/gh/juaml/junifer/pull/401?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml) | | [junifer/utils/singleton.py](https://app.codecov.io/gh/juaml/junifer/pull/401?src=pr&el=tree&filepath=junifer%2Futils%2Fsingleton.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml#diff-anVuaWZlci91dGlscy9zaW5nbGV0b24ucHk=) | 0.00% | [1 Missing :warning: ](https://app.codecov.io/gh/juaml/junifer/pull/401?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml) | <details><summary>Additional details and impacted files</summary> [![Impacted file tree graph](https://app.codecov.io/gh/juaml/junifer/pull/401/graphs/tree.svg?width=650&height=150&src=pr&token=5H21JuZXMw&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml)](https://app.codecov.io/gh/juaml/junifer/pull/401?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml) ```diff @@ Coverage Diff @@ ## main #401 +/- ## ======================================== - Coverage 0.01% 0.01% -0.01% ======================================== Files 132 133 +1 Lines 5502 5540 +38 ======================================== Hits 1 1 - Misses 5501 5539 +38 ``` | [Flag](https://app.codecov.io/gh/juaml/junifer/pull/401/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml) | Coverage Δ | | |---|---|---| | [junifer](https://app.codecov.io/gh/juaml/junifer/pull/401/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml) | `0.00% <0.00%> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml#carryforward-flags-in-the-pull-request-comment) to find out more. | [Files with missing lines](https://app.codecov.io/gh/juaml/junifer/pull/401?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml) | Coverage Δ | | |---|---|---| | [junifer/api/decorators.py](https://app.codecov.io/gh/juaml/junifer/pull/401?src=pr&el=tree&filepath=junifer%2Fapi%2Fdecorators.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml#diff-anVuaWZlci9hcGkvZGVjb3JhdG9ycy5weQ==) | `0.00% <ø> (ø)` | | | [junifer/cli/cli.py](https://app.codecov.io/gh/juaml/junifer/pull/401?src=pr&el=tree&filepath=junifer%2Fcli%2Fcli.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml#diff-anVuaWZlci9jbGkvY2xpLnB5) | `0.00% <ø> (ø)` | | | [junifer/data/\_dispatch.py](https://app.codecov.io/gh/juaml/junifer/pull/401?src=pr&el=tree&filepath=junifer%2Fdata%2F_dispatch.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml#diff-anVuaWZlci9kYXRhL19kaXNwYXRjaC5weQ==) | `0.00% <ø> (ø)` | | | [junifer/storage/utils.py](https://app.codecov.io/gh/juaml/junifer/pull/401?src=pr&el=tree&filepath=junifer%2Fstorage%2Futils.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml#diff-anVuaWZlci9zdG9yYWdlL3V0aWxzLnB5) | `0.00% <ø> (ø)` | | | [junifer/testing/datagrabbers.py](https://app.codecov.io/gh/juaml/junifer/pull/401?src=pr&el=tree&filepath=junifer%2Ftesting%2Fdatagrabbers.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml#diff-anVuaWZlci90ZXN0aW5nL2RhdGFncmFiYmVycy5weQ==) | `0.00% <ø> (ø)` | | | [junifer/utils/logging.py](https://app.codecov.io/gh/juaml/junifer/pull/401?src=pr&el=tree&filepath=junifer%2Futils%2Flogging.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml#diff-anVuaWZlci91dGlscy9sb2dnaW5nLnB5) | `0.00% <ø> (ø)` | | | [junifer/api/functions.py](https://app.codecov.io/gh/juaml/junifer/pull/401?src=pr&el=tree&filepath=junifer%2Fapi%2Ffunctions.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml#diff-anVuaWZlci9hcGkvZnVuY3Rpb25zLnB5) | `0.00% <0.00%> (ø)` | | | [junifer/cli/parser.py](https://app.codecov.io/gh/juaml/junifer/pull/401?src=pr&el=tree&filepath=junifer%2Fcli%2Fparser.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml#diff-anVuaWZlci9jbGkvcGFyc2VyLnB5) | `0.00% <0.00%> (ø)` | | | [junifer/data/masks/\_masks.py](https://app.codecov.io/gh/juaml/junifer/pull/401?src=pr&el=tree&filepath=junifer%2Fdata%2Fmasks%2F_masks.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml#diff-anVuaWZlci9kYXRhL21hc2tzL19tYXNrcy5weQ==) | `0.00% <0.00%> (ø)` | | | [junifer/data/template\_spaces.py](https://app.codecov.io/gh/juaml/junifer/pull/401?src=pr&el=tree&filepath=junifer%2Fdata%2Ftemplate_spaces.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml#diff-anVuaWZlci9kYXRhL3RlbXBsYXRlX3NwYWNlcy5weQ==) | `0.00% <0.00%> (ø)` | | | ... and [5 more](https://app.codecov.io/gh/juaml/junifer/pull/401?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml) | | </details>

synchon (Migrated from github.com) approved these changes

2024-11-26 11:43:11 +00:00