juaml/junifer
Watch
3
0
Fork
You've already forked junifer
 0
Code Issues 14 Pull requests 2 Packages Activity Actions

[ENH]: HDF5 please #147

Merged
synchon merged 122 commits from feature/hdf5-storage into main 2023-03-14 21:43:33 +00:00
Conversation 29 Commits 122 Files changed 20 +2237 -74

2023-03-14

Synchon Mandal
update: simplify element index creation for HDF5FeatureStorage.read_df()
bdbd677f29
2023-03-14 22:16:42 +01:00
Synchon Mandal
fix: add missing constraint for dependency check test
b526fa08ea
2023-03-14 18:37:47 +01:00
Synchon Mandal
fix: adapt test for dependency version checks
75207fad53
2023-03-14 16:53:03 +01:00
Synchon Mandal
chore: flake8
cf22bb1a4c
2023-03-14 16:19:39 +01:00
Synchon Mandal
chore: isort
ad6a14a804
2023-03-14 16:17:44 +01:00
Synchon Mandal
docs: increment docs copyright year to 2023
e4414e5f48
2023-03-14 16:17:19 +01:00
Synchon Mandal
chore: update datalad constraints in pyproject.toml
4b96eea1f2
2023-03-14 16:17:03 +01:00
Synchon Mandal
chore: update nilearn constraints in pyproject.toml
8f2786ef80
2023-03-14 16:16:45 +01:00
Synchon Mandal
update: simplify usage of matrix_to_vector() in HDF5FeatureStorage
b3f908e3e7
2023-03-14 16:16:11 +01:00
Synchon Mandal
update: adapt usage of matrix_to_vector() for SQLiteFeatureStorage
e82f734584
2023-03-14 16:15:27 +01:00
Synchon Mandal
update: adapt tests for storage.utils.matrix_to_vector()
324e7fef36
2023-03-14 16:15:07 +01:00
Synchon Mandal
update: let storage.utils.matrix_to_vector() take 2D and 3D data
6a086c226f
2023-03-14 16:14:36 +01:00
Synchon Mandal
chore: codespell fix
4dbd53ab73
2023-03-14 15:03:33 +01:00
Synchon Mandal
update: adapt read_df() tests of matrix for HDF5FeatureStorage
63ed17a54f
2023-03-14 14:59:05 +01:00
Synchon Mandal
update: use matrix_to_vector() for HDF5FeatureStorage.read_df()
cc0e20e240
2023-03-14 14:58:15 +01:00
Synchon Mandal
update: use matrix_to_vector() for SQLiteFeatureStorage.store_matrix()
473fda41c9
2023-03-14 13:52:22 +01:00
Synchon Mandal
update: add tests for function for matrix to vector for storage-like objects
27e2b5f3f4
2023-03-14 13:50:03 +01:00
Synchon Mandal
update: add function for matrix to vector for storage-like objects
36df658506
2023-03-14 13:49:24 +01:00
Synchon Mandal
update: use store_matrix_checks() for SQLiteFeatureStorage.store_matrix()
e88532eb7f
2023-03-14 11:50:16 +01:00
Synchon Mandal
update: use store_matrix_checks() for HDF5FeatureStorage.store_matrix()
1d19bebb92
2023-03-14 10:53:32 +01:00
Synchon Mandal
update: add tests for function for store_matrix() checks for storage-like objects
9d8db1a5a9
2023-03-14 10:21:30 +01:00
Synchon Mandal
update: add function for store_matrix() checks for storage-like objects
357bfeae0d
2023-03-14 10:20:52 +01:00
Synchon Mandal
chore: correct type annotation for matrix_kind in store_matrix() of storage-like objects
d66ad6a725
2023-03-14 10:15:50 +01:00

2023-03-08

Synchon Mandal
chore: lint
418ea74111
2023-03-08 15:12:05 +01:00
Synchon Mandal
update: fix store_matrix tests for HDF5FeatureStorage
4eeaecae3b
2023-03-08 15:11:23 +01:00
Synchon Mandal
update: make matrix as one row per element in HDF5FeatureStorage.read_df()
5353dfad71
2023-03-08 15:11:23 +01:00
Synchon Mandal
chore: simplify parameter check conditional in HDF5FeatureStorage.read_df()
4456fc6290
2023-03-08 15:11:23 +01:00
Synchon Mandal
refactor: reduce nesting level for parameter check in HDF5FeatureStorage.read_df()
784d0fadb9
2023-03-08 15:11:23 +01:00
Synchon Mandal
update: ignore h5io for coverage
ddf0d0e281
2023-03-08 15:11:23 +01:00
Synchon Mandal
update: add pytest config in pyproject.toml and ignore h5io tests
00d3331592
2023-03-08 15:11:23 +01:00
Synchon Mandal
update: add chunk_size as optional constructor parameter for HDF5FeatureStorage
97e56bc42a
2023-03-08 15:11:23 +01:00
Synchon Mandal
update: add h5io as a package from git submodule
7153e47b1d
2023-03-08 15:11:23 +01:00
Synchon Mandal
chore: add h5py as a dependency in pyproject.toml
74f9c0eddf
2023-03-08 15:11:23 +01:00
Synchon Mandal
update: change recursive to true for submodules in ci.yml
dc76e1e153
2023-03-08 15:11:23 +01:00
Synchon Mandal
chore: remove unused strategy in docs.yml
a05efce080
2023-03-08 15:11:23 +01:00
Synchon Mandal
update: add submodules init in docs.yml
6b7aa2d246
2023-03-08 15:11:23 +01:00
Synchon Mandal
chore: file ending with newline for docs-preview.yml
32ed1d09a1
2023-03-08 15:11:23 +01:00
Synchon Mandal
chore: remove unused strategy in docs-preview.yml
d298abbd51
2023-03-08 15:11:23 +01:00
Synchon Mandal
update: add submodules init for docs-preview.yml
dfd16eb639
2023-03-08 15:11:23 +01:00
Synchon Mandal
chore: update ci.yml
6585b8c4d3
2023-03-08 15:11:23 +01:00
Synchon Mandal
update: enable submodule checkout in ci.yml
1714364e17
2023-03-08 15:11:23 +01:00
Synchon Mandal
chore: updated commit checksum for h5io
2d22c7e9f6
2023-03-08 15:11:23 +01:00
Synchon Mandal
update: make 2D data storage work for ChunkedArray in HDF5FeatureStorage
464035536b
2023-03-08 15:11:23 +01:00
Synchon Mandal
update: make element key in storage always store list of dict in HDF5FeatureStorage
35778ce3f1
2023-03-08 15:11:23 +01:00
Synchon Mandal
Revert "fix: bypass loading issues using JSON for encode / decode in HDF5FeatureStorage" 
This reverts commit 118b3a4ed4f0f6e9e0b415665e9753bbc9c76486.
cb977aa79f
2023-03-08 15:11:23 +01:00
Synchon Mandal
fix: bypass loading issues using JSON for encode / decode in HDF5FeatureStorage
42c2f266e7
2023-03-08 15:11:23 +01:00
Synchon Mandal
docs: add HDF5FeatureStorage in storage.rst
63de952995
2023-03-08 15:11:23 +01:00
Synchon Mandal
chore: lint
fcffef6cc9
2023-03-08 15:11:23 +01:00
Synchon Mandal
chore: add junifer/external/h5io to black exclusion list
7489433537
2023-03-08 15:11:23 +01:00
Synchon Mandal
chore: add junifer/external/h5io to codespell exclusion list
382af41429
2023-03-08 15:11:23 +01:00
Synchon Mandal
chore: add junifer/external/h5io to coverage exclusion list
9b36833875
2023-03-08 15:11:23 +01:00
Synchon Mandal
chore: add junifer/external/h5io to flake8 exclusion list
9258da4f37
2023-03-08 15:11:23 +01:00
Synchon Mandal
chore: add junifer/external/h5io to isort exclusion list
2c464290bf
2023-03-08 15:11:23 +01:00
Synchon Mandal
update: stringify Path objects for read_hdf5() and write_hdf5() in HDF5FeatureStorage
8a620ec134
2023-03-08 15:11:23 +01:00
Synchon Mandal
refactor: use two-pass strategy to collect features to avoid memory spikes
1e73b310db
2023-03-08 15:11:23 +01:00
Synchon Mandal
update: replace h5io froom pip with forked h5io
066208f842
2023-03-08 15:11:23 +01:00
Synchon Mandal
chore: update h5io submodule commit
89ebd03c80
2023-03-08 15:11:23 +01:00
Synchon Mandal
update: add forked h5io as a submodule
5754b4dad2
2023-03-08 15:11:23 +01:00
Synchon Mandal
update: concatenate element data along last axis in HDF5FeatureStorage
8444bc9e40
2023-03-08 15:11:23 +01:00
Synchon Mandal
update: make matrix store as (x, y, t) for HDF5FeatureStorage
2fafbbb1f2
2023-03-08 15:11:23 +01:00
Synchon Mandal
update: make vector store as (x, t) for HDF5FeatureStorage
6d92fee48f
2023-03-08 15:11:23 +01:00
Synchon Mandal
update: make timeseries store as (x, y, t) for HDF5FeatureStorage
385a970571
2023-03-08 15:11:23 +01:00
Synchon Mandal
update: add unit tests for HDF5FeatureStorage data type casting
c81e586588
2023-03-08 15:11:23 +01:00
Synchon Mandal
update: add option for converting float64 to float32 for data storage via HDF5FeatureStorage
7c15628926
2023-03-08 15:11:23 +01:00
Synchon Mandal
chore: update latest.inc
d793fd9fff
2023-03-08 15:11:20 +01:00
Synchon Mandal
update: add more unit tests for HDF5FeatureStorage
ab076d2ab4
2023-03-08 15:08:26 +01:00
Synchon Mandal
update: improve HDF5FeatureStorage unit tests
3609531463
2023-03-08 15:08:26 +01:00
Synchon Mandal
fix: use column length for reshaping matrix in HDF5FeatureStorage.read_df()
98badc1d04
2023-03-08 15:08:26 +01:00
Synchon Mandal
chore: update docstring for HDF5FeatureStorage.store_matrix()
1a6f4d9b36
2023-03-08 15:08:26 +01:00
Synchon Mandal
update: add matrix_kind validation for HDF5FeatureStorage.store_matrix()
633afe1a56
2023-03-08 15:08:26 +01:00
Synchon Mandal
chore: improve logging messages for HDF5FeatureStorage
d2e1aaedd2
2023-03-08 15:08:26 +01:00
Synchon Mandal
chore: add static type checking ignore
f0a315d446
2023-03-08 15:08:26 +01:00
Synchon Mandal
update: make HDF5FeatureStorage.read_df() read data directly
8b9737fc67
2023-03-08 15:08:26 +01:00
Synchon Mandal
update: make HDF5FeatureStorage.read_df() read metadata directly
37345cc1f0
2023-03-08 15:08:26 +01:00
Synchon Mandal
update: make HDF5FeatureStorage.list_features() read metadata directly
712abc468d
2023-03-08 15:08:26 +01:00
Synchon Mandal
update: refine HDF5FeatureStorage.collect()
e62b3008c0
2023-03-08 15:08:26 +01:00
Synchon Mandal
update: add duplicate element store check in HDF5FeatureStorage._store_data()
394a685206
2023-03-08 15:08:26 +01:00
Synchon Mandal
update: make element param a list of dict for HDF5FeatureStorage._store_data()
606e32639a
2023-03-08 15:08:26 +01:00
Synchon Mandal
update: refine HDF5FeatureStorage.read_df()
b5173db351
2023-03-08 15:08:26 +01:00
Synchon Mandal
update: remove HDF5FeatureStorage._index_dict_to_multiindex()
e7cdafa336
2023-03-08 15:08:26 +01:00
Synchon Mandal
update: remove HDF5FeatureStorage._element_metadata_to_index_dict()
b474644a80
2023-03-08 15:08:26 +01:00
Synchon Mandal
chore: black formatting for HDF5FeatureStorage._store_data()
a4fdbfd4fa
2023-03-08 15:08:26 +01:00
Synchon Mandal
update: add row_header_col_name parameter for HDF5FeatureStorage.store_matrix()
2bb430bb27
2023-03-08 15:08:26 +01:00
Synchon Mandal
update: rename row_headers to row_header_column_name in HDF5FeatureStorage.store_timeseries()
9a233be6c9
2023-03-08 15:08:26 +01:00
Synchon Mandal
chore: improve logging messages for HDF5FeatureStorage
35c779aaef
2023-03-08 15:08:26 +01:00
Synchon Mandal
chore: update docstrings for HDF5FeatureStorage
44b2d59d3c
2023-03-08 15:08:26 +01:00
Synchon Mandal
chore: improve error messages for HDF5FeatureStorage
d8ab8ca70e
2023-03-08 15:08:26 +01:00
Synchon Mandal
update: add kwargs validation check for HDF5FeatureStorage._store_data()
52a240b5ba
2023-03-08 15:08:26 +01:00
Synchon Mandal
chore: update type annotation for HDF5FeatureStorage._read_data()
e6a1c50b42
2023-03-08 15:08:26 +01:00
Synchon Mandal
update: refine HDF5FeatureStorage.store_timeseries()
59063bd8da
2023-03-08 15:08:26 +01:00
Synchon Mandal
update: refine HDF5FeatureStorage.store_vector()
d8e94af640
2023-03-08 15:08:26 +01:00
Synchon Mandal
update: refine HDF5FeatureStorage.store_matrix()
0ce9faba9d
2023-03-08 15:08:26 +01:00
Synchon Mandal
update: refine HDF5FeatureStorage._store_data()
fb20576c2e
2023-03-08 15:08:26 +01:00
Synchon Mandal
update: handle data group reading failure in HDF5FeatureStorage._read_data()
ac6c28d711
2023-03-08 15:08:26 +01:00
Synchon Mandal
update: handle "meta" group reading failure in HDF5FeatureStorage._read_metadata()
0a43aae401
2023-03-08 15:08:26 +01:00
Synchon Mandal
update: improve docstring of HDF5FeatureStorage.store_timeseries()
6c6851ca58
2023-03-08 15:08:26 +01:00
Synchon Mandal
update: refine HDF5FeatureStorage.read_df()
fea2fe5452
2023-03-08 15:08:26 +01:00
Synchon Mandal
chore: update type annotations for HDF5FeatureStorage.store_metadata()
5dc39f5dfe
2023-03-08 15:08:26 +01:00
Synchon Mandal
update: refine HDF5FeatureStorage.store_metadata() and avoid duplicate MD5 writes
3c36bd1ba2
2023-03-08 15:08:26 +01:00
Synchon Mandal
update: refine HDF5FeatureStorage.list_features()
d3ab6119bc
2023-03-08 15:08:26 +01:00
Synchon Mandal
update: make metadata dict of dict in HDF5FeatureStorage._read_metadata()
65b5af9763
2023-03-08 15:08:26 +01:00
Synchon Mandal
update: change remaining declarations of table to vector in HDF5FeatureStorage
58b2b3b248
2023-03-08 15:08:26 +01:00
Synchon Mandal
update: remove "meta_md5" storing from HDF5FeatureStorage.store_metadata()
661c0f1ef0
2023-03-08 15:08:26 +01:00
Synchon Mandal
update: remove element param from HDF5FeatureStorage.read_df()
83610a5767
2023-03-08 15:08:26 +01:00
Synchon Mandal
update: rename columns to col_names in HDF5FeatureStorage.store_timeseries()
48fd2a36b3
2023-03-08 15:08:26 +01:00
Synchon Mandal
update: replace columns and rows_col_name with col_names in HDF5FeatureStorage.store_vector()
6ad01d2c13
2023-03-08 15:08:26 +01:00
Synchon Mandal
update: rename table to vector in HDF5FeatureStorage
fb7815d64e
2023-03-08 15:08:26 +01:00
Synchon Mandal
chore: improve docstring for HDF5FeatureStorage.store_matrix()
d29a0b534e
2023-03-08 15:08:26 +01:00
Synchon Mandal
chore: add HDF5FeatureStorage ref in "See Also" of SQLiteFeatureStorage
22253ae073
2023-03-08 15:08:26 +01:00
Synchon Mandal
update: remove _collected flag and refine correct HDF5 URI fetching for HDF5FeatureStorage
2413d9d5d8
2023-03-08 15:08:26 +01:00
Synchon Mandal
chore: update docstring for HDF5FeatureStorage.list_features()
ee24c9b3ff
2023-03-08 15:08:26 +01:00
Synchon Mandal
update: remove element param from HDF5FeatureStorage.list_features()
d0763421cf
2023-03-08 15:08:26 +01:00
Synchon Mandal
update: add more tests for HDF5FeatureStorage
e5abad916e
2023-03-08 15:08:26 +01:00
Synchon Mandal
fix: remove sphinx directives for private methods in HDF5FeatureStorage
ccec087fce
2023-03-08 15:08:26 +01:00
Synchon Mandal
update: add tests for HDF5FeatureStorage
a583f388a3
2023-03-08 15:08:26 +01:00
Synchon Mandal
update: append metadata for multi output HDF5FeatureStorage.collect()
1fed4527a8
2023-03-08 15:08:26 +01:00
Synchon Mandal
update: improve data and metadata read for multi output HDF5FeatureStorage after collect()
2eef5367bd
2023-03-08 15:08:26 +01:00
Synchon Mandal
update: add HDF5FeatureStorage import in storage.__init__
f7e2615705
2023-03-08 15:08:26 +01:00
Synchon Mandal
update: add HDF5FeatureStorage implementation
dd60bc8a63
2023-03-08 15:08:26 +01:00
Synchon Mandal
update: add h5io as dependency in pyproject.toml
31f75157b1
2023-03-08 15:08:26 +01:00