Add new doc section DataLad-based deposition #2

Closed
opened 2023-09-26 10:53:37 +00:00 by Ghost · 5 comments
Ghost commented 2023-09-26 10:53:37 +00:00 (Migrated from hub.psychoinformatics.de)

These instructions would be for users who have adopted DataLad in their sfb1451 data workflows. There are two concrete scenarios:

  1. They have DataLad datasets that they could share (with the SFB audience, via sciebo or elsewhere): How can they annotate self with a tabby record, such that we can clone the dataset and harvest that record (plus all the basic file info).
  2. They have DataLad datasets that they CANNOT share: How can they generate tabby records in the most efficient way that can then be put into the respective project superdataset (by them or by us).
These instructions would be for users who have adopted DataLad in their sfb1451 data workflows. There are two concrete scenarios: 1. They have DataLad datasets that they could share (with the SFB audience, via sciebo or elsewhere): How can they annotate `self` with a `tabby` record, such that we can clone the dataset and harvest that record (plus all the basic file info). 2. They have DataLad datasets that they CANNOT share: How can they generate `tabby` records in the most efficient way that can then be put into the respective project superdataset (by them or by us).
Ghost commented 2023-09-26 10:54:48 +00:00 (Migrated from hub.psychoinformatics.de)

The content that we want to add seem to me to fall nicely under Submitting a record -> DataLad-based deposition (here: https://rdm.sfb1451.de/data-catalog-submission/#submitting-a-record) so i think it would make sense to add DataLad-based deposition as the new section underneath Record components.

The content that we want to add seem to me to fall nicely under `Submitting a record -> DataLad-based deposition` (here: https://rdm.sfb1451.de/data-catalog-submission/#submitting-a-record) so i think it would make sense to add `DataLad-based deposition` as the new section underneath `Record components.`
Ghost commented 2023-09-26 12:09:12 +00:00 (Migrated from hub.psychoinformatics.de)

A more concrete suggestion for the structure of the additional docs:

DataLad-based deposition

Scenario 1: A shareable DataLad dataset

  1. Create a tabby record (as described in the dedicated docs section: https://rdm.sfb1451.de/data-catalog-submission/#creating-a-record)
  2. Save the record as .datalad/tabby/self/dataset.tsv, this location is relative to the root of the DataLad dataset that is being described. (should any mention be made of conventions?)
  3. Any other tabby records that should be reported but do not relate directly to the DataLad dataset being described can be saved in the .datalad/tabby/collection/ location relative to the root of the DataLad dataset.
  4. Save the dataset: datalad save -m "<save message>"
  5. Push the new addition to Sciebo: datalad push --to sciebo. This assumes that the user is familiar with and has followed the data publication procedure outlined in https://rdm.sfb1451.de/publication-catalog/ for the DataLad dataset under consideration.

Scenario 2: A NON-shareable DataLad dataset

  1. For a maintainable structure and keeping the complete dataset, including metadata, under version control, users can still be encouraged to create a tabby record and save it in the same location as described in Scenario 1 steps 1 through 3.
  2. Optional: to create the files table (see: https://rdm.sfb1451.de/data-catalog-submission/#table-files), users can run the datalad-tabby/tools/dir2filetable.py script provided in the datalad-tabby repository:
    python '<path-to-datalad-tabby>/tools/dir2filetable.py' --output '<path-to-datalad-dataset>/.datalad/tabby/self' '<path-to-datalad-dataset>'
    
  3. Copy and send all tabby records (i.e. TSV-files) to INF project coordinator. Alternatively, add these tabby records to the relevant project (should we add a dedicated subdirectory, e.g. tabby?) in the SFB1451 project box on Sciebo (see: https://rdm.sfb1451.de/publication-catalog/#preliminary-sciebo-access) and/or follow the data publication procedure outlined here https://rdm.sfb1451.de/publication-catalog/ if publishing via DataLad.
  4. Save the DataLad dataset after adding all tabby records: datalad save -m "<save message>" (this would be for the users' own benefit and future use)

Additional options

Subdataset linkage

  • something about .datalad/tabby/collection?

...

A more concrete suggestion for the structure of the additional docs: ## DataLad-based deposition ### Scenario 1: A shareable DataLad dataset 1. Create a tabby record (as described in the dedicated docs section: https://rdm.sfb1451.de/data-catalog-submission/#creating-a-record) 2. Save the record as `.datalad/tabby/self/dataset.tsv`, this location is relative to the root of the DataLad dataset that is being described. _(should any mention be made of conventions?)_ 3. Any other tabby records that should be reported but do not relate directly to the DataLad dataset being described can be saved in the `.datalad/tabby/collection/` location relative to the root of the DataLad dataset. 4. Save the dataset: `datalad save -m "<save message>"` 5. Push the new addition to Sciebo: `datalad push --to sciebo`. This assumes that the user is familiar with and has followed the data publication procedure outlined in https://rdm.sfb1451.de/publication-catalog/ for the DataLad dataset under consideration. ### Scenario 2: A NON-shareable DataLad dataset 1. For a maintainable structure and keeping the complete dataset, including metadata, under version control, users can still be encouraged to create a tabby record and save it in the same location as described in **Scenario 1** steps 1 through 3. 2. Optional: to create the `files` table (see: https://rdm.sfb1451.de/data-catalog-submission/#table-files), users can run the `datalad-tabby/tools/dir2filetable.py` script provided in the `datalad-tabby` repository: ``` python '<path-to-datalad-tabby>/tools/dir2filetable.py' --output '<path-to-datalad-dataset>/.datalad/tabby/self' '<path-to-datalad-dataset>' ``` 3. Copy and send all tabby records (i.e. TSV-files) to INF project coordinator. Alternatively, add these tabby records to the relevant project (_should we add a dedicated subdirectory, e.g. `tabby`?_) in the SFB1451 project box on Sciebo (see: https://rdm.sfb1451.de/publication-catalog/#preliminary-sciebo-access) and/or follow the data publication procedure outlined here https://rdm.sfb1451.de/publication-catalog/ if publishing via DataLad. 4. Save the DataLad dataset after adding all tabby records: `datalad save -m "<save message>"` (this would be for the users' own benefit and future use) ### Additional options #### Subdataset linkage - something about `.datalad/tabby/collection`? #### ...
Ghost commented 2023-09-26 15:05:27 +00:00 (Migrated from hub.psychoinformatics.de)

Re place: logically it makes sense, the only reason why we may consider a separate page is to avoid the page with non-DataLad deposition looking scarily long.

I like the suggestion, I think it is concise and contains the relevant message. A couple of comments below.

Scenario 1

  1. Save the record as .datalad/tabby/self/dataset.tsv

That's a bit of a shortcut - as it stands, for our SFB goals we require dataset, data controller, authors & funding. Unless going straight for json, these would have to be four tsv files.

(should any mention be made of conventions?)

Good Q - when saving files sent to me by e-mail I add a convention name (I even have a script to do that). A convention defines json-ld context but is also used to declare which files should be loaded (added to json structure) when reading dataset.tsv. I have my convention here

  1. Push the new addition to Sciebo

FYI, that's not documented / not communicated broadly, but I finally mirrored all Sciebo datasets to GitHub. Accidentally made GitHub repos private, too...

Scenario 2

  1. Optional: to create the files table (see: https://rdm.sfb1451.de/data-catalog-submission/#table-files), users can run the datalad-tabby/tools/dir2filetable.py script provided in the datalad-tabby repository

I'll need to try that on a DataLad dataset, but seeing that the script contains relpath.glob('*') I'm concerned it may try to list .git as well. Maybe git ls-files or git ls-tree or something based on iter_gitworktree would be better? In any case, the suggestion is good, but the script may need to change.

Re place: logically it makes sense, the only reason why we *may* consider a separate page is to avoid the page with non-DataLad deposition looking scarily long. I like the suggestion, I think it is concise and contains the relevant message. A couple of comments below. ### Scenario 1 > 2. Save the record as `.datalad/tabby/self/dataset.tsv` That's a bit of a shortcut - as it stands, for our SFB goals we require dataset, data controller, authors & funding. Unless going straight for json, these would have to be four tsv files. > *(should any mention be made of conventions?)* Good Q - when saving files sent to me by e-mail I add a convention name (I even have a script to do that). A convention defines json-ld context but is also used to declare which files should be loaded (added to json structure) when reading dataset.tsv. I have my convention [here](https://github.com/mslw/tabby-utils/tree/main/conventions/tby-crc1451v0) > 5. Push the new addition to Sciebo FYI, that's not documented / not communicated broadly, but I finally mirrored all Sciebo datasets to GitHub. Accidentally made GitHub repos private, too... ### Scenario 2 > 2. Optional: to create the `files` table (see: https://rdm.sfb1451.de/data-catalog-submission/#table-files), users can run the `datalad-tabby/tools/dir2filetable.py` script provided in the `datalad-tabby` repository I'll need to try that on a DataLad dataset, but seeing that the script contains `relpath.glob('*')` I'm concerned it may try to list `.git` as well. Maybe `git ls-files` or `git ls-tree` or something based on [iter_gitworktree](https://github.com/datalad/datalad-next/blob/main/datalad_next/iter_collections/gitworktree.py) would be better? In any case, the suggestion is good, but the script may need to change.
Ghost commented 2023-09-26 18:33:22 +00:00 (Migrated from hub.psychoinformatics.de)

Re place: logically it makes sense, the only reason why we may consider a separate page is to avoid the page with non-DataLad deposition looking scarily long.

I like this point, and I think it's a good idea to separate the content.

  1. Save the record as .datalad/tabby/self/dataset.tsv

That's a bit of a shortcut - as it stands, for our SFB goals we require dataset, data controller, authors & funding. Unless going straight for json, these would have to be four tsv files.

Indeed, clearer to specify all the TSV files that need to form part of the deposited record.

  1. Optional: to create the files table (see: https://rdm.sfb1451.de/data-catalog-submission/#table-files), users can run the datalad-tabby/tools/dir2filetable.py script provided in the datalad-tabby repository

I'll need to try that on a DataLad dataset, but seeing that the script contains relpath.glob('*') I'm concerned it may try to list .git as well. Maybe git ls-files or git ls-tree or something based on iter_gitworktree would be better? In any case, the suggestion is good, but the script may need to change.

The context behind datalad-tabby/tools/dir2filetable.py is here https://github.com/psychoinformatics-de/datalad-tabby/issues/33 and here https://github.com/psychoinformatics-de/datalad-tabby/pull/60. So indeed we could opt for a different solution given that the users at which these docs are targeted would already be using datalad. Alternatively the script could be updated to exclude hidden files by default, and include them with a flag (this would be a good addition to the script in any case).

> Re place: logically it makes sense, the only reason why we *may* consider a separate page is to avoid the page with non-DataLad deposition looking scarily long. I like this point, and I think it's a good idea to separate the content. >> 2. Save the record as `.datalad/tabby/self/dataset.tsv` > That's a bit of a shortcut - as it stands, for our SFB goals we require dataset, data controller, authors & funding. Unless going straight for json, these would have to be four tsv files. Indeed, clearer to specify all the TSV files that need to form part of the deposited record. >> 2. Optional: to create the `files` table (see: https://rdm.sfb1451.de/data-catalog-submission/#table-files), users can run the `datalad-tabby/tools/dir2filetable.py` script provided in the `datalad-tabby` repository > I'll need to try that on a DataLad dataset, but seeing that the script contains `relpath.glob('*')` I'm concerned it may try to list `.git` as well. Maybe `git ls-files` or `git ls-tree` or something based on [iter_gitworktree](https://github.com/datalad/datalad-next/blob/main/datalad_next/iter_collections/gitworktree.py) would be better? In any case, the suggestion is good, but the script may need to change. The context behind `datalad-tabby/tools/dir2filetable.py` is here https://github.com/psychoinformatics-de/datalad-tabby/issues/33 and here https://github.com/psychoinformatics-de/datalad-tabby/pull/60. So indeed we could opt for a different solution given that the users at which these docs are targeted would already be using datalad. Alternatively the script could be updated to exclude hidden files by default, and include them with a flag (this would be a good addition to the script in any case).
Ghost commented 2024-05-06 18:34:54 +00:00 (Migrated from hub.psychoinformatics.de)

This has long been closed by !27

This has long been closed by !27
Sign in to join this conversation.
No labels
No milestone
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
sfb1451-inf/rdm.sfb1451.de#2
No description provided.