Relearn v9.5 requires hugo 0.166.0 or newer. We shall bump to latest available as of today, which is 0.169.0. Hugo deprecated `languageCode` (now called `locale`) and `_build` (now called `build`). The relearn theme deprecated the `notice` shortcode in favour of the new `callout` shortcode. The new version allows using Markdown callouts, which are IMO nicer.
207 lines
9.8 KiB
Markdown
207 lines
9.8 KiB
Markdown
+++
|
|
date = '2025-02-12T18:01:59+01:00'
|
|
draft = false
|
|
title = 'Accessing motor assessment data collection'
|
|
weight = 4
|
|
+++
|
|
|
|
This note will explain in detail how to set-up your local copy of the aggregated motor assessment data, and help deal with potential configuration-dependent obstacles.
|
|
|
|
Overview of the encrypted workflow and prerequisites are explained in the [encrypted workflow](z3-encrypted-workflow/) page. It is highly recommended that you read it first.
|
|
|
|
The note assumes that you have access to a shared copy of the dataset, and the data is being encrypted to your GPG key.
|
|
|
|
## Distributed storage
|
|
|
|
There are two components to the dataset:
|
|
|
|
- The Git repository (which we can call a "lightweight dataset") contains information about content identity and availability, but does not contain the actual data.
|
|
- The content managed by git-annex ("annexed data"), i.e. the actual data.
|
|
|
|
Each can be stored in / retrieved from multiple locations.
|
|
Depending on the platform used, the Git repository and annexed data can be stored and shared together (as is the case with Cerebra) or separately (JuGit + Sciebo).
|
|
The chart below illustrates the Git and git-annex repositories which can be configured as "remotes" for a local clone.
|
|
In the chart below, a square box symbolizes a Git remote, and cylinders symbolize Git-annex special remotes.
|
|
Retrieving data requires interacting with both kinds of remotes, and the main operations are shown over the lines.
|
|
|
|
```mermaid
|
|
flowchart TD
|
|
A(Your local dataset clone)
|
|
AA["**Cerebra** (restricted)"]
|
|
AB[("**Cerebra-mask** (restricted, encrypted)")]
|
|
subgraph Deprecated
|
|
B["**JuGit** (public)"]
|
|
C[("**Sciebo** (restricted, encrypted)")]
|
|
end
|
|
D[("**INM-7 storage** (restricted, encrypted)")]
|
|
AA --- AB
|
|
A ---o |clone, update| AA
|
|
A ---o |enableremote, get| AB
|
|
A ---o |clone, update| B
|
|
A ---o |initremote, get| C
|
|
A ---o |"🚫"| D
|
|
```
|
|
|
|
Note that Cerebra is a special case: it could store the Git repository and its annex in a single remote. However, using encryption requires us to set up a [mask special remote](https://git-annex.branchable.com/special_remotes/mask/) for the annex, which shows separately in the list of annex remotes, but uses the same transport (think: the same credentials) as the Git remote.
|
|
|
|
Because no data are stored in JuGit, the repository can be made public, and fetching information from JuGit requires no credentials.
|
|
However, access to the data stored in Sciebo or Cerebra-mask requires not only credentials, but also possession of a GPG key for which the data are encrypted.
|
|
Note that the Git repository also indexes data stored in INM7-storage, but that special remote is not accessible outside INM-7.
|
|
|
|
The JuGit + Sciebo setup continues to be maintained for a transition period, but will be phased out in favour of Cerebra.
|
|
|
|
|
|
## Before you begin
|
|
|
|
### Cerebra token
|
|
|
|
Instead of using your password (the one you use to log in to Cerebra), for DataLad (Git, git-annex) operations you should be using a token. Tokens are like passwords, but they are generated by the website rather than created by the user. A token can be revoked when it is no longer used or there is a risk that it got exposed.
|
|
|
|
To generate a token:
|
|
|
|
- Visit Cerebra and log in.
|
|
- Click your avatar in the top-right corner, and navigate to Settings → Applications.
|
|
- Under *Access tokens* click *New access token* and fill in the form:
|
|
- give the token a recognizable name;
|
|
- choose either "All" or "Specific repositories" for *Repository and organization access* (if you choose the latter, don't forget to select the repositories);
|
|
- under *Select permissions*, enable "repository: Read and write";
|
|
- click *Generate token*.
|
|
- The token appears as a long string. Copy the generated value, and store it safely. It disappears after refreshing.
|
|
|
|
Treat your token like a password. Do not share it with anyone. Once generated, the token can be used until revoked.
|
|
|
|
### Git credential helper
|
|
|
|
To avoid having to enter the token multiple times, we recommend using a Git credential helper.
|
|
|
|
The easiest one to configure is [git-credential-cache](https://git-scm.com/docs/git-credential-cache), which temporarily (default timeout: 15 minutes) stores passwords in memory. It can be made default by setting a global Git configuration):
|
|
|
|
```
|
|
git config --global credential.helper cache
|
|
```
|
|
|
|
You may also configure a credential helper with persistent storage, for example one which interacts with your preferred password manager. You can also use OAuth (redirects to browser login and creates a temporary bearer token). Credential management is a matter of personal taste, so we can not document all possibilities here. Instead, see Git documentation about [credential helpers](https://git-scm.com/doc/credential-helpers) for a list of available options [^3].
|
|
|
|
### GPG (Windows)
|
|
|
|
> [!info]
|
|
> This section is Windows-specific, and assumes you are using cmd shell.
|
|
>
|
|
> On GNU/Linux (and likely MacOS), no additional configuration is needed.
|
|
|
|
If you install Git (which is a requirement for using DataLad), it comes with its own `gpg` executable. If you install gpg4win (which is the solution recommended for digital signatures in our docs on [Input signing](digital-signatures) it naturally comes with a separate `gpg` executable. This means it is possible to have two GPG programs... which store their data (e.g. keys) in completely different locations on your computer.
|
|
|
|
What's more, in this case, git-annex operations will default to using the one which came with Git, while user operations (be it via Kleopatra GUI or the command line) will more than likely[^1] use the one which came with gpg4win. This can become confusing. Fortunately, git-annex behaviour is configurable. The goal here is to have both the user and git-annex interact with the same `gpg` executable.
|
|
|
|
If you do not have gpg4win: you probably do not need to do anything.
|
|
|
|
If you have gpg4win and used it to generate (or import) your private key, you need to configure Git and git-annex to use this one.
|
|
|
|
First, find out the path to the gpg executable. The path components would tell you whether it came with Git or gpg4win - the latter is more likely.
|
|
|
|
```
|
|
where gpg
|
|
```
|
|
|
|
Then (if it is gpg4win) tell Git to use this one. Git-annex obeys the same setting. Replace `PATH/TO/GPG` with the path reported by the previous command.
|
|
|
|
```
|
|
git config gpg.program PATH/TO/GPG
|
|
```
|
|
|
|
## Initial setup
|
|
|
|
### Clone the dataset
|
|
|
|
Clone the (lightweight) dataset from Cerebra (using the URL below) into a directory with a name of your choice
|
|
(here: `motor_export`):
|
|
|
|
```
|
|
datalad clone https://cerebra.fz-juelich.de/sfb1451-z03-internal/motor-assessment motor_export
|
|
```
|
|
|
|
Then change your working directory to go into the dataset:
|
|
```
|
|
cd motor_export
|
|
```
|
|
|
|
### Enable the mask special remote
|
|
|
|
Git-annex special remotes represent methods of transporting data. In this particular case, a *mask special remote* was set up to store encrypted data alongside the repository in Cerebra. Enabling it (and transferring data) requires having a matching GPG key. For this reason, it is not enabled automatically.
|
|
|
|
Optionally, you can start by checking the list of available special remotes.
|
|
Each special remote has its own ID and assigned name, which can be used interchangeably inside git-annex commands. To list remotes known to the dataset, run:
|
|
|
|
```
|
|
git annex info
|
|
```
|
|
|
|
Pending future changes, the remote we want to enable is going to be named cerebra-storage,
|
|
and it should appear in the list of *semitrusted repositories* like this:
|
|
|
|
```
|
|
f095040a-18b7-4648-849e-f154c53ce976 -- cerebra-mask
|
|
```
|
|
|
|
To enable the cerebra-mask special remote, run:
|
|
|
|
```
|
|
git annex enableremote cerebra-mask
|
|
```
|
|
|
|
You should see the following output:
|
|
|
|
``` output
|
|
enableremote cerebra-mask (encryption update) (to gpg keys: [abbreviated]) ok
|
|
(recording state in git...)
|
|
```
|
|
|
|
If that didn't work, see the *Troubleshooting* section below.
|
|
|
|
## Data access
|
|
|
|
> [!note]
|
|
> This assumes your working directory is in the dataset (that you `cd`'ed into the dataset).
|
|
|
|
To download data, use:
|
|
|
|
```
|
|
datalad get output
|
|
```
|
|
|
|
To remove your local copy of the data, use `drop` which does the opposite of `get`:
|
|
|
|
```
|
|
datalad drop output
|
|
```
|
|
|
|
To obtain the latest version of the data, use:
|
|
|
|
```
|
|
datalad update --how merge
|
|
datalad get output
|
|
```
|
|
|
|
To see when the data (according to your local state) last changed, use `git log` (up/down arrows scroll, `Q` quits):
|
|
|
|
```
|
|
git log
|
|
```
|
|
|
|
|
|
## Troubleshooting
|
|
|
|
Operations which use your secret key (like downloading encrypted data with git-annex or listing your secret keys with `gpg --list-secret-keys`) require interactions with a program called `gpg-agent` [^2]. Normally, `gpg-agent` should be running in the background, and there is nothing to do.
|
|
|
|
If you notice gpg-agent is not running (e.g. `enableremote` or `get` fails with an error message which says that gpg-agent is not running, or you can not list your secret keys), you can start the gpg-agent with:
|
|
|
|
```
|
|
gpg-connect-agent reloadagent /bye
|
|
```
|
|
|
|
If an operation (e.g. `enableremote` or `get`) fails with an error message which says that a private key is not available (but you know from running `gpg --list-secret-keys` that your private key is configured), or the message says gpg-agent is not running (but you know it is) it is possible that you have more than one gpg program installed, and Git-annex uses a different one that you would expect. See the *GPG (Windows)* part of the *Before you begin* section above.
|
|
|
|
|
|
[^1]: It would, at least in theory, be possible to modify the [PATH variable](https://en.wikipedia.org/wiki/PATH_(variable)) so that running `gpg` would use the Git-bundled gpg instead of the gpg4win gpg.
|
|
[^2]: You call `datalad get`, DataLad calls `git annex get`; git-annex calls `gpg --decrypt`, and gpg asks gpg-agent for a matching secret key.
|
|
[^3]: For a more technical introduction to what Git credential helpers do, see this blog post: [Git The Stupid Password Store](https://tylercipriani.com/blog/2024/07/31/git-as-a-password-prompt/)
|