rdm.sfb1451.de/content/z03/access-data.md
Michał Szczepanik 5f2fd4547f Bump Hugo version; act on deprecations
Relearn v9.5 requires hugo 0.166.0 or newer. We shall bump to latest
available as of today, which is 0.169.0.

Hugo deprecated `languageCode` (now called `locale`) and `_build` (now
called `build`).

The relearn theme deprecated the `notice` shortcode in favour of the
new `callout` shortcode. The new version allows using Markdown callouts,
which are IMO nicer.
2026-11-13 16:59:34 +01:00

9.8 KiB

+++ date = '2025-02-12T18:01:59+01:00' draft = false title = 'Accessing motor assessment data collection' weight = 4 +++

This note will explain in detail how to set-up your local copy of the aggregated motor assessment data, and help deal with potential configuration-dependent obstacles.

Overview of the encrypted workflow and prerequisites are explained in the encrypted workflow page. It is highly recommended that you read it first.

The note assumes that you have access to a shared copy of the dataset, and the data is being encrypted to your GPG key.

Distributed storage

There are two components to the dataset:

  • The Git repository (which we can call a "lightweight dataset") contains information about content identity and availability, but does not contain the actual data.
  • The content managed by git-annex ("annexed data"), i.e. the actual data.

Each can be stored in / retrieved from multiple locations. Depending on the platform used, the Git repository and annexed data can be stored and shared together (as is the case with Cerebra) or separately (JuGit + Sciebo). The chart below illustrates the Git and git-annex repositories which can be configured as "remotes" for a local clone. In the chart below, a square box symbolizes a Git remote, and cylinders symbolize Git-annex special remotes. Retrieving data requires interacting with both kinds of remotes, and the main operations are shown over the lines.

flowchart TD
    A(Your local dataset clone)
    AA["**Cerebra** (restricted)"]
    AB[("**Cerebra-mask** (restricted, encrypted)")]
    subgraph Deprecated
      B["**JuGit** (public)"]
      C[("**Sciebo** (restricted, encrypted)")]
    end
    D[("**INM-7 storage** (restricted, encrypted)")]
    AA --- AB
    A ---o |clone, update| AA
    A ---o |enableremote, get| AB
    A ---o |clone, update| B
    A ---o |initremote, get| C
    A ---o |"🚫"| D

Note that Cerebra is a special case: it could store the Git repository and its annex in a single remote. However, using encryption requires us to set up a mask special remote for the annex, which shows separately in the list of annex remotes, but uses the same transport (think: the same credentials) as the Git remote.

Because no data are stored in JuGit, the repository can be made public, and fetching information from JuGit requires no credentials. However, access to the data stored in Sciebo or Cerebra-mask requires not only credentials, but also possession of a GPG key for which the data are encrypted. Note that the Git repository also indexes data stored in INM7-storage, but that special remote is not accessible outside INM-7.

The JuGit + Sciebo setup continues to be maintained for a transition period, but will be phased out in favour of Cerebra.

Before you begin

Cerebra token

Instead of using your password (the one you use to log in to Cerebra), for DataLad (Git, git-annex) operations you should be using a token. Tokens are like passwords, but they are generated by the website rather than created by the user. A token can be revoked when it is no longer used or there is a risk that it got exposed.

To generate a token:

  • Visit Cerebra and log in.
  • Click your avatar in the top-right corner, and navigate to Settings → Applications.
  • Under Access tokens click New access token and fill in the form:
    • give the token a recognizable name;
    • choose either "All" or "Specific repositories" for Repository and organization access (if you choose the latter, don't forget to select the repositories);
    • under Select permissions, enable "repository: Read and write";
    • click Generate token.
  • The token appears as a long string. Copy the generated value, and store it safely. It disappears after refreshing.

Treat your token like a password. Do not share it with anyone. Once generated, the token can be used until revoked.

Git credential helper

To avoid having to enter the token multiple times, we recommend using a Git credential helper.

The easiest one to configure is git-credential-cache, which temporarily (default timeout: 15 minutes) stores passwords in memory. It can be made default by setting a global Git configuration):

git config --global credential.helper cache

You may also configure a credential helper with persistent storage, for example one which interacts with your preferred password manager. You can also use OAuth (redirects to browser login and creates a temporary bearer token). Credential management is a matter of personal taste, so we can not document all possibilities here. Instead, see Git documentation about credential helpers for a list of available options 1.

GPG (Windows)

[!info] This section is Windows-specific, and assumes you are using cmd shell.

On GNU/Linux (and likely MacOS), no additional configuration is needed.

If you install Git (which is a requirement for using DataLad), it comes with its own gpg executable. If you install gpg4win (which is the solution recommended for digital signatures in our docs on Input signing it naturally comes with a separate gpg executable. This means it is possible to have two GPG programs... which store their data (e.g. keys) in completely different locations on your computer.

What's more, in this case, git-annex operations will default to using the one which came with Git, while user operations (be it via Kleopatra GUI or the command line) will more than likely2 use the one which came with gpg4win. This can become confusing. Fortunately, git-annex behaviour is configurable. The goal here is to have both the user and git-annex interact with the same gpg executable.

If you do not have gpg4win: you probably do not need to do anything.

If you have gpg4win and used it to generate (or import) your private key, you need to configure Git and git-annex to use this one.

First, find out the path to the gpg executable. The path components would tell you whether it came with Git or gpg4win - the latter is more likely.

where gpg

Then (if it is gpg4win) tell Git to use this one. Git-annex obeys the same setting. Replace PATH/TO/GPG with the path reported by the previous command.

git config gpg.program PATH/TO/GPG

Initial setup

Clone the dataset

Clone the (lightweight) dataset from Cerebra (using the URL below) into a directory with a name of your choice (here: motor_export):

datalad clone https://cerebra.fz-juelich.de/sfb1451-z03-internal/motor-assessment motor_export

Then change your working directory to go into the dataset:

cd motor_export

Enable the mask special remote

Git-annex special remotes represent methods of transporting data. In this particular case, a mask special remote was set up to store encrypted data alongside the repository in Cerebra. Enabling it (and transferring data) requires having a matching GPG key. For this reason, it is not enabled automatically.

Optionally, you can start by checking the list of available special remotes. Each special remote has its own ID and assigned name, which can be used interchangeably inside git-annex commands. To list remotes known to the dataset, run:

git annex info

Pending future changes, the remote we want to enable is going to be named cerebra-storage, and it should appear in the list of semitrusted repositories like this:

f095040a-18b7-4648-849e-f154c53ce976 -- cerebra-mask

To enable the cerebra-mask special remote, run:

git annex enableremote cerebra-mask

You should see the following output:

enableremote cerebra-mask (encryption update) (to gpg keys: [abbreviated]) ok
(recording state in git...)

If that didn't work, see the Troubleshooting section below.

Data access

Note

This assumes your working directory is in the dataset (that you cd'ed into the dataset).

To download data, use:

datalad get output

To remove your local copy of the data, use drop which does the opposite of get:

datalad drop output

To obtain the latest version of the data, use:

datalad update --how merge
datalad get output

To see when the data (according to your local state) last changed, use git log (up/down arrows scroll, Q quits):

git log

Troubleshooting

Operations which use your secret key (like downloading encrypted data with git-annex or listing your secret keys with gpg --list-secret-keys) require interactions with a program called gpg-agent 3. Normally, gpg-agent should be running in the background, and there is nothing to do.

If you notice gpg-agent is not running (e.g. enableremote or get fails with an error message which says that gpg-agent is not running, or you can not list your secret keys), you can start the gpg-agent with:

gpg-connect-agent reloadagent /bye

If an operation (e.g. enableremote or get) fails with an error message which says that a private key is not available (but you know from running gpg --list-secret-keys that your private key is configured), or the message says gpg-agent is not running (but you know it is) it is possible that you have more than one gpg program installed, and Git-annex uses a different one that you would expect. See the GPG (Windows) part of the Before you begin section above.


  1. For a more technical introduction to what Git credential helpers do, see this blog post: Git The Stupid Password Store ↩︎

  2. It would, at least in theory, be possible to modify the PATH variable so that running gpg would use the Git-bundled gpg instead of the gpg4win gpg. ↩︎

  3. You call datalad get, DataLad calls git annex get; git-annex calls gpg --decrypt, and gpg asks gpg-agent for a matching secret key. ↩︎