191 lines
8.3 KiB
Markdown
191 lines
8.3 KiB
Markdown
+++
|
|
date = '2025-06-17T18:45:00+02:00'
|
|
draft = false
|
|
title = 'Forgejo with encryption'
|
|
+++
|
|
|
|
This walkthrough demonstrates how to encrypt annexed file content in Forgejo,
|
|
a bare Git repository, or another Git hosting platform with annex support.
|
|
|
|
## Introduction
|
|
|
|
When storing DataLad datasets on third party infrastructure, Git
|
|
remotes and git-annex special remotes are often configured pointing to
|
|
different services. However, some repository hosting services
|
|
(e.g. G-node GIN, or some instances of Forgejo), as well as bare Git
|
|
repositories[^1] on systems with git-annex installed, support storing
|
|
annex objects alongside the Git repository, which makes them very
|
|
convenient in terms of configuration and usage. Throughout the
|
|
article, we will refer to remotes supporting both Git and annex
|
|
content as "git+annex". For background, see DataLad handbook's chapter
|
|
[Beyond shared
|
|
infrastructure](https://handbook.datalad.org/en/latest/basics/101-138-sharethirdparty.html).
|
|
|
|
Some of the SFB1451 datasets may require using data encryption at rest
|
|
to guarantee data privacy or access control. Git-annex [supports
|
|
encryption](https://git-annex.branchable.com/encryption/) and it can
|
|
be very easy to set up. However, encryption is configured for *special
|
|
remotes* and as such it would not apply to such "git+annex" remotes[^2].
|
|
|
|
This has been addressed in [git-annex v.10.20250520](https://git-annex.branchable.com/news/version_10.20250520/)
|
|
by the addition of the [mask special remote](https://git-annex.branchable.com/special_remotes/mask/).
|
|
|
|
With the mask special remote, setting up encrypted storage for
|
|
git+annex comes down to the following steps:
|
|
|
|
- set up a Git remote,
|
|
- set up a mask special remote with encryption enabled,
|
|
- make sure that annexed contents are pushed only to the latter.
|
|
|
|
The model use case for this workflow is publishing datasets on Forgejo
|
|
instances with git-annex support. [Forgejo](https://forgejo.org/) is
|
|
a "self-hosted lightweight software forge", which can be used to set
|
|
up and maintain a platform similar to GitHub or GitLab.
|
|
[Forgejo-aneksajo](https://codeberg.org/forgejo-aneksajo/forgejo-aneksajo)
|
|
is a lightly modified version of Forgejo which adds git-annex support.
|
|
While no public instances are readily available, and the SFB1451 does
|
|
not (currently) maintain its own instance, it is nevertheless very
|
|
relevant for the SFB1451 community because Forgejo-aneksajo can be
|
|
deployed by individual groups (see [Collaborative infrastructure for a
|
|
lab: Forgejo](https://blog.datalad.org/posts/lab-infra-forgejo/) for
|
|
more background information).
|
|
|
|
## Prerequisites
|
|
|
|
- git-annex v.10.20250520 or newer
|
|
- gpg, and a generated private/public key
|
|
- DataLad-next (optional but highly recommended)
|
|
|
|
## Walkthrough
|
|
|
|
### Initial set-up and publishing
|
|
|
|
Publishing a dataset requires having a dataset in the first place.
|
|
In this example we create a dataset with one file in Git (README) and one file annexed (an image).
|
|
|
|
```
|
|
datalad create foo
|
|
cd foo
|
|
echo "Hello world" > README
|
|
datalad download https://images.pexels.com/photos/21533358/pexels-photo-21533358.jpeg
|
|
datalad save --to-git -m "Add a readme" README
|
|
datalad save -m "Add a photo of a penguin" pexels-photo-21533358.jpeg
|
|
```
|
|
|
|
Add a Git remote pointing to a valid address -- for example, an empty Forgejo-aneksajo repository,
|
|
or (for easier testing) a local bare git repository.
|
|
|
|
{{< tabs >}}
|
|
{{% tab title="forgejo" %}}
|
|
```
|
|
git remote add origin https://hub.example.com/user/dataset.git
|
|
```
|
|
{{% /tab %}}
|
|
{{% tab title="local" %}}
|
|
```
|
|
git init --bare /tmp/foo.git
|
|
git remote add origin /tmp/foo.git
|
|
```
|
|
{{% /tab %}}
|
|
{{< /tabs >}}
|
|
|
|
Push (only the Git part -- we do not want to send unencrypted data)
|
|
to ensure that the remote repository is initialized with annex UUID.
|
|
|
|
```
|
|
datalad push --to origin --data nothing
|
|
```
|
|
|
|
At this point, the remote should be recognized by Git-annex, and have an annex UUID.
|
|
You can check this with `git annex info origin`.
|
|
If the UUID is present, the mask remote can be set up[^3].
|
|
Replace the value of the `keyid` parameter with the ID (either e-mail or the numeric key ID) of your GPG key.
|
|
|
|
```
|
|
git annex initremote encrypted-origin type=mask remote=origin encryption=hybrid keyid=me@example.com
|
|
```
|
|
|
|
With the remote configuration, the annexed content and the Git content can be pushed *separately* to these remotes.
|
|
|
|
```
|
|
datalad push --to encrypted-origin
|
|
datalad push --data nothing --to origin
|
|
```
|
|
|
|
That's it! The repository is a regular repository, but the annexed content can not be accessed.
|
|
The web interface will show the file tree, but report that the files are unavailable.
|
|
Cloning the repository is possible, but getting the annexed content requires possession of a private GPG key
|
|
for which the encryption has been enabled[^4].
|
|
|
|
### Automating
|
|
|
|
Remembering to push to the mask remote first, and to use `--data nothing` with the unencrypted remote
|
|
can be tedious and error prone.
|
|
However, both of these behaviors can be automated.
|
|
Git-annex wanted mechanism can be used to state that no data are wanted by the unencrypted remote.
|
|
DataLad's publication dependency can be used to say that the encrypted remote should be pushed automatically
|
|
when a push to origin is requested.
|
|
|
|
```
|
|
git annex wanted origin "exclude=*"
|
|
datalad siblings configure -s origin --publish-depends encrypted-origin
|
|
```
|
|
|
|
With this, it is sufficient to `datalad push --to origin`[^5].
|
|
|
|
This can be taken even further.
|
|
For example, only *some* annexed files can be pushed encrypted.
|
|
It is advisable to configure the encrypted and regular remotes with opposite settings to avoid duplicating the storage.
|
|
For example, consider a situation where raw images are stored encrypted, but jpegs are not:
|
|
|
|
```
|
|
git annex wanted origin include=*jpeg
|
|
git annex wanted encrypted-origin include=*arw
|
|
```
|
|
|
|
## Looking under the hood
|
|
|
|
Although we used Forgejo as an example (due to its potential relevance),
|
|
the walkthrough is valid for any "git+annex" repository. Using a local
|
|
bare Git repository is a good way to "look under the hood" and see that
|
|
the annex keys in the bare repository are indeed stored encrypted,
|
|
with HMAC hashing of the filenames.
|
|
|
|
The example below shows the annex objects directory with three files,
|
|
two encrypted and one unencrypted.
|
|
|
|
```
|
|
❱ tree ../foo.git/annex/objects
|
|
../foo.git/annex/objects
|
|
├── 39e
|
|
│ └── bcf
|
|
│ └── GPGHMACSHA1--2c4b73cff92c9d186065f338551f8c5c8ec94b5b
|
|
│ └── GPGHMACSHA1--2c4b73cff92c9d186065f338551f8c5c8ec94b5b
|
|
├── 727
|
|
│ └── f52
|
|
│ └── GPGHMACSHA1--fbcb8f7931cb90c8fa9930368bfae671f36fec6b
|
|
│ └── GPGHMACSHA1--fbcb8f7931cb90c8fa9930368bfae671f36fec6b
|
|
└── a45
|
|
└── ab5
|
|
└── SHA256E-s2273030--fbeaece1f05264c39781adeac159ba33fbc1f4d2aea273106600bc4e3e3aa6c7.jpeg
|
|
└── SHA256E-s2273030--fbeaece1f05264c39781adeac159ba33fbc1f4d2aea273106600bc4e3e3aa6c7.jpeg
|
|
```
|
|
|
|
In Forgejo, the layout would be identical.
|
|
|
|
|
|
[^1]: Bare Git repository is a repository which does not contain a working directory.
|
|
It can be cloned from and pushed to.
|
|
See [Git on the server](https://git-scm.com/book/en/v2/Git-on-the-Server-Getting-Git-on-a-Server) in the Pro Git book.
|
|
[^2]: Git annex does have a [git special remote](https://git-annex.branchable.com/special_remotes/git/),
|
|
but its purpose is to record a Git remote URL in the repository (normally, Git remote URL is a local configuration)
|
|
and optionally autoenable it. It does not provide a way to set up encryption.
|
|
[^3]: In Forgejo-aneksajo versions prior to 11.0-0-git-annex2, the UUID was not always assigned immediately;
|
|
see discussion in [forgejo-aneksajo/issues/22](https://codeberg.org/forgejo-aneksajo/forgejo-aneksajo/issues/22).
|
|
A workaround was to perform any action (including viewing the page) in the web interface.
|
|
If the UUID is empty, visit the repository page in a browser (or refresh it) and try `git annex info origin` again.
|
|
[^4]: In the hybrid encryption scheme, data is encrypted using a symmetric cipher,
|
|
and that cipher is asymmetrically encrypted (to selected GPG keys) and stored in the Git repository.
|
|
This means that new recipient keys can be added without having to re-encrypt the data.
|
|
See the documentation for [git annex encryption](https://git-annex.branchable.com/encryption/).
|
|
[^5]: or `git annex push` which by default pushes to all available remotes
|