middleware

Middleware for an IPFS pinning service.

Introduction

This project sits between IPFS Cluster and the people or tools that pin content. It is meant to run behind a trusted reverse proxy for the dashboard user experience, where the proxy authenticates human users and passes only a vetted identity header upstream. The backend also exposes a standalone /admin dashboard and /admin/* API that are authenticated directly by the middleware using an argon2 admin token from the environment.

The app keeps the operational model simple:

• dashboard traffic is authenticated by the edge proxy
• /admin and /admin/* traffic are authenticated directly by the backend using ADMIN_TOKEN
• /pins/* traffic is authenticated inside the app with bearer tokens
• pin ownership, quotas, and metadata live in SQLite
• the service talks to IPFS Cluster for pin lifecycle work

This makes it suitable for a small deployment where a private reverse proxy handles user authentication and the middleware focuses on pin ownership and quota enforcement.

Pins move through a queued/pinning lifecycle asynchronously. The app launches a background worker on startup that polls queued pins every 5 seconds and processes them in batches. If the worker is temporarily unavailable, the queue is backed up in SQLite, and a pin can legitimately remain in queued for a while before it transitions to pinning, pinned, or failed.

Features

• trusted reverse-proxy identity checks
• per-user pin ownership and quota tracking
• bearer-token API for pin operations
• backend-verified /admin dashboard and API protected by an argon2 token
• gateway URL resolution for pinned CIDs
• public/private visibility controls on dashboard pins
• default-off public checkbox for dashboard CID pins and uploads
• SQLite-backed metadata storage and deduplication accounting
• health checks and admin actions for user maintenance
• remote backend pin drift checks that compare local pins with the configured cluster API
• delayed Kubo repo gc after unpins and failed pin attempts when the queue is idle
• per-pin status_updated_at tracking so dashboard users can see when the last state change occurred
• time-bound pin expiry and worker-driven unpinning with repo GC kick-off
• per-pin immutability lock with explicit dangerous-action double-confirmation in the dashboard

How to operate (for admins)

1) Keep the middleware private

The middleware should not be exposed directly to the internet. Put Caddy, Nginx, Traefik, or another edge proxy in front of it and keep the app on a private network or localhost-only bind.

The important rule is simple:

• the reverse proxy authenticates dashboard users
• the reverse proxy overwrites and strips any untrusted identity headers
• the middleware trusts only the configured identity header, such as X-Remote-User
• /admin and /admin/* are protected by an argon2 token verified in the backend via ADMIN_TOKEN
• the login page is served at /admin/login; the protected dashboard itself lives at /admin
• /pins/* is not protected with basicauth; it is validated by the app with Authorization: Bearer <token>

2) Configure the service

Typical environment variables:

DATABASE_URL=file:middleware.db?_foreign_keys=on
CLUSTER_API_URL=http://127.0.0.1:9097
IPFS_API_URL=http://127.0.0.1:5001
GATEWAY_URL_TEMPLATE=https://gateway.example.com/ipfs/{cid}
TRUSTED_IDENTITY_HEADER=X-Remote-User
ADMIN_TOKEN='$argon2id$v=19$m=65536,t=3,p=4$...'
DEFAULT_USER_QUOTA_BYTES=10737418240
BACKEND_REQUEST_TIMEOUT=60s
LISTEN_ADDR=127.0.0.1:8080

Generate an admin token hash:

python3 -m venv .venv
. .venv/bin/activate
python -m pip install argon2-cffi
python - <<'PY'
from argon2 import PasswordHasher
ph = PasswordHasher(time_cost=3, memory_cost=65536, parallelism=4, hash_len=32, salt_len=16)
print(ph.hash('super-secret-token'))
PY

The printed value is the ADMIN_TOKEN value to put in the environment.

When a pin is removed or a queued pin fails after it has pulled extra blocks, the middleware arms a delayed Kubo repo gc call. The GC waits 10 minutes, is skipped while queued or pinning work is active, and is re-armed only after the next relevant cleanup event.

Start the service:

go run ./cmd/server

3) Protect the dashboard and admin routes in the proxy

A minimal Caddy pattern looks like this:

https://pin.example.com {
    @dashboard path / /me /me/* /dashboard /dashboard/*
    handle @dashboard {
        basicauth {
            admin $2a$14$...
        }

        reverse_proxy 127.0.0.1:8080 {
            header_up -Authorization
            header_up -X-Remote-User
            header_up -X-Forwarded-User
            header_up X-Remote-User {http.auth.user.id}
        }
    }

    handle /admin* {
        reverse_proxy 127.0.0.1:8080 {
            # Important: do not strip Authorization here. The backend verifies the admin token itself.
            header_up -X-Remote-User
            header_up -X-Forwarded-User
        }
    }

    handle /pins* {
        reverse_proxy 127.0.0.1:8080 {
            header_up -Authorization
            header_up -X-Remote-User
            header_up -X-Forwarded-User
        }
    }
}

This keeps the human login in the edge proxy while the app validates bearer tokens for /pins/* itself.

4) Admin tasks

The admin routes are authenticated by the backend using the configured ADMIN_TOKEN. The login form is served at /admin/login, while the protected dashboard itself is served at /admin and requires a valid token in the request headers. The dashboard is rendered by the middleware itself and does not depend on the reverse proxy for access control.

Common operations:

• recalculate a user’s pin sizes and usage

  curl -H 'Authorization: Bearer <admin-token>' -X POST https://pin.example.com/admin/users/alice/recalculate
  

• set or change a user quota

  curl -H 'Authorization: Bearer <admin-token>' -H 'Content-Type: application/json' \
    -X POST -d '{"quota_bytes":10737418240}' \
    https://pin.example.com/admin/users/alice/quota
  

• compare all local pins with the pinning backend and report the global delta

  curl -H 'Authorization: Bearer <admin-token>' -X POST \
    https://pin.example.com/admin/pins/remote-delta
  

  The response includes a delta object with:

  • missing_from_remote: local pins that are absent on the remote backend, each with owner metadata showing which user owns the pin
  • missing_from_local: remote pins that are absent from the local SQLite database

• queue missing local CIDs onto the remote Cluster backend using the same pinned-only comparison logic

  curl -H 'Authorization: Bearer <admin-token>' -X POST \
    https://pin.example.com/admin/pins/remote-sync
  

  This only checks remote pins whose status is pinned; any local CID that is missing from the remote pinned set is requested from Cluster via /pins.

• resolve a gateway URL for a CID through the dashboard context

  curl -H 'Authorization: Bearer <admin-token>' https://pin.example.com/me/gateway-url/bafy...
  

5) API usage

Use bearer tokens for the pinning API:

curl -H 'Authorization: Bearer <api-key>' \
  -H 'Content-Type: application/json' \
  -d '{"cid":"bafy...","name":"example","expected_bytes":12345,"public":true}' \
  https://pin.example.com/pins

The app validates the token and resolves the user from the database. It does not need an edge basicauth check for /pins/*.

Pins may also carry an expires_at deadline and an immutable lock window. The worker automatically unpins expired pins and triggers repo GC, while the immutable lock prevents delete operations until the deadline is reached. Expiration and lock updates refuse invalid combinations that would shorten an active immutability window, the dashboard dangerous-actions mode requires a clear double confirmation before creating a lock, and the expiry/lock forms use a native datetime-local picker for selecting date and time.

The dashboard also includes a default-off "Make public" checkbox when you pin a CID or upload files/folders. A public pin is surfaced on the user’s public profile only after it reaches the pinned state.

Repository overview

The code is organised into a small set of packages:

• cmd/server: HTTP server entrypoint
• internal/app: router and handlers
• internal/auth: trusted identity and bearer-token middleware
• internal/service: pin and quota logic
• internal/store: SQLite schema and queries
• internal/model: data structures
• internal/cluster: IPFS Cluster client integration
• internal/config: environment configuration

The project is intentionally small and explicit. It is built for local deployments, private networking, and straightforward admin scripts rather than large-scale identity systems.