Psyche: Distributed Transformer Training Over the Internet — A First Look

TL;DR — Psyche is a protocol that coordinates independent computers across the internet to jointly train a single transformer model, without requiring participants to trust each other. It combines peer-to-peer networking, a custom low-bandwidth optimizer (DisTrO), and Solana-based on-chain coordination with slashing and reward mechanics. This post is a baseline tour — what it is, how it works, and what questions it raises — not an evaluation of its relevance to our own work.

What Is Psyche?

Psyche is a system built by Nous Research that enables distributed training of transformer-based AI models over the internet. The pitch is straightforward but ambitious: instead of running training on a centralized cluster with high-speed interconnects (the standard GPU-farm model), Psyche distributes the workload across many independent machines — each contributing a small piece of the training process.

The core challenge it addresses is not computational but coordinational: how do you get untrusted, heterogeneous, loosely connected nodes to collaboratively train a coherent model without any single party being able to corrupt the result?

Nous Research frames Psyche as a response to the centralization of AI development, arguing that “enormous amounts of computing power sit idle or underutilized worldwide” and that the current concentration of power “threatens innovation and limits who can contribute to AI progress.” The project’s roadmap has two stages: (1) cooperative training, beginning with a permissioned testnet and transitioning toward a fully decentralized environment, and (2) accessible inference and advanced capabilities, including reinforcement learning and reasoning models.

How It Works — The Mechanics

Epoch-Based Training Rounds

Training on Psyche proceeds in structured epochs. Each run is configured with:

• Warmup time — a window for nodes to download the current model checkpoint from peers.
• Training time — the actual training interval, bounded by max_round_train_time. This constrains which GPUs can participate: too short and slower hardware cannot complete a training step in time.
• Cooldown time — nodes bring the model from GPU to disk and decide whether to join the next round.
• Witness time — selected witness nodes publish proof messages before the next round begins.

A configurable min_clients threshold determines whether an epoch proceeds; if too many clients drop out, the run enters cooldown and returns to a waiting state.

The DisTrO Optimizer

Psyche does not use standard optimizers. The only supported optimizer is DisTrO (Distributed Optimizer), which builds on Nous Research’s earlier DeMo: Decoupled Momentum Optimization paper. The core idea is to compress the information exchanged between nodes so aggressively that training becomes feasible over public-internet bandwidth.

DisTrO works by applying a discrete cosine transform (DCT) — the same transform used in JPEG compression — to the momentum tensor that each node tracks during training. In the frequency domain, most of the signal energy concentrates in a few large coefficients. DisTrO extracts only the top-k momentum components with the highest energy and communicates those to other nodes, discarding the rest. Unlike JPEG, which always compresses high frequencies more, DeMo/DisTrO selects the top-magnitude components regardless of their position in the frequency spectrum, avoiding systematic bias toward fixed coordinates.

Psyche’s implementation adds two improvements over the original DeMo algorithm:

1. Overlapped training: Nodes no longer wait for all updates from the previous step to arrive before starting the next training step. Because DisTrO results grow sub-linearly with model parameter size, communication latency should theoretically not be a bottleneck as models scale — the wall-clock time spent training the next step outpaces the time spent communicating the previous step’s results.

2. 1-bit quantization: Psyche discovered that simply communicating the sign of the DCT coefficients (positive or negative, 1 or -1) conveys all the information needed to reconstruct the correct momentum matrix. The magnitude carries practically no additional useful information. This compresses results by more than 3× and is enabled via the quantize_1bit flag.

In practice, the DCT basis matrices can be chunked and pregenerated, reducing the wall-clock overhead of compression and decompression to less than 1% of total training time.

[model.LLM.optimizer.Distro]
clip_grad_norm = 1.0
compression_decay = 0.999
compression_chunk = 64
compression_topk = 8
quantize_1bit = true

The key parameters — compression_topk, compression_chunk, and quantize_1bit — configure the aggressive gradient compression. This makes sense for the setting: if you are synchronizing gradients across nodes connected by the public internet rather than NVLink, bandwidth is the bottleneck, and standard all-reduce is infeasible.

On-Chain Coordination via Solana

This is where Psyche diverges most sharply from traditional distributed training. Coordination — run creation, participant authorization, reward distribution, slashing — happens on the Solana blockchain via a smart contract that acts as the coordinator for the training run. The coordinator stores metadata about the run and the list of participants, handles state transitions, provides randomness for witness assignments, and serves as a synchronization point.

Each participant needs a Solana keypair to identify themselves. Runs are created on-chain with a unique run ID, a join authority (controlling who can participate), and optional token-based reward configuration. Authorization can be:

• Public — the authorizer is set to the Solana null address (11111111111111111111111111111111), allowing anyone to join.
• Private — the run creator issues per-user authorizations signed with their keypair.

The Three Actors

Psyche’s architecture has three main roles:

1. Coordinator — the on-chain smart contract that manages run state, participant lists, and synchronization.
2. Clients — GPU nodes that perform training, optionally act as witnesses, and upload checkpoints. Clients compute gradients, share them with other clients, and update model parameters.
3. Data Provider — supplies the training data. Can be local (each client responsible for their own copy) or a shared HTTP/TCP provider.

Rewards and Slashing

Runs can optionally distribute token rewards to participants. The mechanism is configurable:

run-manager set-future-epoch-rates \
  --earning-rate-total-shared [EARNING_RATE] \
  --slashing-rate-per-client [SLASHING_RATE]

Participants earn points per epoch for contributing computation. The slashing-rate-per-client parameter suggests that misbehaving nodes can be penalized — losing accumulated rewards for producing incorrect or invalid training results. This is the game-theoretic stick: if a node submits bad gradients, it risks losing stake.

Witnessing and Verification

A configurable number of witness nodes are selected each round to publish proofs. The witness_nodes parameter controls how many; setting it to 0 selects all nodes. There is also a verification_percent parameter, though the docs note it should “always be set to 0 for now” — suggesting that active verification of training correctness is still being developed.

What It Looks Like to Join

From the join-a-run documentation, the prerequisites are:

• Linux with NVIDIA GPU and drivers
• Docker + NVIDIA Container Toolkit
• A Solana keypair

Participants receive a run-manager binary and configure a .env file with their wallet path, RPC endpoints, and run ID. The run-manager handles Docker image downloads, container lifecycle, and updates. Training progress streams as logs. Rewards can be claimed later via a treasurer command.

The configuration supports both data parallelism (DATA_PARALLELISM) and tensor parallelism (TENSOR_PARALLELISM), so a single node with multiple GPUs can shard the model across them — important for training models that don’t fit on a single GPU.

Framing and Positioning

Psyche’s framing is notable. The docs open with “foster collaboration between untrusted parties to create state-of-the-art machine learning models” — this is a crypto-native narrative applied to ML training. The use of Solana for coordination, token rewards for participation, and slashing for penalties places Psyche firmly in the intersection of decentralized infrastructure and AI.

What the project emphasizes: open participation, internet-scale distribution, game-theoretic integrity guarantees. What it glosses over (at least in the current docs): throughput comparisons with centralized training, the practical meaning of “untrusted” (what can a malicious participant actually do?), and the current state of verification (set to 0, meaning witnessing exists but active correctness checking is not yet enabled).

The community being courted is technically capable but not necessarily ML-specialist — the docs assume comfort with Docker, Solana CLI, and keypair management, but not with distributed training internals.