Confirmed availability of complete weights
ByteDance Seed has published the complete weights of the Cross-Layer TopK CLT, an interpretability model designed specifically for Qwen3-8B-Base, in its public repository. Verification of the file tree confirms the presence of the 36 encoding files (W_enc) and 36 decoding files (W_dec), along with normalization factors and configuration.
The upload was completed at the end of October 2026, with most files loaded between November 3 and 5. The total set occupies 377 GB in FP32 format and contains 94.2 billion parameters. The availability of all components necessary for encoding and reconstruction eliminates dependence on partial or unverified versions.
Components of the Cross-Layer TopK CLT repository
Breakdown of the essential files for running the transcoder.
| File | Content | Approximate Size |
|---|---|---|
| W_enc_{l}.safetensors | Encoder for layer l: [32768, 4096] | 537 MB per file |
| W_dec_{l}.safetensors | Decoder for layer l: [32768, 36-l, 4096] | 2.68 GB to 19.3 GB |
| W_dec_norm.safetensors | Decoder column norms | 4.72 MB |
| factors.safetensors | Mean activation norms (resid_mid, mlp_out) | Small |
Architecture and operating mechanism
The transcoder operates as a system of latent features shared across layers. Each of the 36 layers of Qwen3-8B-Base possesses 32,768 latents that read the residual stream just before the post-attention normalization layer (resid_mid) and write to the outputs of the MLPs of that layer and all subsequent layers.
The encoding process applies a ReLU function to the projection of the residual stream by the corresponding layer's encoder. Next, a score is calculated for each latent by multiplying its activation by the norm of its decoding column. The system retains the 360 highest-scoring features (36 layers × k=10) jointly across all layers, ensuring a sparse and global representation of the model's internal state.
Infrastructure requirements and execution
Running the complete transcoder requires significant computing infrastructure. The model was trained with FP32 activations, so the base Qwen3-8B-Base model must also run in FP32 to maintain numerical coherence. The total storage of 377 GB demands a system with adequate disk and memory capacity.
For encoding without reconstruction, it is possible to exclude the decoding files (W_dec), reducing memory load. In this mode, a single 80 GB GPU is sufficient to process the latents. For complete reconstruction of MLP outputs, it is necessary to distribute the latents across multiple GPUs, as each GPU stores a portion of the 32,768 features of each layer.
Encoding and reconstruction process
Technical steps to run the transcoder on an input phrase.
- 01Download the complete checkpoint or only the encoders depending on the objective.
- 02Load Qwen3-8B-Base in FP32 and register hooks on resid_mid and mlp_out.
- 03Calculate latent activations via ReLU(W_enc · x).
- 04Select the 360 latents with the highest global score (activation × decoder norm).
- 05Reconstruct mlp_out by summing the contributions of active latents from each layer.
License and terms of use
The weights and code of the Cross-Layer TopK CLT are distributed under the Apache License 2.0, with copyright by ByteDance and/or its affiliates in 2026. This license permits use, modification, and commercial distribution, provided that copyright notices are maintained and a copy of the license is included in derivative works.
Training data comes from Dolma 3, Dolci-Think, and Dolci-Instruct, mixed in a 5:2:1 ratio by tokens (approximately 3.4 billion tokens). These datasets are licensed under ODC-BY. The combination of a permissive code license and open data facilitates the integration of the transcoder into interpretability pipelines without commercial use restrictions.
Limits and considerations for evaluation
The official documentation details the encoding and reconstruction procedure but does not provide a verified numerical score for the quality of the approximation. Evaluating the fidelity of the transcoder requires measuring the reconstruction error of mlp_out and the sensitivity to the ablation of specific features.
Tokens whose resid_mid or mlp_out reaches 20 times the mean norm of the layer are discarded during inference, as occurred during training. In practice, this mainly affects the first token of the sequence. File sizes and the degree of sparsity of the latents are not quality metrics in themselves, but structural characteristics of the model.
Keep exploring
Corrections and transparency
If you spot incorrect or outdated information, send us a correction with the page and source we should review.
Submit a correction