Goodfire AI researchers introduced Block-Sparse Featurizers (BSF), a new framework for interpreting model activations that decomposes them into multidimensional subspaces, unlike Sparse Autoencoders (SAEs) which isolate concepts into single one-dimensional directions. SAEs, the current gold standard of mechanistic interpretability, are trained by projecting dense neural network activations into a wider, overcomplete layer, using a sparsity penalty to reconstruct data with only a few active 1D directions. However, the researchers noted that “recent work points to more structure in these models than a single direction allows,” as complex concepts in neural networks, especially in vision models, reside on continuous, low-dimensional geometric spaces called manifolds. Because SAEs enforce sparsity on individual 1D lines, they force these continuous clouds into rigid, isolated points, leading to feature splitting—for example, assigning separate lines for a 10-degree curve and a 20-degree curve instead of capturing a full rotation. Michael Byun, Member of Technical Staff at Goodfire, described an “aha moment” when the team saw “really clean examples of how SAE features ‘shatter’ manifold features like temperature, years, and colors.” BSF’s geometric shift aims to provide a fine-grained explanation of model internals and enable better steerability and control in some AI applications.