Introduction
I’m not an expert in machine learning. I've been studying the field out of curiosity and an almost irrational drive to understand if some things could be done differently. I ended up thinking about a simple idea — which might already exist in more sophisticated forms — but I thought it was worth sharing: a regularization strategy based on weight blocks.
The idea came as an attempt to simplify regularization processes. What if we grouped weights into trios forming structural subsets (blocks), and applied a smoothing average?
Key detail: these blocks can be organized as sliding trios, like:
[w₀, w₁, w₂], [w₁, w₂, w₃], [w₂, w₃, w₄]...
That is, the blocks overlap, and each trio of adjacent weights becomes a regularized unit. The goal is to reduce abrupt contrasts, creating a form of structural continuity that smooths transitions between activations and helps prevent overfitting — without needing to eliminate features.
Each intermediate weight (like w₂) participates in multiple overlapping blocks, undergoing multiple smoothing passes. This creates an effect similar to an implicit smoothing hidden layer, acting in the weight space even before regular propagation and backpropagation — like an internal wire “stretching” the net's shape.
This approach also handles the extremes (w₀ and wₙ₊₁) using phantom blocks to ensure these edge weights are regularized fairly.
Where:
w₀ and wₙ₊₁ are edge weights;Bᵢ is each block trio [wᵢ, wᵢ₊₁, wᵢ₊₂];λ is the regularization coefficient;N is the number of sliding blocks.½(y - ŷ)²;I'm a curious person trying to understand how models can be built more simply. This text reflects a simple, but possibly useful idea — and I’d really appreciate any insights, criticism, or counterpoints.
Proposal written by Otto, based on personal intuition and structured with help from writing tools.