One "+x" That Made 100-Layer Networks Trainable: ResNet Skip Connections
Deep networks have a cruel paradox. In theory, more layers should never hurt — the extra ones could just learn to pass their input through unchanged. In practice, before 2015, stacking more plain layers made. . .
Deep networks have a cruel paradox. In theory, more layers should never hurt — the extra ones could just learn to pass their input through unchanged. In practice, before 2015, stacking more plain layers made networksworse: a 56-layer net had higher training error than a 20-layer one. The gradient vanished on its way back to the early layers, and optimisation couldn't even find that "do nothing" identity mapping. ResNet fixed it with almost absurdly little. The residual reformulationInstead of asking a block to learn a full mappingH(x), ask it to learn theresidualF(x) = H(x) − x, and add the input back:defforward(self,x):returnF. relu(x+self. f(x))# y = x + F(x) 0 (each F' ~O(1) ^ the identity path never vanishesEnter fullscreen modeExit fullscreen modeThe identity path is a gradient highway straight back to the earliest layers. Projection shortcutsWhen a block changes the feature dimensions (a conv that halves spatial size, doubles channels),xandF(x)no longer match, so you can't add them. Put a 1×1 conv on the skip to projectxinto the new shape first — the "projection shortcut" from the paper. Most shortcuts are plain identity; only dimension-changing ones need this. The impactWith residual blocks, the 2015 ResNet paper trained 152-layer networks — an order of magnitude deeper than what worked before — and won ImageNet. Deeper finally meant better again. And skip connections are noweverywhere: ResNets, U-Nets, and every Transformer block (x + Sublayer(x)). The same +1 quietly keeps gradients healthy inside modern LLMs. See a plain net vs a ResNet at the same depth, gradient-by-gradient:https://dev48v. infy. uk/dl/day22-resnet-skip-connections. html