A fully connected layer

Feedforward neural networks have this name because the flow of information being evaluated by the neural network starts at the input  , flows all the way through the hidden layers, and finally reaches the output  . Note that the output of each layer does flow back to itself. In some models there are residual connections in which the input of the layer is added or concatenated to the output of the layer itself. In the following figure, we provide a visualization in which the input of Layer 2, Out 1, is concatenated with the output of Layer 2: