Computation Graph
Input Variables
Operations
Output
Active Node
Derivative Propagation
x = 2.0
y = 1.0
Results
f(x,y) =
-
∂f/∂x =
-
∂f/∂y =
-
Step:
0
Forward vs Reverse
Forward Mode:
Propagate ∂x/∂x = 1 forward
Compute ∂f/∂x directly
O(n) for n inputs, 1 output
Reverse Mode:
Forward pass: compute values
Backward pass: ∂f/∂f = 1
O(m) for n inputs, m outputs
Used in backprop!
Propagate ∂x/∂x = 1 forward
Compute ∂f/∂x directly
O(n) for n inputs, 1 output
Reverse Mode:
Forward pass: compute values
Backward pass: ∂f/∂f = 1
O(m) for n inputs, m outputs
Used in backprop!
Chain Rule
∂f/∂x = Σᵢ (∂f/∂vᵢ)(∂vᵢ/∂x)
Each node applies the chain
rule to combine gradients
Each node applies the chain
rule to combine gradients