utorch.autograd

all `module-attribute`

__all__ = ['Tensor']

Tensor

Tensor(value=None, parents=None, fun=None, name=None, args=None)

A multidimensional array and computational graph node.

Tensor tracks numerical data alongside the operations and parent nodes that produced it, dynamically building a Directed Acyclic Graph (DAG) during the forward pass.

Parameters:

Name	Type	Description	Default
`value`	`Any`	Numerical data (scalar, list, or NumPy ndarray). If None, the node is initialized as an intermediate operation node.	`None`
`parents`	`Sequence[Tensor] \| None`	Upstream antecedent nodes that fed into this computation.	`None`
`fun`	`Callable[..., Any] \| None`	The forward mathematical function evaluated at this node.	`None`
`name`	`str \| None`	Optional human-readable identifier for visualization.	`None`
`args`	`Any`	Optional extra static arguments passed to `fun` (e.g. axes, min/max).	`None`

Attributes:

Name	Type	Description
`value`	`ndarray`	The evaluated forward output tensor.
`parents`	`list[Tensor]`	Direct input nodes in the computational graph.
`fun`	`Callable \| None`	The mathematical operation that generated this node.
`args`	`Any`	Additional constants or axes supplied to the operation.
`name`	`str \| None`	Label used in graph diagrams and debugging.
`grad`	`int \| float \| ndarray`	Accumulated gradient (computed in Branch 3+).

Example

import utorch as ut

x = ut.Tensor(2.0, name="x")
y = ut.Tensor(3.0, name="y")
z = x * y + 1.0

print(z.value)   # 7.0
print(z.is_leaf) # False

Initialize a computational graph node via leaf or inner node dispatch.

In a dynamic Directed Acyclic Graph (DAG), every node belongs to one of two fundamental categories:

Leaf Nodes (Inputs, Constants, Parameters):
Created directly by the user with a known starting value.
Have no antecedent parents (parents = []) and no creator function (fun = None).
Represent independent variables at the base of the tree (e.g., weights, biases, inputs).
Dispatched to: _init_if_leaf(value).
Inner Nodes (Intermediate Operations):
Created automatically as the output of an operation (e.g., +, *, relu).
Have antecedent parent nodes (parents) and an associated mathematical function (fun).
Numerical value is evaluated dynamically during the forward pass by applying fun(*parent_values).
Dispatched to: _init_as_inner_node(parents, fun, args).

Parameters:

Name	Type	Description	Default
`value`	`Any`	Numerical data (scalar int/float, list, or NumPy array). If provided, initialized as a leaf node.	`None`
`parents`	`Sequence[Tensor] \| None`	Upstream antecedent `Tensor` nodes that fed into this operation. Used when initializing an inner node.	`None`
`fun`	`Callable[..., Any] \| None`	The forward mathematical function evaluated at this node (e.g., `np.add`, `np.sin`). Required for inner nodes; must be `None` for leaf nodes.	`None`
`name`	`str \| None`	Optional human-readable identifier for visualization.	`None`
`args`	`Any`	Optional static constants passed to `fun` (e.g., reduction axis, power exponent).	`None`

Source code in utorch/autograd/tensor.py

def __init__(
    self,
    value: Any = None,
    parents: Sequence[Tensor] | None = None,
    fun: Callable[..., Any] | None = None,
    name: str | None = None,
    args: Any = None,
) -> None:
    r"""Initialize a computational graph node via leaf or inner node dispatch.

    In a dynamic Directed Acyclic Graph (DAG), every node belongs to one of
    two fundamental categories:

    1. **Leaf Nodes (Inputs, Constants, Parameters)**:
       - Created directly by the user with a known starting `value`.
       - Have **no antecedent parents** (`parents = []`) and **no creator
         function** (`fun = None`).
       - Represent independent variables at the base of the tree (e.g.,
         weights, biases, inputs).
       - Dispatched to: `_init_if_leaf(value)`.

    2. **Inner Nodes (Intermediate Operations)**:
       - Created automatically as the output of an operation (e.g., `+`,
         `*`, `relu`).
       - Have **antecedent parent nodes** (`parents`) and an **associated
         mathematical function** (`fun`).
       - Numerical value is evaluated dynamically during the forward pass by
         applying `fun(*parent_values)`.
       - Dispatched to: `_init_as_inner_node(parents, fun, args)`.

    Args:
        value: Numerical data (scalar int/float, list, or NumPy array).
            If provided, initialized as a **leaf node**.
        parents: Upstream antecedent `Tensor` nodes that fed into this
            operation. Used when initializing an **inner node**.
        fun: The forward mathematical function evaluated at this node
            (e.g., `np.add`, `np.sin`). Required for inner nodes; must be
            `None` for leaf nodes.
        name: Optional human-readable identifier for visualization.
        args: Optional static constants passed to `fun` (e.g., reduction
            axis, power exponent).

    """
    self.name: str | None = name
    if value is not None:
        self._init_if_leaf(value)
    else:
        self._init_as_inner_node(parents or [], fun, args)
    if value is not None:
        self._init_if_leaf(value)
    else:
        self._init_as_inner_node(parents or [], fun, args)

    if value is not None:
        self._init_if_leaf(value)
    else:
        self._init_as_inner_node(parents or [], fun, args)

    self.grad = None

is_leaf `property`

is_leaf

Return True if this node has no parents (user input or parameter).

shape `property`

shape

Return the dimensional shape of the underlying tensor.

add

__add__(other)

Element-wise addition: self + other.

Source code in utorch/autograd/tensor.py

def __add__(self, other: Any) -> Tensor:
    """Element-wise addition: `self + other`."""
    return self._two_variable_operator(other, lambda x, y: np.add(x, y))

eq

__eq__(other)

Element-wise equal: self == other.

Source code in utorch/autograd/tensor.py

def __eq__(self, other: Any) -> Any:
    """Element-wise equal: `self == other`."""
    if not isinstance(other, Tensor):
        return False
    return np.array_equal(self.value, other.value)

ge

__ge__(other)

Element-wise greater or equal: self >= other.

Source code in utorch/autograd/tensor.py

def __ge__(self, other: Any) -> Any:
    """Element-wise greater or equal: `self >= other`."""
    other_val = other.value if isinstance(other, Tensor) else other
    return np.greater_equal(self.value, other_val)

gt

__gt__(other)

Element-wise greater than: self > other.

Source code in utorch/autograd/tensor.py

def __gt__(self, other: Any) -> Any:
    """Element-wise greater than: `self > other`."""
    other_val = other.value if isinstance(other, Tensor) else other
    return np.greater(self.value, other_val)

hash

__hash__()

Calculates hash(value)`.

Source code in utorch/autograd/tensor.py

def __hash__(self) -> int:
    """Calculates hash(value)`."""
    return hash(id(self))

le

__le__(other)

Element-wise less or equal: self <= other.

Source code in utorch/autograd/tensor.py

def __le__(self, other: Any) -> Any:
    """Element-wise less or equal: `self <= other`."""
    other_val = other.value if isinstance(other, Tensor) else other
    return np.less_equal(self.value, other_val)

lt

__lt__(other)

Element-wise less than: self < other.

Source code in utorch/autograd/tensor.py

def __lt__(self, other: Any) -> Any:
    """Element-wise less than: `self < other`."""
    other_val = other.value if isinstance(other, Tensor) else other
    return np.less(self.value, other_val)

matmul

__matmul__(other)

Matrix multiplication: self @ other.

Source code in utorch/autograd/tensor.py

def __matmul__(self, other: Any) -> Tensor:
    """Matrix multiplication: `self @ other`."""
    return self._two_variable_operator(other, lambda x, y: np.matmul(x, y))

mul

__mul__(other)

Element-wise multiplication: self * other.

Source code in utorch/autograd/tensor.py

def __mul__(self, other: Any) -> Tensor:
    """Element-wise multiplication: `self * other`."""
    return self._two_variable_operator(other, lambda x, y: np.multiply(x, y))

neg

__neg__()

Element-wise negation: -self.

Source code in utorch/autograd/tensor.py

def __neg__(self) -> Tensor:
    """Element-wise negation: `-self`."""
    return self.single_operator(lambda x: np.negative(x))

pow

__pow__(exponent)

Element-wise power: self ** exponent.

Source code in utorch/autograd/tensor.py

def __pow__(self, exponent: int | float) -> Tensor:
    """Element-wise power: `self ** exponent`."""
    return self.single_operator(lambda x, p: np.power(x, p), args=exponent)

radd

__radd__(other)

Right-hand element-wise addition: other + self.

Source code in utorch/autograd/tensor.py

def __radd__(self, other: Any) -> Tensor:
    """Right-hand element-wise addition: `other + self`."""
    return self.__add__(other)

repr

__repr__()

Return a string representation of the tensor for inspection.

Source code in utorch/autograd/tensor.py

def __repr__(self) -> str:
    """Return a string representation of the tensor for inspection."""
    name_str = f"'{self.name}', " if self.name else ""
    if self.is_leaf:
        return f"ut.Tensor({name_str}value={self.value})"
    return (
        f"ut,.Tensor({name_str}value={self.value}, "
        f"fun={_get_function_name(self.fun)})"  # type: ignore[arg-type]
    )

rmul

__rmul__(other)

Right-hand element-wise multiplication: other * self.

Source code in utorch/autograd/tensor.py

def __rmul__(self, other: Any) -> Tensor:
    """Right-hand element-wise multiplication: `other * self`."""
    return self.__mul__(other)

rsub

__rsub__(other)

Right-hand element-wise subtraction: other - self.

Source code in utorch/autograd/tensor.py

def __rsub__(self, other: Any) -> Tensor:
    """Right-hand element-wise subtraction: `other - self`."""
    other_var = self._ensure_tensor(other)
    return other_var._two_variable_operator(self, lambda x, y: np.subtract(x, y))

rtruediv

__rtruediv__(other)

Right-hand element-wise division: other / self.

Source code in utorch/autograd/tensor.py

def __rtruediv__(self, other: Any) -> Tensor:
    """Right-hand element-wise division: `other / self`."""
    other_var = self._ensure_tensor(other)
    return other_var._two_variable_operator(self, lambda x, y: np.divide(x, y))

sub

__sub__(other)

Element-wise subtraction: self - other.

Source code in utorch/autograd/tensor.py

def __sub__(self, other: Any) -> Tensor:
    """Element-wise subtraction: `self - other`."""
    return self._two_variable_operator(other, lambda x, y: np.subtract(x, y))

truediv

__truediv__(other)

Element-wise division: self / other.

Source code in utorch/autograd/tensor.py

def __truediv__(self, other: Any) -> Tensor:
    """Element-wise division: `self / other`."""
    return self._two_variable_operator(other, lambda x, y: np.divide(x, y))

abs

abs()

Element-wise absolute value: \(|x|\).

Source code in utorch/autograd/tensor.py

def abs(self) -> Tensor:
    r"""Element-wise absolute value: $|x|$."""
    return self.single_operator(lambda x: np.abs(x))

backward

backward(grad_tensor=None)

Execute reverse-mode automatic differentiation (backpropagation).

Computes the gradient of this output tensor with respect to all upstream nodes in the computational graph using the multivariate chain rule.

Because gradients are accumulated as Tensor instances rather than raw floats, the backward pass dynamically constructs a secondary computation graph, enabling arbitrary higher-order derivatives (e.g., double backward for Hessians).

The backward algorithm executes in five distinct pedagogical phases:

Scalar Verification & Seed Initialization:
Reverse-mode autograd starts at a scalar objective (e.g., loss).
The root gradient is seeded with the identity: dL/dL = 1.0.
If an explicit grad_tensor is supplied, it is used as the seed.
Reverse Topological Ordering:
Performs a Depth-First Search (DFS) from self backwards to all reachable ancestor nodes.
Arranges nodes so every child appears strictly before its parents.
Guarantees that when a node is processed, it has already collected all incoming gradient contributions from all downstream paths.
Primitive Derivative Dispatch:
Traverses nodes in reverse topological order.
For each intermediate operation node, looks up its analytical derivative rule in SCALAR_PRIMITIVES using node.op.
Local Chain Rule Evaluation:
For binary operations z = f(x, y): dx = d_rule_left(grad_z, x, y) dy = d_rule_right(grad_z, x, y)
For unary operations y = f(x): dx = d_rule(grad_y, x)
Gradient Accumulation (Multivariate Chain Rule):
Local gradient contributions are added to antecedent parents using += (parent.grad = parent.grad + d_parent).
Accumulation (rather than overwriting) is mathematically mandatory when a variable is reused across multiple paths (diamond graphs).

Parameters:

Name	Type	Description	Default
`grad_tensor`	`Tensor \| None`	Seed gradient for this node. If None, defaults to Tensor(1.0).	`None`

Raises:

Type	Description
`RuntimeError`	If called on a non-scalar tensor without an explicit `grad_tensor`.
`ValueError`	If an operation node has no matching primitive rule in `SCALAR_PRIMITIVES`.

Examples:

import utorch as ut

# 1. First-order derivative of a polynomial: y = x^2 + 3x
x = ut.Tensor(4.0, name="x")
y = (x ** 2) + 3.0 * x  # dy/dx = 2x + 3 = 11.0
y.backward()
print(x.grad.value)     # 11.0

# 2. Second-order derivative (double backward): d2y/dx2 = 2.0
first_grad = x.grad
x.zero_grad()
first_grad.backward()
print(x.grad.value)     # 2.0

Source code in utorch/autograd/tensor.py

def backward(self, grad_tensor: Tensor | None = None) -> None:
    """Execute reverse-mode automatic differentiation (backpropagation).

    Computes the gradient of this output tensor with respect to all upstream
    nodes in the computational graph using the multivariate chain rule.

    Because gradients are accumulated as `Tensor` instances rather than raw
    floats, the backward pass dynamically constructs a secondary computation
    graph, enabling arbitrary higher-order derivatives (e.g., double backward
    for Hessians).

    The backward algorithm executes in five distinct pedagogical phases:

    1. **Scalar Verification & Seed Initialization**:
       - Reverse-mode autograd starts at a scalar objective (e.g., loss).
       - The root gradient is seeded with the identity: dL/dL = 1.0.
       - If an explicit `grad_tensor` is supplied, it is used as the seed.

    2. **Reverse Topological Ordering**:
       - Performs a Depth-First Search (DFS) from `self` backwards to all
         reachable ancestor nodes.
       - Arranges nodes so every child appears strictly before its parents.
       - Guarantees that when a node is processed, it has already collected
         all incoming gradient contributions from all downstream paths.

    3. **Primitive Derivative Dispatch**:
       - Traverses nodes in reverse topological order.
       - For each intermediate operation node, looks up its analytical
         derivative rule in `SCALAR_PRIMITIVES` using `node.op`.

    4. **Local Chain Rule Evaluation**:
       - For binary operations z = f(x, y):
         dx = d_rule_left(grad_z, x, y)
         dy = d_rule_right(grad_z, x, y)
       - For unary operations y = f(x):
         dx = d_rule(grad_y, x)

    5. **Gradient Accumulation (Multivariate Chain Rule)**:
       - Local gradient contributions are added to antecedent parents using
         `+=` (`parent.grad = parent.grad + d_parent`).
       - Accumulation (rather than overwriting) is mathematically mandatory
         when a variable is reused across multiple paths (diamond graphs).

    Args:
        grad_tensor: Seed gradient for this node. If None, defaults to
            Tensor(1.0).

    Raises:
        RuntimeError: If called on a non-scalar tensor without an explicit
            `grad_tensor`.
        ValueError: If an operation node has no matching primitive rule in
            `SCALAR_PRIMITIVES`.

    Examples:
        ```python
        import utorch as ut

        # 1. First-order derivative of a polynomial: y = x^2 + 3x
        x = ut.Tensor(4.0, name="x")
        y = (x ** 2) + 3.0 * x  # dy/dx = 2x + 3 = 11.0
        y.backward()
        print(x.grad.value)     # 11.0

        # 2. Second-order derivative (double backward): d2y/dx2 = 2.0
        first_grad = x.grad
        x.zero_grad()
        first_grad.backward()
        print(x.grad.value)     # 2.0
        ```
    """
    from utorch.autograd.ops import PRIMITIVES
    from utorch.autograd.unbroadcast import unbroadcast

    # 1. Seed root gradient as a Tensor node
    if grad_tensor is None:
        self.grad = Tensor(1.0)  # type: ignore[assignment]
    else:
        self.grad = grad_tensor  # type: ignore[assignment]

    # 2. Get reverse topological evaluation order (root first, down to leaves)
    ordered_nodes = topological_sort(self)

    # 3. Propagate gradients backwards
    for node in ordered_nodes:
        # Leaf nodes have no parents to propagate into
        if not node.parents:
            continue

        op_key = getattr(node, "op", None) or _get_function_name(node.fun)
        if op_key not in PRIMITIVES:
            raise ValueError(f"No derivative registered for op: '{op_key}'")

        rule = PRIMITIVES[op_key]

        # Binary operations (2 parents)
        if len(node.parents) == 2:
            parent_left, parent_right = node.parents

            # Support both a list [d_left, d_right]
            # or a function returning (d_left, d_right)
            if isinstance(rule, list):
                d_left = rule[0](node.grad, parent_left, parent_right, node.args)
                d_right = rule[1](node.grad, parent_left, parent_right, node.args)
            else:
                d_left, d_right = rule(
                    node.grad, parent_left, parent_right, node.args
                )
            # reverse broadcasting
            d_left = unbroadcast(d_left, parent_left.shape)
            d_right = unbroadcast(d_right, parent_right.shape)

            # Accumulate gradients
            # (+= records computation graph if grad is a Tensor)
            parent_left.grad = (
                d_left if parent_left.grad is None else parent_left.grad + d_left
            )
            parent_right.grad = (
                d_right
                if parent_right.grad is None
                else parent_right.grad + d_right
            )

        # Unary operations (1 parent)
        elif len(node.parents) == 1:
            parent = node.parents[0]
            rule_fn = rule[0] if isinstance(rule, list) else rule
            d_parent = rule_fn(node.grad, parent, node.args)
            d_parent = unbroadcast(d_parent, parent.shape)
            parent.grad = (
                d_parent if parent.grad is None else parent.grad + d_parent
            )

clip_min

clip_min(min_val)

Clip tensor values below a minimum threshold.

Source code in utorch/autograd/tensor.py

def clip_min(self, min_val: float) -> Tensor:
    """Clip tensor values below a minimum threshold."""
    return self.single_operator(
        lambda x, m: np.clip(x, a_min=m, a_max=None), args=min_val
    )

cos

cos()

Element-wise cosine: \(\\cos(x)\).

Source code in utorch/autograd/tensor.py

def cos(self) -> Tensor:
    r"""Element-wise cosine: $\\cos(x)$."""
    return self.single_operator(lambda x: np.cos(x))

cosh

cosh()

Element-wise hyperbolic cosine: \(\\cosh(x)\).

Source code in utorch/autograd/tensor.py

def cosh(self) -> Tensor:
    r"""Element-wise hyperbolic cosine: $\\cosh(x)$."""
    return self.single_operator(lambda x: np.cosh(x))

exp

exp()

Element-wise exponential: \(\\exp(x)\).

Source code in utorch/autograd/tensor.py

def exp(self) -> Tensor:
    r"""Element-wise exponential: $\\exp(x)$."""
    return self.single_operator(lambda x: np.exp(x))

log

log()

Element-wise natural logarithm: \(\\ln(x)\).

Source code in utorch/autograd/tensor.py

def log(self) -> Tensor:
    r"""Element-wise natural logarithm: $\\ln(x)$."""
    return self.single_operator(lambda x: np.log(x))

mean

mean(axis=None)

Compute the arithmetic mean of tensor elements across specified axis.

Source code in utorch/autograd/tensor.py

def mean(self, axis: int | tuple[int, ...] | None = None) -> Tensor:
    """Compute the arithmetic mean of tensor elements across specified axis."""
    if axis is None:
        return self.single_operator(lambda x: np.mean(x), args=None)
    return self.single_operator(lambda x, ax: np.mean(x, axis=ax), args=axis)

relu

relu()

Rectified Linear Unit activation: \(\\max(0, x)\).

Source code in utorch/autograd/tensor.py

def relu(self) -> Tensor:
    r"""Rectified Linear Unit activation: $\\max(0, x)$."""
    return self.single_operator(lambda x: np.maximum(0, x))

sigmoid

sigmoid()

Sigmoid activation function: \(\\sigma(x) = \\frac{1}{1 + e^{-x}}\).

Source code in utorch/autograd/tensor.py

def sigmoid(self) -> Tensor:
    r"""Sigmoid activation function: $\\sigma(x) = \\frac{1}{1 + e^{-x}}$."""
    return self.single_operator(lambda x: 1.0 / (1.0 + np.exp(-x)))

sign

sign()

Element-wise sign indicator (-1, 0, or 1).

Source code in utorch/autograd/tensor.py

def sign(self) -> Tensor:
    r"""Element-wise sign indicator (-1, 0, or 1)."""
    return self.single_operator(lambda x: np.sign(x))

sin

sin()

Element-wise sine: \(\\sin(x)\).

Source code in utorch/autograd/tensor.py

def sin(self) -> Tensor:
    r"""Element-wise sine: $\\sin(x)$."""
    return self.single_operator(lambda x: np.sin(x))

single_operator

single_operator(fun, args=None)

Apply a unary operation taking self as the sole parent node.

Parameters:

Name	Type	Description	Default
`fun`	`Callable[..., Any]`	Unary mathematical function (e.g. np.sin, np.exp).	required
`args`	`Any`	Optional static parameters.	`None`

Returns:

Type	Description
`Tensor`	A new Tensor representing the operation's output.

Source code in utorch/autograd/tensor.py

def single_operator(self, fun: Callable[..., Any], args: Any = None) -> Tensor:
    """Apply a unary operation taking self as the sole parent node.

    Args:
        fun: Unary mathematical function (e.g. np.sin, np.exp).
        args: Optional static parameters.

    Returns:
        A new Tensor representing the operation's output.
    """
    return Tensor(parents=[self], fun=fun, args=args)

sum

sum(axis=None)

Sum tensor elements across specified axis.

Source code in utorch/autograd/tensor.py

def sum(self, axis: int | tuple[int, ...] | None = None) -> Tensor:
    """Sum tensor elements across specified axis."""
    if axis is None:
        return self.single_operator(lambda x: np.sum(x), args=None)
    return self.single_operator(lambda x, ax: np.sum(x, axis=ax), args=axis)

tanh

tanh()

Element-wise hyperbolic tangent: \(\\tanh(x)\).

Source code in utorch/autograd/tensor.py

def tanh(self) -> Tensor:
    r"""Element-wise hyperbolic tangent: $\\tanh(x)$."""
    return self.single_operator(lambda x: np.tanh(x))

transpose

transpose(axes=None)

Permute the dimensions of the tensor.

Source code in utorch/autograd/tensor.py

def transpose(self, axes: Sequence[int] | None = None) -> Tensor:
    """Permute the dimensions of the tensor."""
    if axes is None:
        return self.single_operator(lambda x: np.transpose(x), args=None)
    return self.single_operator(lambda x, ax: np.transpose(x, axes=ax), args=axes)

zero_grad

zero_grad()

Reset gradient to zero on this node.

Source code in utorch/autograd/tensor.py

def zero_grad(self) -> None:
    """Reset gradient to zero on this node."""
    self.grad = None

ops

Scalar derivative rules for reverse-mode automatic differentiation.

matmul_backward

matmul_backward(grad, left, right)

Compute the Vector-Jacobian Product (VJP) for matrix multiplication.

Mathematical Intuition & Derivation: Let Y = A @ B, where: A has shape (M, K) B has shape (K, N) Y has shape (M, N) Let G = dL/dY be the incoming gradient of shape (M, N).

Instead of computing the massive (M*N) x (M*K) Jacobian matrix,
reverse-mode autograd evaluates the Vector-Jacobian Product directly.

By index notation, each output element is:
    Y_{i, j} = sum_k (A_{i, k} * B_{k, j})

Taking partial derivatives with respect to an entry A_{i, k}:
    dL/dA_{i, k} = sum_j (dL/dY_{i, j} * dY_{i, j}/dA_{i, k})
                 = sum_j (G_{i, j} * B_{k, j})
                 = sum_j (G_{i, j} * (B^T)_{j, k})
In matrix form:
    dL/dA = G @ B^T   (Shape: (M, N) @ (N, K) -> (M, K))

Similarly, for an entry B_{k, j}:
    dL/dB_{k, j} = sum_i (dL/dY_{i, j} * dY_{i, j}/dB_{k, j})
                 = sum_i (G_{i, j} * A_{i, k})
                 = sum_i ((A^T)_{k, i} * G_{i, j})
In matrix form:
    dL/dB = A^T @ G   (Shape: (K, M) @ (M, N) -> (K, N))

Dimensional Sanity Check (The "Shape Alignment Rule"): - dL/dA must match A.shape (M, K): G (M, N) @ B.T (N, K) -> (M, K) [Inner dimension N cancels] - dL/dB must match B.shape (K, N): A.T (K, M) @ G (M, N) -> (K, N) [Inner dimension M cancels]

Parameters:

Name	Type	Description	Default
`grad`	`Tensor`	Incoming gradient Tensor G of shape (M, N).	required
`left`	`Tensor`	Matrix A of shape (M, K).	required
`right`	`Tensor`	Matrix B of shape (K, N).	required

Returns:

Type	Description
`tuple[Tensor, Tensor]`	Tuple of (dL/dA, dL/dB) matching shapes of A and B respectively.

Source code in utorch/autograd/ops.py

def matmul_backward(
    grad: Tensor,
    left: Tensor,
    right: Tensor,
) -> tuple[Tensor, Tensor]:
    """Compute the Vector-Jacobian Product (VJP) for matrix multiplication.

    Mathematical Intuition & Derivation:
        Let Y = A @ B, where:
            A has shape (M, K)
            B has shape (K, N)
            Y has shape (M, N)
        Let G = dL/dY be the incoming gradient of shape (M, N).

        Instead of computing the massive (M*N) x (M*K) Jacobian matrix,
        reverse-mode autograd evaluates the Vector-Jacobian Product directly.

        By index notation, each output element is:
            Y_{i, j} = sum_k (A_{i, k} * B_{k, j})

        Taking partial derivatives with respect to an entry A_{i, k}:
            dL/dA_{i, k} = sum_j (dL/dY_{i, j} * dY_{i, j}/dA_{i, k})
                         = sum_j (G_{i, j} * B_{k, j})
                         = sum_j (G_{i, j} * (B^T)_{j, k})
        In matrix form:
            dL/dA = G @ B^T   (Shape: (M, N) @ (N, K) -> (M, K))

        Similarly, for an entry B_{k, j}:
            dL/dB_{k, j} = sum_i (dL/dY_{i, j} * dY_{i, j}/dB_{k, j})
                         = sum_i (G_{i, j} * A_{i, k})
                         = sum_i ((A^T)_{k, i} * G_{i, j})
        In matrix form:
            dL/dB = A^T @ G   (Shape: (K, M) @ (M, N) -> (K, N))

    Dimensional Sanity Check (The "Shape Alignment Rule"):
        - dL/dA must match A.shape (M, K):
          G (M, N) @ B.T (N, K) -> (M, K)  [Inner dimension N cancels]
        - dL/dB must match B.shape (K, N):
          A.T (K, M) @ G (M, N) -> (K, N)  [Inner dimension M cancels]

    Args:
        grad: Incoming gradient Tensor G of shape (M, N).
        left: Matrix A of shape (M, K).
        right: Matrix B of shape (K, N).

    Returns:
        Tuple of (dL/dA, dL/dB) matching shapes of A and B respectively.
    """

    g_val = grad.value
    l_val = left.value
    r_val = right.value

    # Case 1: Dot product of two 1D vectors (output is a 0-D scalar)
    # y = u @ v -> dy/du = v, dy/dv = u
    if g_val.ndim == 0 or (l_val.ndim == 1 and r_val.ndim == 1):
        d_left = grad * right
        d_right = grad * left
        return d_left, d_right

    # Case 2: Matrix @ 1D vector: (B, D) @ (D,) -> (B,)
    if l_val.ndim == 2 and r_val.ndim == 1:
        # dL/dX = outer product G @ w^T: (B, 1) * (1, D) -> (B, D)
        g_2d = Tensor(g_val.reshape(-1, 1))
        r_2d = Tensor(r_val.reshape(1, -1))
        d_left = g_2d @ r_2d
        # dL/dw = X^T @ G: (D, B) @ (B,) -> (D,)
        d_right = left.transpose() @ grad
        return d_left, d_right

    # Case 3: Standard 2D or batched matrix multiplication
    d_left = grad @ right.transpose()
    d_right = left.transpose() @ grad
    return d_left, d_right

mean_backward

mean_backward(grad, x, axis=None)

Compute the Vector-Jacobian Product (VJP) for tensor mean reduction.

Mathematical Intuition

The arithmetic mean is a scaled summation: mean(X) = (1 / N) * sum(X) where N is the number of elements collapsed along the reduced axes.

By the chain rule, every element that contributed to the mean receives an equal fractional share of the incoming gradient: dL/dX_i = (1 / N) * dL/dY

Therefore, mean_backward is simply sum_backward scaled by (1 / N): dL/dX = (1 / N) * sum_backward(grad, x, axis)

Broadcasting & Normalization Mechanics: 1. Determine N (the count of elements reduced): - If axis is None: N is the total count of all elements in X. - If axis is specified (e.g. axis=0 on shape (4, 3)): N is the product of the lengths of the reduced dimensions (here N = 4). 2. Leverage sum_backward to re-insert singleton dimensions and stretch the gradient back to x.shape. 3. Divide the broadcasted gradient by N.

Parameters:

Name	Type	Description	Default
`grad`	`Tensor`	Incoming gradient Tensor from downstream nodes.	required
`x`	`Tensor`	The parent Tensor that was reduced.	required
`axis`	`int \| tuple[int, ...] \| None`	The axis or axes along which the mean was computed.	`None`

Returns:

Type	Description
`Tensor`	Tensor of shape `x.shape` scaled by the inverse element count (1 / N).

Source code in utorch/autograd/ops.py

def mean_backward(
    grad: Tensor,
    x: Tensor,
    axis: int | tuple[int, ...] | None = None,
) -> Tensor:
    """Compute the Vector-Jacobian Product (VJP) for tensor mean reduction.

    Mathematical Intuition:
        The arithmetic mean is a scaled summation:
            mean(X) = (1 / N) * sum(X)
        where N is the number of elements collapsed along the reduced axes.

        By the chain rule, every element that contributed to the mean receives
        an equal fractional share of the incoming gradient:
            dL/dX_i = (1 / N) * dL/dY

        Therefore, mean_backward is simply sum_backward scaled by (1 / N):
            dL/dX = (1 / N) * sum_backward(grad, x, axis)

    Broadcasting & Normalization Mechanics:
        1. Determine N (the count of elements reduced):
           - If axis is None: N is the total count of all elements in X.
           - If axis is specified (e.g. axis=0 on shape (4, 3)): N is the
             product of the lengths of the reduced dimensions (here N = 4).
        2. Leverage sum_backward to re-insert singleton dimensions and stretch
           the gradient back to x.shape.
        3. Divide the broadcasted gradient by N.

    Args:
        grad: Incoming gradient Tensor from downstream nodes.
        x: The parent Tensor that was reduced.
        axis: The axis or axes along which the mean was computed.

    Returns:
        Tensor of shape `x.shape` scaled by the inverse element count (1 / N).
    """
    x_shape = x.shape

    # Step 1: Calculate the normalization factor N (number of collapsed elements).
    if axis is None:
        # Total mean: all elements in the entire array were averaged together.
        num_elements = int(np.prod(x_shape)) if x_shape else 1
    else:
        # Axis-specific mean: count elements along the specified reduced dimensions.
        axes = (axis,) if isinstance(axis, int) else tuple(axis)
        pos_axes = tuple(a if a >= 0 else len(x_shape) + a for a in axes)
        num_elements = int(np.prod([x_shape[a] for a in pos_axes]))

    # Step 2: Use sum_backward to broadcast grad back to the original x_shape.
    grad_sum = sum_backward(grad, x, axis)

    # Step 3: Scale by 1 / N so each contributing element receives its share.
    return Tensor(grad_sum.value / float(num_elements))

sum_backward

sum_backward(grad, x, axis=None)

Compute the Vector-Jacobian Product (VJP) for tensor summation.

Mathematical Intuition

Summation distributes incoming gradients equally across all elements that contributed to the sum. For Y = sum(X), dY/dX_i = 1.0. Therefore: dL/dX = dL/dY * 1.0 (broadcasted to X's original shape).

Broadcasting Mechanics

When an axis is collapsed during sum(), that dimension is eliminated from the output shape (e.g., shape (2, 4, 3) reduced along axis=1 becomes shape (2, 3)).

To broadcast the incoming gradient back to (2, 4, 3), we must: 1. Normalize negative indices (e.g., axis=-1 becomes axis=2). 2. Re-insert a singleton dimension (size 1) at the collapsed axes: (2, 3) -> reshape -> (2, 1, 3). 3. Use np.broadcast_to to stretch size 1 back to the original size 4: (2, 1, 3) -> broadcast -> (2, 4, 3).

Source code in utorch/autograd/ops.py

def sum_backward(
    grad: Tensor,
    x: Tensor,
    axis: int | tuple[int, ...] | None = None,
) -> Tensor:
    """Compute the Vector-Jacobian Product (VJP) for tensor summation.

    Mathematical Intuition:
        Summation distributes incoming gradients equally across all elements
        that contributed to the sum. For Y = sum(X), dY/dX_i = 1.0.
        Therefore: dL/dX = dL/dY * 1.0 (broadcasted to X's original shape).

    Broadcasting Mechanics:
        When an axis is collapsed during sum(), that dimension is eliminated
        from the output shape (e.g., shape (2, 4, 3) reduced along axis=1
        becomes shape (2, 3)).

        To broadcast the incoming gradient back to (2, 4, 3), we must:
        1. Normalize negative indices (e.g., axis=-1 becomes axis=2).
        2. Re-insert a singleton dimension (size 1) at the collapsed axes:
           (2, 3) -> reshape -> (2, 1, 3).
        3. Use np.broadcast_to to stretch size 1 back to the original size 4:
           (2, 1, 3) -> broadcast -> (2, 4, 3).
    """
    x_shape = x.shape

    # Scenario 1: Total sum across all elements (axis is None).
    # The output was a scalar; every element in X receives the incoming gradient.
    if axis is None:
        return Tensor(np.broadcast_to(grad.value, x_shape).copy())

    # Scenario 2: Sum along specific axis or axes.
    # Step A: Normalize input axis into a tuple of positive dimension indices.
    # (e.g., for a 3D tensor, axis=-1 is converted to 3 + (-1) = 2).
    axes = (axis,) if isinstance(axis, int) else tuple(axis)
    pos_axes = tuple(a if a >= 0 else len(x_shape) + a for a in axes)

    # Step B: Reconstruct shape with singleton dimensions (1) at reduced positions.
    # Example: If x_shape is (2, 4, 3) and pos_axes is (1,),
    # expanded_shape becomes [2, 1, 3].
    expanded_shape = list(x_shape)
    for a in pos_axes:
        expanded_shape[a] = 1

    # Step C: Reshape gradient to include the singleton dimensions, then stretch.
    # grad.value (2, 3) -> reshaped to (2, 1, 3) -> broadcasted to (2, 4, 3).
    grad_reshaped = np.reshape(grad.value, expanded_shape)
    return Tensor(np.broadcast_to(grad_reshaped, x_shape).copy())

tensor

Computational graph node and tensor abstraction.

Tensor

Tensor(value=None, parents=None, fun=None, name=None, args=None)

A multidimensional array and computational graph node.

Tensor tracks numerical data alongside the operations and parent nodes that produced it, dynamically building a Directed Acyclic Graph (DAG) during the forward pass.

Parameters:

Name	Type	Description	Default
`value`	`Any`	Numerical data (scalar, list, or NumPy ndarray). If None, the node is initialized as an intermediate operation node.	`None`
`parents`	`Sequence[Tensor] \| None`	Upstream antecedent nodes that fed into this computation.	`None`
`fun`	`Callable[..., Any] \| None`	The forward mathematical function evaluated at this node.	`None`
`name`	`str \| None`	Optional human-readable identifier for visualization.	`None`
`args`	`Any`	Optional extra static arguments passed to `fun` (e.g. axes, min/max).	`None`

Attributes:

Name	Type	Description
`value`	`ndarray`	The evaluated forward output tensor.
`parents`	`list[Tensor]`	Direct input nodes in the computational graph.
`fun`	`Callable \| None`	The mathematical operation that generated this node.
`args`	`Any`	Additional constants or axes supplied to the operation.
`name`	`str \| None`	Label used in graph diagrams and debugging.
`grad`	`int \| float \| ndarray`	Accumulated gradient (computed in Branch 3+).

Example

import utorch as ut

x = ut.Tensor(2.0, name="x")
y = ut.Tensor(3.0, name="y")
z = x * y + 1.0

print(z.value)   # 7.0
print(z.is_leaf) # False

Initialize a computational graph node via leaf or inner node dispatch.

In a dynamic Directed Acyclic Graph (DAG), every node belongs to one of two fundamental categories:

Leaf Nodes (Inputs, Constants, Parameters):
Created directly by the user with a known starting value.
Have no antecedent parents (parents = []) and no creator function (fun = None).
Represent independent variables at the base of the tree (e.g., weights, biases, inputs).
Dispatched to: _init_if_leaf(value).
Inner Nodes (Intermediate Operations):
Created automatically as the output of an operation (e.g., +, *, relu).
Have antecedent parent nodes (parents) and an associated mathematical function (fun).
Numerical value is evaluated dynamically during the forward pass by applying fun(*parent_values).
Dispatched to: _init_as_inner_node(parents, fun, args).

Parameters:

Name	Type	Description	Default
`value`	`Any`	Numerical data (scalar int/float, list, or NumPy array). If provided, initialized as a leaf node.	`None`
`parents`	`Sequence[Tensor] \| None`	Upstream antecedent `Tensor` nodes that fed into this operation. Used when initializing an inner node.	`None`
`fun`	`Callable[..., Any] \| None`	The forward mathematical function evaluated at this node (e.g., `np.add`, `np.sin`). Required for inner nodes; must be `None` for leaf nodes.	`None`
`name`	`str \| None`	Optional human-readable identifier for visualization.	`None`
`args`	`Any`	Optional static constants passed to `fun` (e.g., reduction axis, power exponent).	`None`

Source code in utorch/autograd/tensor.py

def __init__(
    self,
    value: Any = None,
    parents: Sequence[Tensor] | None = None,
    fun: Callable[..., Any] | None = None,
    name: str | None = None,
    args: Any = None,
) -> None:
    r"""Initialize a computational graph node via leaf or inner node dispatch.

    In a dynamic Directed Acyclic Graph (DAG), every node belongs to one of
    two fundamental categories:

    1. **Leaf Nodes (Inputs, Constants, Parameters)**:
       - Created directly by the user with a known starting `value`.
       - Have **no antecedent parents** (`parents = []`) and **no creator
         function** (`fun = None`).
       - Represent independent variables at the base of the tree (e.g.,
         weights, biases, inputs).
       - Dispatched to: `_init_if_leaf(value)`.

    2. **Inner Nodes (Intermediate Operations)**:
       - Created automatically as the output of an operation (e.g., `+`,
         `*`, `relu`).
       - Have **antecedent parent nodes** (`parents`) and an **associated
         mathematical function** (`fun`).
       - Numerical value is evaluated dynamically during the forward pass by
         applying `fun(*parent_values)`.
       - Dispatched to: `_init_as_inner_node(parents, fun, args)`.

    Args:
        value: Numerical data (scalar int/float, list, or NumPy array).
            If provided, initialized as a **leaf node**.
        parents: Upstream antecedent `Tensor` nodes that fed into this
            operation. Used when initializing an **inner node**.
        fun: The forward mathematical function evaluated at this node
            (e.g., `np.add`, `np.sin`). Required for inner nodes; must be
            `None` for leaf nodes.
        name: Optional human-readable identifier for visualization.
        args: Optional static constants passed to `fun` (e.g., reduction
            axis, power exponent).

    """
    self.name: str | None = name
    if value is not None:
        self._init_if_leaf(value)
    else:
        self._init_as_inner_node(parents or [], fun, args)
    if value is not None:
        self._init_if_leaf(value)
    else:
        self._init_as_inner_node(parents or [], fun, args)

    if value is not None:
        self._init_if_leaf(value)
    else:
        self._init_as_inner_node(parents or [], fun, args)

    self.grad = None

is_leaf `property`

is_leaf

Return True if this node has no parents (user input or parameter).

shape `property`

shape

Return the dimensional shape of the underlying tensor.

add

__add__(other)

Element-wise addition: self + other.

Source code in utorch/autograd/tensor.py

def __add__(self, other: Any) -> Tensor:
    """Element-wise addition: `self + other`."""
    return self._two_variable_operator(other, lambda x, y: np.add(x, y))

eq

__eq__(other)

Element-wise equal: self == other.

Source code in utorch/autograd/tensor.py

def __eq__(self, other: Any) -> Any:
    """Element-wise equal: `self == other`."""
    if not isinstance(other, Tensor):
        return False
    return np.array_equal(self.value, other.value)

ge

__ge__(other)

Element-wise greater or equal: self >= other.

Source code in utorch/autograd/tensor.py

def __ge__(self, other: Any) -> Any:
    """Element-wise greater or equal: `self >= other`."""
    other_val = other.value if isinstance(other, Tensor) else other
    return np.greater_equal(self.value, other_val)

gt

__gt__(other)

Element-wise greater than: self > other.

Source code in utorch/autograd/tensor.py

def __gt__(self, other: Any) -> Any:
    """Element-wise greater than: `self > other`."""
    other_val = other.value if isinstance(other, Tensor) else other
    return np.greater(self.value, other_val)

hash

__hash__()

Calculates hash(value)`.

Source code in utorch/autograd/tensor.py

def __hash__(self) -> int:
    """Calculates hash(value)`."""
    return hash(id(self))

le

__le__(other)

Element-wise less or equal: self <= other.

Source code in utorch/autograd/tensor.py

def __le__(self, other: Any) -> Any:
    """Element-wise less or equal: `self <= other`."""
    other_val = other.value if isinstance(other, Tensor) else other
    return np.less_equal(self.value, other_val)

lt

__lt__(other)

Element-wise less than: self < other.

Source code in utorch/autograd/tensor.py

def __lt__(self, other: Any) -> Any:
    """Element-wise less than: `self < other`."""
    other_val = other.value if isinstance(other, Tensor) else other
    return np.less(self.value, other_val)

matmul

__matmul__(other)

Matrix multiplication: self @ other.

Source code in utorch/autograd/tensor.py

def __matmul__(self, other: Any) -> Tensor:
    """Matrix multiplication: `self @ other`."""
    return self._two_variable_operator(other, lambda x, y: np.matmul(x, y))

mul

__mul__(other)

Element-wise multiplication: self * other.

Source code in utorch/autograd/tensor.py

def __mul__(self, other: Any) -> Tensor:
    """Element-wise multiplication: `self * other`."""
    return self._two_variable_operator(other, lambda x, y: np.multiply(x, y))

neg

__neg__()

Element-wise negation: -self.

Source code in utorch/autograd/tensor.py

def __neg__(self) -> Tensor:
    """Element-wise negation: `-self`."""
    return self.single_operator(lambda x: np.negative(x))

pow

__pow__(exponent)

Element-wise power: self ** exponent.

Source code in utorch/autograd/tensor.py

def __pow__(self, exponent: int | float) -> Tensor:
    """Element-wise power: `self ** exponent`."""
    return self.single_operator(lambda x, p: np.power(x, p), args=exponent)

radd

__radd__(other)

Right-hand element-wise addition: other + self.

Source code in utorch/autograd/tensor.py

def __radd__(self, other: Any) -> Tensor:
    """Right-hand element-wise addition: `other + self`."""
    return self.__add__(other)

repr

__repr__()

Return a string representation of the tensor for inspection.

Source code in utorch/autograd/tensor.py

def __repr__(self) -> str:
    """Return a string representation of the tensor for inspection."""
    name_str = f"'{self.name}', " if self.name else ""
    if self.is_leaf:
        return f"ut.Tensor({name_str}value={self.value})"
    return (
        f"ut,.Tensor({name_str}value={self.value}, "
        f"fun={_get_function_name(self.fun)})"  # type: ignore[arg-type]
    )

rmul

__rmul__(other)

Right-hand element-wise multiplication: other * self.

Source code in utorch/autograd/tensor.py

def __rmul__(self, other: Any) -> Tensor:
    """Right-hand element-wise multiplication: `other * self`."""
    return self.__mul__(other)

rsub

__rsub__(other)

Right-hand element-wise subtraction: other - self.

Source code in utorch/autograd/tensor.py

def __rsub__(self, other: Any) -> Tensor:
    """Right-hand element-wise subtraction: `other - self`."""
    other_var = self._ensure_tensor(other)
    return other_var._two_variable_operator(self, lambda x, y: np.subtract(x, y))

rtruediv

__rtruediv__(other)

Right-hand element-wise division: other / self.

Source code in utorch/autograd/tensor.py

def __rtruediv__(self, other: Any) -> Tensor:
    """Right-hand element-wise division: `other / self`."""
    other_var = self._ensure_tensor(other)
    return other_var._two_variable_operator(self, lambda x, y: np.divide(x, y))

sub

__sub__(other)

Element-wise subtraction: self - other.

Source code in utorch/autograd/tensor.py

def __sub__(self, other: Any) -> Tensor:
    """Element-wise subtraction: `self - other`."""
    return self._two_variable_operator(other, lambda x, y: np.subtract(x, y))

truediv

__truediv__(other)

Element-wise division: self / other.

Source code in utorch/autograd/tensor.py

def __truediv__(self, other: Any) -> Tensor:
    """Element-wise division: `self / other`."""
    return self._two_variable_operator(other, lambda x, y: np.divide(x, y))

abs

abs()

Element-wise absolute value: \(|x|\).

Source code in utorch/autograd/tensor.py

def abs(self) -> Tensor:
    r"""Element-wise absolute value: $|x|$."""
    return self.single_operator(lambda x: np.abs(x))

backward

backward(grad_tensor=None)

Execute reverse-mode automatic differentiation (backpropagation).

Computes the gradient of this output tensor with respect to all upstream nodes in the computational graph using the multivariate chain rule.

Because gradients are accumulated as Tensor instances rather than raw floats, the backward pass dynamically constructs a secondary computation graph, enabling arbitrary higher-order derivatives (e.g., double backward for Hessians).

The backward algorithm executes in five distinct pedagogical phases:

Scalar Verification & Seed Initialization:
Reverse-mode autograd starts at a scalar objective (e.g., loss).
The root gradient is seeded with the identity: dL/dL = 1.0.
If an explicit grad_tensor is supplied, it is used as the seed.
Reverse Topological Ordering:
Performs a Depth-First Search (DFS) from self backwards to all reachable ancestor nodes.
Arranges nodes so every child appears strictly before its parents.
Guarantees that when a node is processed, it has already collected all incoming gradient contributions from all downstream paths.
Primitive Derivative Dispatch:
Traverses nodes in reverse topological order.
For each intermediate operation node, looks up its analytical derivative rule in SCALAR_PRIMITIVES using node.op.
Local Chain Rule Evaluation:
For binary operations z = f(x, y): dx = d_rule_left(grad_z, x, y) dy = d_rule_right(grad_z, x, y)
For unary operations y = f(x): dx = d_rule(grad_y, x)
Gradient Accumulation (Multivariate Chain Rule):
Local gradient contributions are added to antecedent parents using += (parent.grad = parent.grad + d_parent).
Accumulation (rather than overwriting) is mathematically mandatory when a variable is reused across multiple paths (diamond graphs).

Parameters:

Name	Type	Description	Default
`grad_tensor`	`Tensor \| None`	Seed gradient for this node. If None, defaults to Tensor(1.0).	`None`

Raises:

Type	Description
`RuntimeError`	If called on a non-scalar tensor without an explicit `grad_tensor`.
`ValueError`	If an operation node has no matching primitive rule in `SCALAR_PRIMITIVES`.

Examples:

import utorch as ut

# 1. First-order derivative of a polynomial: y = x^2 + 3x
x = ut.Tensor(4.0, name="x")
y = (x ** 2) + 3.0 * x  # dy/dx = 2x + 3 = 11.0
y.backward()
print(x.grad.value)     # 11.0

# 2. Second-order derivative (double backward): d2y/dx2 = 2.0
first_grad = x.grad
x.zero_grad()
first_grad.backward()
print(x.grad.value)     # 2.0

Source code in utorch/autograd/tensor.py

def backward(self, grad_tensor: Tensor | None = None) -> None:
    """Execute reverse-mode automatic differentiation (backpropagation).

    Computes the gradient of this output tensor with respect to all upstream
    nodes in the computational graph using the multivariate chain rule.

    Because gradients are accumulated as `Tensor` instances rather than raw
    floats, the backward pass dynamically constructs a secondary computation
    graph, enabling arbitrary higher-order derivatives (e.g., double backward
    for Hessians).

    The backward algorithm executes in five distinct pedagogical phases:

    1. **Scalar Verification & Seed Initialization**:
       - Reverse-mode autograd starts at a scalar objective (e.g., loss).
       - The root gradient is seeded with the identity: dL/dL = 1.0.
       - If an explicit `grad_tensor` is supplied, it is used as the seed.

    2. **Reverse Topological Ordering**:
       - Performs a Depth-First Search (DFS) from `self` backwards to all
         reachable ancestor nodes.
       - Arranges nodes so every child appears strictly before its parents.
       - Guarantees that when a node is processed, it has already collected
         all incoming gradient contributions from all downstream paths.

    3. **Primitive Derivative Dispatch**:
       - Traverses nodes in reverse topological order.
       - For each intermediate operation node, looks up its analytical
         derivative rule in `SCALAR_PRIMITIVES` using `node.op`.

    4. **Local Chain Rule Evaluation**:
       - For binary operations z = f(x, y):
         dx = d_rule_left(grad_z, x, y)
         dy = d_rule_right(grad_z, x, y)
       - For unary operations y = f(x):
         dx = d_rule(grad_y, x)

    5. **Gradient Accumulation (Multivariate Chain Rule)**:
       - Local gradient contributions are added to antecedent parents using
         `+=` (`parent.grad = parent.grad + d_parent`).
       - Accumulation (rather than overwriting) is mathematically mandatory
         when a variable is reused across multiple paths (diamond graphs).

    Args:
        grad_tensor: Seed gradient for this node. If None, defaults to
            Tensor(1.0).

    Raises:
        RuntimeError: If called on a non-scalar tensor without an explicit
            `grad_tensor`.
        ValueError: If an operation node has no matching primitive rule in
            `SCALAR_PRIMITIVES`.

    Examples:
        ```python
        import utorch as ut

        # 1. First-order derivative of a polynomial: y = x^2 + 3x
        x = ut.Tensor(4.0, name="x")
        y = (x ** 2) + 3.0 * x  # dy/dx = 2x + 3 = 11.0
        y.backward()
        print(x.grad.value)     # 11.0

        # 2. Second-order derivative (double backward): d2y/dx2 = 2.0
        first_grad = x.grad
        x.zero_grad()
        first_grad.backward()
        print(x.grad.value)     # 2.0
        ```
    """
    from utorch.autograd.ops import PRIMITIVES
    from utorch.autograd.unbroadcast import unbroadcast

    # 1. Seed root gradient as a Tensor node
    if grad_tensor is None:
        self.grad = Tensor(1.0)  # type: ignore[assignment]
    else:
        self.grad = grad_tensor  # type: ignore[assignment]

    # 2. Get reverse topological evaluation order (root first, down to leaves)
    ordered_nodes = topological_sort(self)

    # 3. Propagate gradients backwards
    for node in ordered_nodes:
        # Leaf nodes have no parents to propagate into
        if not node.parents:
            continue

        op_key = getattr(node, "op", None) or _get_function_name(node.fun)
        if op_key not in PRIMITIVES:
            raise ValueError(f"No derivative registered for op: '{op_key}'")

        rule = PRIMITIVES[op_key]

        # Binary operations (2 parents)
        if len(node.parents) == 2:
            parent_left, parent_right = node.parents

            # Support both a list [d_left, d_right]
            # or a function returning (d_left, d_right)
            if isinstance(rule, list):
                d_left = rule[0](node.grad, parent_left, parent_right, node.args)
                d_right = rule[1](node.grad, parent_left, parent_right, node.args)
            else:
                d_left, d_right = rule(
                    node.grad, parent_left, parent_right, node.args
                )
            # reverse broadcasting
            d_left = unbroadcast(d_left, parent_left.shape)
            d_right = unbroadcast(d_right, parent_right.shape)

            # Accumulate gradients
            # (+= records computation graph if grad is a Tensor)
            parent_left.grad = (
                d_left if parent_left.grad is None else parent_left.grad + d_left
            )
            parent_right.grad = (
                d_right
                if parent_right.grad is None
                else parent_right.grad + d_right
            )

        # Unary operations (1 parent)
        elif len(node.parents) == 1:
            parent = node.parents[0]
            rule_fn = rule[0] if isinstance(rule, list) else rule
            d_parent = rule_fn(node.grad, parent, node.args)
            d_parent = unbroadcast(d_parent, parent.shape)
            parent.grad = (
                d_parent if parent.grad is None else parent.grad + d_parent
            )

clip_min

clip_min(min_val)

Clip tensor values below a minimum threshold.

Source code in utorch/autograd/tensor.py

def clip_min(self, min_val: float) -> Tensor:
    """Clip tensor values below a minimum threshold."""
    return self.single_operator(
        lambda x, m: np.clip(x, a_min=m, a_max=None), args=min_val
    )

cos

cos()

Element-wise cosine: \(\\cos(x)\).

Source code in utorch/autograd/tensor.py

def cos(self) -> Tensor:
    r"""Element-wise cosine: $\\cos(x)$."""
    return self.single_operator(lambda x: np.cos(x))

cosh

cosh()

Element-wise hyperbolic cosine: \(\\cosh(x)\).

Source code in utorch/autograd/tensor.py

def cosh(self) -> Tensor:
    r"""Element-wise hyperbolic cosine: $\\cosh(x)$."""
    return self.single_operator(lambda x: np.cosh(x))

exp

exp()

Element-wise exponential: \(\\exp(x)\).

Source code in utorch/autograd/tensor.py

def exp(self) -> Tensor:
    r"""Element-wise exponential: $\\exp(x)$."""
    return self.single_operator(lambda x: np.exp(x))

log

log()

Element-wise natural logarithm: \(\\ln(x)\).

Source code in utorch/autograd/tensor.py

def log(self) -> Tensor:
    r"""Element-wise natural logarithm: $\\ln(x)$."""
    return self.single_operator(lambda x: np.log(x))

mean

mean(axis=None)

Compute the arithmetic mean of tensor elements across specified axis.

Source code in utorch/autograd/tensor.py

def mean(self, axis: int | tuple[int, ...] | None = None) -> Tensor:
    """Compute the arithmetic mean of tensor elements across specified axis."""
    if axis is None:
        return self.single_operator(lambda x: np.mean(x), args=None)
    return self.single_operator(lambda x, ax: np.mean(x, axis=ax), args=axis)

relu

relu()

Rectified Linear Unit activation: \(\\max(0, x)\).

Source code in utorch/autograd/tensor.py

def relu(self) -> Tensor:
    r"""Rectified Linear Unit activation: $\\max(0, x)$."""
    return self.single_operator(lambda x: np.maximum(0, x))

sigmoid

sigmoid()

Sigmoid activation function: \(\\sigma(x) = \\frac{1}{1 + e^{-x}}\).

Source code in utorch/autograd/tensor.py

def sigmoid(self) -> Tensor:
    r"""Sigmoid activation function: $\\sigma(x) = \\frac{1}{1 + e^{-x}}$."""
    return self.single_operator(lambda x: 1.0 / (1.0 + np.exp(-x)))

sign

sign()

Element-wise sign indicator (-1, 0, or 1).

Source code in utorch/autograd/tensor.py

def sign(self) -> Tensor:
    r"""Element-wise sign indicator (-1, 0, or 1)."""
    return self.single_operator(lambda x: np.sign(x))

sin

sin()

Element-wise sine: \(\\sin(x)\).

Source code in utorch/autograd/tensor.py

def sin(self) -> Tensor:
    r"""Element-wise sine: $\\sin(x)$."""
    return self.single_operator(lambda x: np.sin(x))

single_operator

single_operator(fun, args=None)

Apply a unary operation taking self as the sole parent node.

Parameters:

Name	Type	Description	Default
`fun`	`Callable[..., Any]`	Unary mathematical function (e.g. np.sin, np.exp).	required
`args`	`Any`	Optional static parameters.	`None`

Returns:

Type	Description
`Tensor`	A new Tensor representing the operation's output.

Source code in utorch/autograd/tensor.py

def single_operator(self, fun: Callable[..., Any], args: Any = None) -> Tensor:
    """Apply a unary operation taking self as the sole parent node.

    Args:
        fun: Unary mathematical function (e.g. np.sin, np.exp).
        args: Optional static parameters.

    Returns:
        A new Tensor representing the operation's output.
    """
    return Tensor(parents=[self], fun=fun, args=args)

sum

sum(axis=None)

Sum tensor elements across specified axis.

Source code in utorch/autograd/tensor.py

def sum(self, axis: int | tuple[int, ...] | None = None) -> Tensor:
    """Sum tensor elements across specified axis."""
    if axis is None:
        return self.single_operator(lambda x: np.sum(x), args=None)
    return self.single_operator(lambda x, ax: np.sum(x, axis=ax), args=axis)

tanh

tanh()

Element-wise hyperbolic tangent: \(\\tanh(x)\).

Source code in utorch/autograd/tensor.py

def tanh(self) -> Tensor:
    r"""Element-wise hyperbolic tangent: $\\tanh(x)$."""
    return self.single_operator(lambda x: np.tanh(x))

transpose

transpose(axes=None)

Permute the dimensions of the tensor.

Source code in utorch/autograd/tensor.py

def transpose(self, axes: Sequence[int] | None = None) -> Tensor:
    """Permute the dimensions of the tensor."""
    if axes is None:
        return self.single_operator(lambda x: np.transpose(x), args=None)
    return self.single_operator(lambda x, ax: np.transpose(x, axes=ax), args=axes)

zero_grad

zero_grad()

Reset gradient to zero on this node.

Source code in utorch/autograd/tensor.py

def zero_grad(self) -> None:
    """Reset gradient to zero on this node."""
    self.grad = None

topological_sort

Graph traversal and topological sorting for the autograd engine.

topological_sort

topological_sort(root)

Return nodes in reverse topological order (root first, down to leaves).

Uses DFS with a visited set so shared parent nodes (diamond dependencies) are processed only after all their dependent children have been evaluated.

Parameters:

Name	Type	Description	Default
`root`	`Any`	The output node from which to trace backwards.	required

Returns:

Type	Description
`list[Any]`	List of nodes starting with root and ending at leaf nodes.

Source code in utorch/autograd/topological_sort.py

def topological_sort(root: Any) -> list[Any]:
    """Return nodes in reverse topological order (root first, down to leaves).

    Uses DFS with a visited set so shared parent nodes (diamond dependencies)
    are processed only after all their dependent children have been evaluated.

    Args:
        root: The output node from which to trace backwards.

    Returns:
        List of nodes starting with root and ending at leaf nodes.
    """
    ordered: list[Any] = []
    visited: set[int] = set()

    def dfs(node: Any) -> None:
        node_id = id(node)
        if node_id in visited:
            return
        visited.add(node_id)
        for parent in node.parents:
            dfs(parent)
        ordered.append(node)

    dfs(root)
    # Reverse so root is at index 0 and leaves are at the end
    return list(reversed(ordered))

unbroadcast

unbroadcast(grad, target_shape)

Project a broadcasted gradient tensor back to its input operand shape.

During forward evaluation of binary operations (e.g., A + B), smaller operands are implicitly expanded via NumPy broadcasting rules (e.g., a bias vector of shape (10,) added to a batch matrix of shape (32, 10)).

In the backward pass, incoming gradients match the expanded output shape (32, 10). To maintain dimensional alignment, the gradient must be summed across all broadcasted axes so that grad.shape == target_shape.

Parameters:

Name	Type	Description	Default
`grad`	`Tensor`	Incoming gradient Tensor with the expanded shape.	required
`target_shape`	`tuple[int, ...]`	The original shape of the parent input Tensor.	required

Returns:

Type	Description
`Tensor`	A new Tensor reduced to match `target_shape`.

Examples:

import numpy as np
import utorch as ut
from utorch.autograd.ops import unbroadcast

# Batch gradient of shape (32, 10)
grad = ut.Tensor(np.ones((32, 10)))

# Unbroadcast to bias vector shape (10,)
bias_grad = unbroadcast(grad, (10,))
print(bias_grad.shape)  # (10,)
print(bias_grad.value[0])  # 32.0 (summed across batch dimension 0)

Source code in utorch/autograd/unbroadcast.py

def unbroadcast(grad: ut.Tensor, target_shape: tuple[int, ...]) -> ut.Tensor:
    """Project a broadcasted gradient tensor back to its input operand shape.

    During forward evaluation of binary operations (e.g., A + B), smaller
    operands are implicitly expanded via NumPy broadcasting rules (e.g., a bias
    vector of shape (10,) added to a batch matrix of shape (32, 10)).

    In the backward pass, incoming gradients match the expanded output shape
    (32, 10). To maintain dimensional alignment, the gradient must be summed
    across all broadcasted axes so that grad.shape == target_shape.

    Args:
        grad: Incoming gradient Tensor with the expanded shape.
        target_shape: The original shape of the parent input Tensor.

    Returns:
        A new Tensor reduced to match `target_shape`.

    Examples:
        ```python
        import numpy as np
        import utorch as ut
        from utorch.autograd.ops import unbroadcast

        # Batch gradient of shape (32, 10)
        grad = ut.Tensor(np.ones((32, 10)))

        # Unbroadcast to bias vector shape (10,)
        bias_grad = unbroadcast(grad, (10,))
        print(bias_grad.shape)  # (10,)
        print(bias_grad.value[0])  # 32.0 (summed across batch dimension 0)
        ```
    """
    grad_val = grad.value
    if grad_val.shape == target_shape:
        return grad

    grad_ndim = grad_val.ndim
    target_ndim = len(target_shape)

    # 1. Sum over extra leading dimensions (e.g., (32, 10) -> (10,))
    if grad_ndim > target_ndim:
        leading_axes = tuple(range(grad_ndim - target_ndim))
        grad_val = np.sum(grad_val, axis=leading_axes)

    # 2. Sum over size-1 dimensions where broadcasting expanded size > 1
    axes_to_keep: list[int] = []
    for i, (g_dim, t_dim) in enumerate(zip(grad_val.shape, target_shape, strict=False)):
        if t_dim == 1 and g_dim > 1:
            axes_to_keep.append(i)

    if axes_to_keep:
        grad_val = np.sum(grad_val, axis=tuple(axes_to_keep), keepdims=True)

    return ut.Tensor(grad_val.reshape(target_shape))

utorch.autograd

utorch.autograd

__all__ module-attribute

Tensor

is_leaf property

shape property

__add__

__eq__

__ge__

__gt__

__hash__

__le__

__lt__

__matmul__

__mul__

__neg__

__pow__

__radd__

__repr__

__rmul__

__rsub__

__rtruediv__

__sub__

__truediv__

abs

backward

clip_min

cos

cosh

exp

log

mean

relu

sigmoid

sign

sin

single_operator

sum

tanh

transpose

zero_grad

ops

matmul_backward

mean_backward

sum_backward

tensor

Tensor

is_leaf property

shape property

__add__

__eq__

__ge__

__gt__

__hash__

__le__

__lt__

__matmul__

__mul__

__neg__

__pow__

__radd__

__repr__

__rmul__

__rsub__

__rtruediv__

__sub__

__truediv__

abs

backward

clip_min

cos

cosh

exp

log

mean

relu

sigmoid

sign

sin

single_operator

all `module-attribute`

is_leaf `property`

shape `property`

add

eq

ge

gt

hash

le

lt

matmul

mul

neg

pow

radd

repr

rmul

rsub

rtruediv

sub

truediv

is_leaf `property`

shape `property`

add

eq

ge

gt

hash

le

lt

matmul

mul

neg

pow

radd

repr

rmul

rsub

rtruediv

sub

truediv