grilly.optim.natural_gradient

Natural Gradient Optimizer

Uses: fisher-natural-gradient.glsl

Implements natural gradient descent using Fisher information matrix.

Classes

Iterator()

NaturalGradient(params[, lr, ...])

Natural Gradient optimizer using Fisher information matrix.

Optimizer(params, defaults)

Base class for all optimizers.
class grilly.optim.natural_gradient.NaturalGradient(params, lr=0.001, fisher_momentum=0.9, use_gpu=True)[source]

Bases: Optimizer

Natural Gradient optimizer using Fisher information matrix.

Uses: fisher-natural-gradient.glsl

Implements natural gradient descent: - F = Fisher information matrix - param = param - lr * F^(-1) * grad

Reference: grilly/backend/learning.py natural_gradient

Initialize Natural Gradient optimizer.

Parameters

  • params (Iterator[numpy.ndarray]) – Iterator of parameter arrays to optimize

  • lr (float) – Learning rate (default: 1e-3)

  • fisher_momentum (float) – Momentum for Fisher information estimate (default: 0.9)

  • use_gpu (bool) – Whether to use GPU acceleration (default: True)

__init__(params, lr=0.001, fisher_momentum=0.9, use_gpu=True)[source]

Initialize Natural Gradient optimizer.

Parameters

  • params (Iterator[numpy.ndarray]) – Iterator of parameter arrays to optimize

  • lr (float) – Learning rate (default: 1e-3)

  • fisher_momentum (float) – Momentum for Fisher information estimate (default: 0.9)

  • use_gpu (bool) – Whether to use GPU acceleration (default: True)

Dependencies: None detected from callable globals.

Variables: params (collections.abc.Iterator[numpy.ndarray], required); lr (float, optional, default 0.001); fisher_momentum (float, optional, default 0.9); use_gpu (bool, optional, default True).

Usage Example

import numpy as np
from grilly.optim.natural_gradient import NaturalGradient

instance = NaturalGradient(...)
result = instance.__init__(params=np.zeros(1, dtype=np.float32), lr=0.001, fisher_momentum=0.9, use_gpu=True)
_get_backend()[source]

Get or create backend instance

Dependencies: None detected from callable globals.

Variables: This callable does not take explicit input variables.

Usage Example

from grilly.optim.natural_gradient import NaturalGradient

instance = NaturalGradient(...)
result = instance._get_backend()
step(closure=None)[source]

Perform a single optimization step.

Parameters

closure – Optional closure that reevaluates the model and returns loss

Dependencies: numpy.

Variables: closure (Any, optional, default None).

Usage Example

from grilly.optim.natural_gradient import NaturalGradient

instance = NaturalGradient(...)
result = instance.step(closure=None)
load_state_dict(state_dict)

Load optimizer state from state_dict.

Parameters

state_dict (dict[str, Any]) – Dictionary containing optimizer state

Dependencies: None detected from callable globals.

Variables: state_dict (dict[str, typing.Any], required).

Usage Example

from grilly.optim.base import Optimizer

instance = Optimizer(...)
result = instance.load_state_dict(state_dict='example')
state_dict()

Return the state of the optimizer as a dict.

Returns

Dictionary containing optimizer state

Return type

dict[str, Any]

Dependencies: None detected from callable globals.

Variables: This callable does not take explicit input variables.

Usage Example

from grilly.optim.base import Optimizer

instance = Optimizer(...)
result = instance.state_dict()
zero_grad()

Clear gradients for all parameters.

Note: In this implementation, gradients are expected to be stored in a separate structure (e.g., in the model’s backward pass). This method is provided for API compatibility.

Dependencies: None detected from callable globals.

Variables: This callable does not take explicit input variables.

Usage Example

from grilly.optim.base import Optimizer

instance = Optimizer(...)
result = instance.zero_grad()