grilly.optim.lr_scheduler

Learning Rate Schedulers

Implements various learning rate scheduling strategies to match PyTorch’s API.

Classes

CosineAnnealingLR(optimizer, T_max[, ...])	Set the learning rate using a cosine annealing schedule.
LRScheduler(optimizer[, last_epoch])	Base class for learning rate schedulers.
OneCycleLR(optimizer, max_lr[, total_steps, ...])	Sets the learning rate according to the 1cycle learning rate policy.
ReduceLROnPlateau(optimizer[, mode, factor, ...])	Reduce learning rate when a metric has stopped improving.
StepLR(optimizer, step_size[, gamma, last_epoch])	Decays the learning rate by gamma every step_size epochs.

class grilly.optim.lr_scheduler.LRScheduler(optimizer, last_epoch=-1)

Bases: object

Base class for learning rate schedulers.

All schedulers should inherit from this class and implement the get_lr() method.

Initialize base scheduler.

Parameters

• optimizer – Wrapped optimizer

• last_epoch – The index of last epoch (default: -1)

__init__(optimizer, last_epoch=-1)

Initialize base scheduler.

Parameters

• optimizer – Wrapped optimizer

• last_epoch – The index of last epoch (default: -1)

Dependencies: None detected from callable globals.

Variables: optimizer (Any, required); last_epoch (Any, optional, default -1).

Usage Example

from grilly.optim.lr_scheduler import LRScheduler

instance = LRScheduler(...)
result = instance.__init__(optimizer=None, last_epoch=-1)

state_dict()

Returns the state of the scheduler as a dict.

Dependencies: None detected from callable globals.

Variables: This callable does not take explicit input variables.

Usage Example

from grilly.optim.lr_scheduler import LRScheduler

instance = LRScheduler(...)
result = instance.state_dict()

load_state_dict(state_dict)

Loads the scheduler state.

Dependencies: None detected from callable globals.

Variables: state_dict (Any, required).

Usage Example

from grilly.optim.lr_scheduler import LRScheduler

instance = LRScheduler(...)
result = instance.load_state_dict(state_dict=None)

get_last_lr()

Return last computed learning rate by current scheduler.

Dependencies: None detected from callable globals.

Variables: This callable does not take explicit input variables.

Usage Example

from grilly.optim.lr_scheduler import LRScheduler

instance = LRScheduler(...)
result = instance.get_last_lr()

get_lr()

Compute learning rate using chainable form of the scheduler.

This method should be implemented by subclasses.

Dependencies: None detected from callable globals.

Variables: This callable does not take explicit input variables.

Usage Example

from grilly.optim.lr_scheduler import LRScheduler

instance = LRScheduler(...)
result = instance.get_lr()

step(epoch=None)

Perform a scheduler step.

Parameters

epoch – Optional epoch number to use instead of incrementing

Dependencies: None detected from callable globals.

Variables: epoch (Any, optional, default None).

Usage Example

from grilly.optim.lr_scheduler import LRScheduler

instance = LRScheduler(...)
result = instance.step(epoch=None)

class grilly.optim.lr_scheduler.StepLR(optimizer, step_size, gamma=0.1, last_epoch=-1)

Bases: LRScheduler

Decays the learning rate by gamma every step_size epochs.

Matches torch.optim.lr_scheduler.StepLR

Initialize StepLR scheduler.

Parameters

• optimizer – Wrapped optimizer

• step_size – Period of learning rate decay

• gamma – Multiplicative factor of learning rate decay (default: 0.1)

• last_epoch – The index of last epoch (default: -1)

__init__(optimizer, step_size, gamma=0.1, last_epoch=-1)

Initialize StepLR scheduler.

Parameters

• optimizer – Wrapped optimizer

• step_size – Period of learning rate decay

• gamma – Multiplicative factor of learning rate decay (default: 0.1)

• last_epoch – The index of last epoch (default: -1)

Dependencies: None detected from callable globals.

Variables: optimizer (Any, required); step_size (Any, required); gamma (Any, optional, default 0.1); last_epoch (Any, optional, default -1).

Usage Example

from grilly.optim.lr_scheduler import StepLR

instance = StepLR(...)
result = instance.__init__(optimizer=None, step_size=None, gamma=0.1, last_epoch=-1)

get_lr()

Compute learning rate for current epoch.

Dependencies: None detected from callable globals.

Variables: This callable does not take explicit input variables.

Usage Example

from grilly.optim.lr_scheduler import StepLR

instance = StepLR(...)
result = instance.get_lr()

get_last_lr()

Return last computed learning rate by current scheduler.

Dependencies: None detected from callable globals.

Variables: This callable does not take explicit input variables.

Usage Example

from grilly.optim.lr_scheduler import LRScheduler

instance = LRScheduler(...)
result = instance.get_last_lr()

load_state_dict(state_dict)

Loads the scheduler state.

Dependencies: None detected from callable globals.

Variables: state_dict (Any, required).

Usage Example

from grilly.optim.lr_scheduler import LRScheduler

instance = LRScheduler(...)
result = instance.load_state_dict(state_dict=None)

state_dict()

Returns the state of the scheduler as a dict.

Dependencies: None detected from callable globals.

Variables: This callable does not take explicit input variables.

Usage Example

from grilly.optim.lr_scheduler import LRScheduler

instance = LRScheduler(...)
result = instance.state_dict()

step(epoch=None)

Perform a scheduler step.

Parameters

epoch – Optional epoch number to use instead of incrementing

Dependencies: None detected from callable globals.

Variables: epoch (Any, optional, default None).

Usage Example

from grilly.optim.lr_scheduler import LRScheduler

instance = LRScheduler(...)
result = instance.step(epoch=None)

class grilly.optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max, eta_min=0, last_epoch=-1)

Bases: LRScheduler

Set the learning rate using a cosine annealing schedule.

Matches torch.optim.lr_scheduler.CosineAnnealingLR

Initialize CosineAnnealingLR scheduler.

Parameters

• optimizer – Wrapped optimizer

• T_max – Maximum number of iterations

• eta_min – Minimum learning rate (default: 0)

• last_epoch – The index of last epoch (default: -1)

__init__(optimizer, T_max, eta_min=0, last_epoch=-1)

Initialize CosineAnnealingLR scheduler.

Parameters

• optimizer – Wrapped optimizer

• T_max – Maximum number of iterations

• eta_min – Minimum learning rate (default: 0)

• last_epoch – The index of last epoch (default: -1)

Dependencies: None detected from callable globals.

Variables: optimizer (Any, required); T_max (Any, required); eta_min (Any, optional, default 0); last_epoch (Any, optional, default -1).

Usage Example

from grilly.optim.lr_scheduler import CosineAnnealingLR

instance = CosineAnnealingLR(...)
result = instance.__init__(optimizer=None, T_max=None, eta_min=0, last_epoch=-1)

get_lr()

Compute learning rate using cosine annealing.

Dependencies: math.

Variables: This callable does not take explicit input variables.

Usage Example

from grilly.optim.lr_scheduler import CosineAnnealingLR

instance = CosineAnnealingLR(...)
result = instance.get_lr()

get_last_lr()

Return last computed learning rate by current scheduler.

Dependencies: None detected from callable globals.

Variables: This callable does not take explicit input variables.

Usage Example

from grilly.optim.lr_scheduler import LRScheduler

instance = LRScheduler(...)
result = instance.get_last_lr()

load_state_dict(state_dict)

Loads the scheduler state.

Dependencies: None detected from callable globals.

Variables: state_dict (Any, required).

Usage Example

from grilly.optim.lr_scheduler import LRScheduler

instance = LRScheduler(...)
result = instance.load_state_dict(state_dict=None)

state_dict()

Returns the state of the scheduler as a dict.

Dependencies: None detected from callable globals.

Variables: This callable does not take explicit input variables.

Usage Example

from grilly.optim.lr_scheduler import LRScheduler

instance = LRScheduler(...)
result = instance.state_dict()

step(epoch=None)

Perform a scheduler step.

Parameters

epoch – Optional epoch number to use instead of incrementing

Dependencies: None detected from callable globals.

Variables: epoch (Any, optional, default None).

Usage Example

from grilly.optim.lr_scheduler import LRScheduler

instance = LRScheduler(...)
result = instance.step(epoch=None)

class grilly.optim.lr_scheduler.ReduceLROnPlateau(optimizer, mode='min', factor=0.1, patience=10, threshold=0.0001, threshold_mode='rel', cooldown=0, min_lr=0, eps=1e-08)

Bases: object

Reduce learning rate when a metric has stopped improving.

Matches torch.optim.lr_scheduler.ReduceLROnPlateau

Initialize ReduceLROnPlateau scheduler.

Parameters

• optimizer – Wrapped optimizer

• mode – One of ‘min’ or ‘max’. In ‘min’ mode, lr will be reduced when the quantity monitored has stopped decreasing (default: ‘min’)

• factor – Factor by which the learning rate will be reduced (default: 0.1)

• patience – Number of epochs with no improvement after which learning rate will be reduced (default: 10)

• threshold – Threshold for measuring the new optimum (default: 1e-4)

• threshold_mode – One of ‘rel’, ‘abs’ (default: ‘rel’)

• cooldown – Number of epochs to wait before resuming normal operation after lr has been reduced (default: 0)

• min_lr – A lower bound on the learning rate (default: 0)

• eps – Minimal decay applied to lr (default: 1e-8)

__init__(optimizer, mode='min', factor=0.1, patience=10, threshold=0.0001, threshold_mode='rel', cooldown=0, min_lr=0, eps=1e-08)

Initialize ReduceLROnPlateau scheduler.

Parameters

• optimizer – Wrapped optimizer

• mode – One of ‘min’ or ‘max’. In ‘min’ mode, lr will be reduced when the quantity monitored has stopped decreasing (default: ‘min’)

• factor – Factor by which the learning rate will be reduced (default: 0.1)

• patience – Number of epochs with no improvement after which learning rate will be reduced (default: 10)

• threshold – Threshold for measuring the new optimum (default: 1e-4)

• threshold_mode – One of ‘rel’, ‘abs’ (default: ‘rel’)

• cooldown – Number of epochs to wait before resuming normal operation after lr has been reduced (default: 0)

• min_lr – A lower bound on the learning rate (default: 0)

• eps – Minimal decay applied to lr (default: 1e-8)

Dependencies: None detected from callable globals.

Variables: optimizer (Any, required); mode (Any, optional, default 'min'); factor (Any, optional, default 0.1); patience (Any, optional, default 10); threshold (Any, optional, default 0.0001); threshold_mode (Any, optional, default 'rel'); cooldown (Any, optional, default 0); min_lr (Any, optional, default 0); eps (Any, optional, default 1e-08).

Usage Example

from grilly.optim.lr_scheduler import ReduceLROnPlateau

instance = ReduceLROnPlateau(...)
result = instance.__init__(optimizer=None, mode='min', factor=0.1, patience=10, threshold=0.0001, threshold_mode='rel', cooldown=0, min_lr=0, eps=1e-08)

_reset()

Reset num_bad_epochs counter and cooldown counter.

Dependencies: None detected from callable globals.

Variables: This callable does not take explicit input variables.

Usage Example

from grilly.optim.lr_scheduler import ReduceLROnPlateau

instance = ReduceLROnPlateau(...)
result = instance._reset()

step(metrics, epoch=None)

Perform a scheduler step based on metric.

Parameters

• metrics – The metric to monitor

• epoch – Optional epoch number

Dependencies: None detected from callable globals.

Variables: metrics (Any, required); epoch (Any, optional, default None).

Usage Example

from grilly.optim.lr_scheduler import ReduceLROnPlateau

instance = ReduceLROnPlateau(...)
result = instance.step(metrics=None, epoch=None)

_reduce_lr(epoch)

Reduce learning rate.

Dependencies: None detected from callable globals.

Variables: epoch (Any, required).

Usage Example

from grilly.optim.lr_scheduler import ReduceLROnPlateau

instance = ReduceLROnPlateau(...)
result = instance._reduce_lr(epoch=None)

property in_cooldown

Check if scheduler is in cooldown period.

is_better(a, best)

Check if metric ‘a’ is better than ‘best’.

Dependencies: None detected from callable globals.

Variables: a (Any, required); best (Any, required).

Usage Example

from grilly.optim.lr_scheduler import ReduceLROnPlateau

instance = ReduceLROnPlateau(...)
result = instance.is_better(a=None, best=None)

_init_is_better(mode, threshold, threshold_mode)

Initialize comparison function.

Dependencies: None detected from callable globals.

Variables: mode (Any, required); threshold (Any, required); threshold_mode (Any, required).

Usage Example

from grilly.optim.lr_scheduler import ReduceLROnPlateau

instance = ReduceLROnPlateau(...)
result = instance._init_is_better(mode=None, threshold=None, threshold_mode=None)

state_dict()

Returns the state of the scheduler as a dict.

Dependencies: None detected from callable globals.

Variables: This callable does not take explicit input variables.

Usage Example

from grilly.optim.lr_scheduler import ReduceLROnPlateau

instance = ReduceLROnPlateau(...)
result = instance.state_dict()

load_state_dict(state_dict)

Loads the scheduler state.

Dependencies: None detected from callable globals.

Variables: state_dict (Any, required).

Usage Example

from grilly.optim.lr_scheduler import ReduceLROnPlateau

instance = ReduceLROnPlateau(...)
result = instance.load_state_dict(state_dict=None)

class grilly.optim.lr_scheduler.OneCycleLR(optimizer, max_lr, total_steps=None, epochs=None, steps_per_epoch=None, pct_start=0.3, anneal_strategy='cos', cycle_momentum=True, base_momentum=0.85, max_momentum=0.95, div_factor=25.0, final_div_factor=10000.0, last_epoch=-1)

Bases: LRScheduler

Sets the learning rate according to the 1cycle learning rate policy.

Matches torch.optim.lr_scheduler.OneCycleLR

Initialize OneCycleLR scheduler.

Parameters

• optimizer – Wrapped optimizer

• max_lr – Upper learning rate boundary in the cycle

• total_steps – Total number of steps in the cycle (optional)

• epochs – Number of epochs to train for (optional)

• steps_per_epoch – Number of steps per epoch (optional)

• pct_start – Percentage of the cycle spent increasing the learning rate (default: 0.3)

• anneal_strategy – Specifies the annealing strategy: ‘cos’ or ‘linear’ (default: ‘cos’)

• cycle_momentum – If True, momentum is cycled inversely (default: True)

• base_momentum – Lower momentum boundary in the cycle (default: 0.85)

• max_momentum – Upper momentum boundary in the cycle (default: 0.95)

• div_factor – Determines the initial learning rate via initial_lr = max_lr/div_factor (default: 25)

• final_div_factor – Determines the minimum learning rate via min_lr = initial_lr/final_div_factor (default: 1e4)

• last_epoch – The index of last epoch (default: -1)

__init__(optimizer, max_lr, total_steps=None, epochs=None, steps_per_epoch=None, pct_start=0.3, anneal_strategy='cos', cycle_momentum=True, base_momentum=0.85, max_momentum=0.95, div_factor=25.0, final_div_factor=10000.0, last_epoch=-1)

Initialize OneCycleLR scheduler.

Parameters

• optimizer – Wrapped optimizer

• max_lr – Upper learning rate boundary in the cycle

• total_steps – Total number of steps in the cycle (optional)

• epochs – Number of epochs to train for (optional)

• steps_per_epoch – Number of steps per epoch (optional)

• pct_start – Percentage of the cycle spent increasing the learning rate (default: 0.3)

• anneal_strategy – Specifies the annealing strategy: ‘cos’ or ‘linear’ (default: ‘cos’)

• cycle_momentum – If True, momentum is cycled inversely (default: True)

• base_momentum – Lower momentum boundary in the cycle (default: 0.85)

• max_momentum – Upper momentum boundary in the cycle (default: 0.95)

• div_factor – Determines the initial learning rate via initial_lr = max_lr/div_factor (default: 25)

• final_div_factor – Determines the minimum learning rate via min_lr = initial_lr/final_div_factor (default: 1e4)

• last_epoch – The index of last epoch (default: -1)

Dependencies: None detected from callable globals.

Variables: optimizer (Any, required); max_lr (Any, required); total_steps (Any, optional, default None); epochs (Any, optional, default None); steps_per_epoch (Any, optional, default None); pct_start (Any, optional, default 0.3); anneal_strategy (Any, optional, default 'cos'); cycle_momentum (Any, optional, default True); base_momentum (Any, optional, default 0.85); max_momentum (Any, optional, default 0.95); div_factor (Any, optional, default 25.0); final_div_factor (Any, optional, default 10000.0); last_epoch (Any, optional, default -1).

Usage Example

from grilly.optim.lr_scheduler import OneCycleLR

instance = OneCycleLR(...)
result = instance.__init__(optimizer=None, max_lr=None, total_steps=None, epochs=None, steps_per_epoch=None, pct_start=0.3, anneal_strategy='cos', cycle_momentum=True, base_momentum=0.85, max_momentum=0.95, div_factor=25.0, final_div_factor=10000.0, last_epoch=-1)

_format_param(name, optimizer, param)

Format parameter to be a list per parameter group.

Dependencies: None detected from callable globals.

Variables: name (Any, required); optimizer (Any, required); param (Any, required).

Usage Example

from grilly.optim.lr_scheduler import OneCycleLR

instance = OneCycleLR(...)
result = instance._format_param(name=None, optimizer=None, param=None)

_annealing_cos(start, end, pct)

Cosine annealing from start to end as pct goes from 0.0 to 1.0.

Dependencies: math.

Variables: start (Any, required); end (Any, required); pct (Any, required).

Usage Example

from grilly.optim.lr_scheduler import OneCycleLR

instance = OneCycleLR(...)
result = instance._annealing_cos(start=None, end=None, pct=None)

get_last_lr()

Return last computed learning rate by current scheduler.

Dependencies: None detected from callable globals.

Variables: This callable does not take explicit input variables.

Usage Example

from grilly.optim.lr_scheduler import LRScheduler

instance = LRScheduler(...)
result = instance.get_last_lr()

load_state_dict(state_dict)

Loads the scheduler state.

Dependencies: None detected from callable globals.

Variables: state_dict (Any, required).

Usage Example

from grilly.optim.lr_scheduler import LRScheduler

instance = LRScheduler(...)
result = instance.load_state_dict(state_dict=None)

state_dict()

Returns the state of the scheduler as a dict.

Dependencies: None detected from callable globals.

Variables: This callable does not take explicit input variables.

Usage Example

from grilly.optim.lr_scheduler import LRScheduler

instance = LRScheduler(...)
result = instance.state_dict()

step(epoch=None)

Perform a scheduler step.

Parameters

epoch – Optional epoch number to use instead of incrementing

Dependencies: None detected from callable globals.

Variables: epoch (Any, optional, default None).

Usage Example

from grilly.optim.lr_scheduler import LRScheduler

instance = LRScheduler(...)
result = instance.step(epoch=None)

_annealing_linear(start, end, pct)

Linear annealing from start to end as pct goes from 0.0 to 1.0.

Dependencies: None detected from callable globals.

Variables: start (Any, required); end (Any, required); pct (Any, required).

Usage Example

from grilly.optim.lr_scheduler import OneCycleLR

instance = OneCycleLR(...)
result = instance._annealing_linear(start=None, end=None, pct=None)

get_lr()

Compute learning rate at current step.

Dependencies: None detected from callable globals.

Variables: This callable does not take explicit input variables.

Usage Example

from grilly.optim.lr_scheduler import OneCycleLR

instance = OneCycleLR(...)
result = instance.get_lr()