L-BFGS

L-BFGS is a classical optimization method that uses past gradients and parameters information to iteratively refine a solution to a minimization problem. In this notebook, we illustrate

1. how to use L-BFGS as a simple gradient transformation,

2. how to wrap L-BFGS in a solver, and how linesearches are incorporated,

3. how to debug the solver if needed.

from typing import NamedTuple

import jax
import jax.numpy as jnp
import jax.random as jrd

import optax
import optax.tree

L-BFGS as a gradient transformation

What is L-BFGS?

To solve a problem of the form

\[ \min_w f(w), \]

L-BFGS (Limited memory Broyden–Fletcher–Goldfarb–Shanno algorithm) makes steps of the form

\[ w_{k+1} = w_k - \eta_k P_k g_k, \]

where, at iteration \(k\), \(w_k\) are the parameters, \(g_k = \nabla f_k\) are the gradients, \(\eta_k\) is the stepsize, and \(P_k\) is a preconditioning matrix, that is, a matrix that transforms the gradients to ease the optimization process.

L-BFGS builds the preconditioning matrix \(P_k\) as an approximation of the Hessian inverse \(P_k \approx \nabla^2 f(w_k)^{-1}\) using past gradient and parameters information. Briefly, at iteration \(k\), the previous preconditioning matrix \(P_{k-1}\) is updated such that \(P_k\) satisfies the secant condition \(P_k(w_k-w_{k-1}) = g_k -g_{k-1}\). The original BFGS algorithm updates \(P_k\) using all past information, the limited-memory variant only uses a fixed number of past parameters and gradients to build \(P_k\). See Nocedal and Wright, Numerical Optimization, 1999 or the documentation for more details on the implementation.

Using L-BFGS as a gradient transformation

The function optax.scale_by_lbfgs() implements the update of the preconditioning matrix given a running optimizer state \(s_k\). Given \((g_k, s_k, w_k)\), this function returns \((P_kg_k, s_{k+1})\). We illustrate its performance below on a simple convex quadratic.

# Define objective
dim = 8
w_opt = jnp.ones(dim)
mat = jrd.normal(jrd.PRNGKey(0), (dim, dim))
mat = mat.dot(mat.T)


def fun(w):
  return 0.5 * (w - w_opt).dot(mat.dot(w - w_opt))


# Define optimizer
lr = 1e-1
opt = optax.scale_by_lbfgs()

# Initialize optimization
w = jrd.normal(jrd.PRNGKey(1), (dim,))
state = opt.init(w_opt)

# Run optimization
for i in range(16):
  v, g = jax.value_and_grad(fun)(w)
  print(f'Iteration: {i}, Value:{v:.2e}')
  u, state = opt.update(g, state, w)
  w = w - lr * u

print(f'Final value: {fun(w):.2e}')

Iteration: 0, Value:1.28e+01

Iteration: 1, Value:1.12e+01
Iteration: 2, Value:9.76e+00

Iteration: 3, Value:8.30e+00
Iteration: 4, Value:6.98e+00

Iteration: 5, Value:5.85e+00
Iteration: 6, Value:4.89e+00

Iteration: 7, Value:4.08e+00
Iteration: 8, Value:3.39e+00

Iteration: 9, Value:2.82e+00
Iteration: 10, Value:2.34e+00

Iteration: 11, Value:1.94e+00
Iteration: 12, Value:1.60e+00

Iteration: 13, Value:1.32e+00
Iteration: 14, Value:1.09e+00

Iteration: 15, Value:9.03e-01
Final value: 7.46e-01

L-BFGS as a solver

L-BFGS is a sample in numerical optimization to solve medium scale problems. It is often the backend of generic minimization functions in software libraries like scipy. A key ingredient to make it a simple optimization blackbox, is to remove the need of tuning the stepsize, a.k.a. learning rate in machine learning. In a deterministic setting (no additional varying inputs like inputs/labels), such automatic tuning of the stepsize is done by means of linesearches reviewed below.

What are linesearches?

Given current parameters \(w_k\), an update direction \(u_k\) (such as the negative preconditioned gradient \(u_k = -P_k g_k\) returned by L-BFGS), a linesearch computes a stepsize \(\eta_k\) such that the next iterate \(w_{k+1} = w_k + \eta_k u_k\) satisfies some criterions.

Sufficient decrease (Armijo-Goldstein criterion)

The first criterion that a good stepsize may need to satisfy is to ensure that the next iterate decreases the value of the objective by a a sufficient amount. Mathematically, the criterion is expressed as finding \(\eta_k\) such that

\[ f(w_k + \eta_k u_k) \leq f(w_k) + c_1 \eta_k \langle u_k, g_k\rangle \]

where \(c_1\) is some constant set to \(10^{-4}\) by default. Consider for example the update direction to be \(u_k = -g_k\), i.e., moving along the negative gradient direction. In that case the criterion above reduces to \(f(w_k - \eta_k g_k) \leq f(w_k) - c_1 \eta_k ||g_k||_2^2\). The criterion amounts then to choosing the stepsize such that it decreases the objective by an amount proportional to the squared gradient norm.

As long as the update direction is a descent direction, that is, \(\langle u_k, g_k\rangle < 0\) the above criterion is guaranteed to be satisfied by some sufficiently small stepsize. A simple linesearch technique to ensure a sufficient decrease is then to decrease a candidate stepsize by a constant factor up until the criterion is satisfied. This amounts to the backtracking linesearch implemented in optax.scale_by_backtracking_linesearch() and briefly reviewed below.

Small curvature (Strong wolfe criterion)

The sufficient decrease criterion ensures that the algorithm does not produce a sequence of diverging objective values. However, we may want to not only reduce a current stepsize but also increase it to ensure maximal speed. Ideally, we would like to find the stepsize that minimizes the function along the current update, i.e., \(\eta_k^* = \arg\min_\eta f(w_k + \eta u_k)\). Such an endeavor can be computationally prohibitive, so we may select a stepsize that ensures some properties that an optimal stepsize would satisfy. In particular, we may search for a stepsize such that the derivative of \(h(\eta) = f(w_k + \eta u_k)\) is small enough compared to its derivativeœ at \(\eta=0\). Formally, we may want to select the stepsize \(\eta_k\) such that \(|h'(eta_k)| \leq |h'(0)|\), that is,

\[ |\langle \nabla f(w_k + \eta_k u_k), u_k\rangle| \leq |\langle \nabla f(w_k), u_k\rangle|. \]

See Chapter 3 of Nocedal and Wright, Numerical Optimization, 1999 for some illustrations of this criterion. A linesearch method that can ensure both criterions require some form of bisection method implemented in optax with the optax.scale_by_zoom_linesearch() method. Several other linesearch techniques exist, see e.g. https://github.com/JuliaNLSolvers/LineSearches.jl. It is generally recommended to combine L-BFGS with a line-search ensuring both sufficient decrease and small curvature, which the optax.scale_by_zoom_linesearch() ensures.

Linesearches in practice

To find a stepsize satisfying the above criterions, a linesearch needs to access the value and potentially the gradient of the function. So linesearches in optax are implemented as optax.GradientTransformationExtraArgs(), which take the current value, gradient of the objective as well as the function itself. We illustrate this below with optax.scale_by_backtracking_linesearch().

# Objective
def fun(w):
  return jnp.sum(jnp.abs(w))


# Linesearch, comment/uncomment the desired one
linesearch = optax.scale_by_backtracking_linesearch(max_backtracking_steps=15)
# linesearch = optax.scale_by_zoom_linesearch(max_linesearch_steps=15)

# Optimizer
opt = optax.chain(
    optax.sgd(learning_rate=1.0),
    # Compare with or without linesearch by commenting this line
    linesearch,
)

# Initialize
w = jrd.normal(jrd.PRNGKey(0), (8,))
state = opt.init(w)

# Run optimization
for i in range(16):
  v, g = jax.value_and_grad(fun)(w)
  print(f'Iteration: {i}, Value:{v:.2e}')
  u, state = opt.update(g, state, w, value=v, grad=g, value_fn=fun)
  w = w + u

print(f'Final value: {fun(w):.2e}')

Iteration: 0, Value:6.30e+00
Iteration: 1, Value:5.00e+00
Iteration: 2, Value:3.05e+00
Iteration: 3, Value:2.73e+00

Iteration: 4, Value:2.18e+00
Iteration: 5, Value:1.59e+00
Iteration: 6, Value:1.41e+00
Iteration: 7, Value:1.37e+00
Iteration: 8, Value:1.30e+00
Iteration: 9, Value:1.26e+00

Iteration: 10, Value:1.20e+00
Iteration: 11, Value:1.16e+00
Iteration: 12, Value:1.12e+00
Iteration: 13, Value:1.06e+00
Iteration: 14, Value:1.04e+00

Iteration: 15, Value:9.69e-01
Final value: 6.97e-01

To validate the stepsize the linesearch calls the function several times. If a stepsize is accepted, we have then a priori access to the value of the function, and, potentially its gradient. The implementation of the linesearches in optax store the value and the gradient computed by the linesearch to avoid recomputing them at the next step. In practice, the code above can be modified as follows.

Note: The backtracking linesearch only evaluates the function and does not compute the gradient natively. To make the backtracking linesearch compute and store the gradient at the stepsize taken, we add the flag store_grad=True, see below. The zoom linesearch always compute both function and gradient so there is no need to specify an additional flag.

# Objective
def fun(w):
  return jnp.sum(jnp.abs(w))


# Linesearch
linesearch = optax.scale_by_backtracking_linesearch(
    max_backtracking_steps=15, store_grad=True
)
# linesearch = optax.scale_by_zoom_linesearch(max_linesearch_steps=15)

# Optimizer
opt = optax.chain(optax.sgd(learning_rate=1.0), linesearch)

# Initialize
w = jrd.normal(jrd.PRNGKey(0), (8,))
state = opt.init(w)

# Run optimization
for _ in range(16):
  # Replace `v, g = jax.value_and_grad(fun)(w)` by
  v, g = optax.value_and_grad_from_state(fun)(w, state=state)
  u, state = opt.update(g, state, w, value=v, grad=g, value_fn=fun)
  w = w + u

print(f'Final value: {fun(w):.2e}')

Final value: 6.97e-01

L-BFGS solver

Optax combines then the gradient transformation of L-BFGS and a linesearch in optax.lbfgs().

We present below a wrapper that combines both into a solver which tries to find the minimizer of a function given

1. some initial parameters init_params,

2. the function to optimize fun,

3. the instance of the L-BFGS solver considered opt,

4. a maximal number of iteration max_iter,

5. a tolerance tol on the optimization error measured here as the gradient norm.

def run_opt(init_params, fun, opt, max_iter, tol):
  value_and_grad_fun = optax.value_and_grad_from_state(fun)

  def step(carry):
    params, state = carry
    value, grad = value_and_grad_fun(params, state=state)
    updates, state = opt.update(
        grad, state, params, value=value, grad=grad, value_fn=fun
    )
    params = optax.apply_updates(params, updates)
    return params, state

  def continuing_criterion(carry):
    _, state = carry
    iter_num = optax.tree.get(state, 'count')
    grad = optax.tree.get(state, 'grad')
    err = optax.tree.norm(grad)
    return (iter_num == 0) | ((iter_num < max_iter) & (err >= tol))

  init_carry = (init_params, opt.init(init_params))
  final_params, final_state = jax.lax.while_loop(
      continuing_criterion, step, init_carry
  )
  return final_params, final_state

We can test the solver on the Rosenbrock function.

def fun(w):
  return jnp.sum(100.0 * (w[1:] - w[:-1] ** 2) ** 2 + (1.0 - w[:-1]) ** 2)

opt = optax.lbfgs()
init_params = jnp.zeros((8,))
print(
    f'Initial value: {fun(init_params):.2e} '
    f'Initial gradient norm: {optax.tree.norm(jax.grad(fun)(init_params)):.2e}'
)
final_params, _ = run_opt(init_params, fun, opt, max_iter=100, tol=1e-3)
print(
    f'Final value: {fun(final_params):.2e}, '
    f'Final gradient norm: {optax.tree.norm(jax.grad(fun)(final_params)):.2e}'
)

Initial value: 7.00e+00 Initial gradient norm: 5.29e+00

Final value: 4.79e-11, Final gradient norm: 3.14e-04

We may add additional information by simply chaining optax.lbfgs with an identity transform that just prints relevant information as follows.

class InfoState(NamedTuple):
  iter_num: jax.typing.ArrayLike


def print_info():
  def init_fn(params):
    del params
    return InfoState(iter_num=0)

  def update_fn(updates, state, params, *, value, grad, **extra_args):
    del params, extra_args

    jax.debug.print(
        'Iteration: {i}, Value: {v}, Gradient norm: {e}',
        i=state.iter_num,
        v=value,
        e=optax.tree.norm(grad),
    )
    return updates, InfoState(iter_num=state.iter_num + 1)

  return optax.GradientTransformationExtraArgs(init_fn, update_fn)


def fun(w):
  return jnp.sum(100.0 * (w[1:] - w[:-1] ** 2) ** 2 + (1.0 - w[:-1]) ** 2)


opt = optax.chain(print_info(), optax.lbfgs())
init_params = jnp.zeros((8,))
print(
    f'Initial value: {fun(init_params):.2e} '
    f'Initial gradient norm: {optax.tree.norm(jax.grad(fun)(init_params)):.2e}'
)
final_params, _ = run_opt(init_params, fun, opt, max_iter=100, tol=1e-3)
print(
    f'Final value: {fun(final_params):.2e}, '
    f'Final gradient norm: {optax.tree.norm(jax.grad(fun)(final_params)):.2e}'
)

Initial value: 7.00e+00 Initial gradient norm: 5.29e+00

Iteration: 0, Value: 7.0, Gradient norm: 5.291502475738525
Iteration: 1, Value: 6.919715404510498, Gradient norm: 2.7134430408477783
Iteration: 2, Value: 6.892337799072266, Gradient norm: 2.2310452461242676
Iteration: 3, Value: 6.756089687347412, Gradient norm: 8.751387596130371
Iteration: 4, Value: 6.6396660804748535, Gradient norm: 4.321895122528076
Iteration: 5, Value: 6.210091590881348, Gradient norm: 4.574130535125732
Iteration: 6, Value: 6.0793328285217285, Gradient norm: 11.160279273986816
Iteration: 7, Value: 5.667567253112793, Gradient norm: 7.809230327606201
Iteration: 8, Value: 5.033491134643555, Gradient norm: 7.547211647033691
Iteration: 9, Value: 4.838204860687256, Gradient norm: 10.114582061767578
Iteration: 10, Value: 4.733406066894531, Gradient norm: 10.091991424560547
Iteration: 11, Value: 4.34665584564209, Gradient norm: 8.955520629882812
Iteration: 12, Value: 3.9885854721069336, Gradient norm: 16.55974006652832
Iteration: 13, Value: 3.7100651264190674, Gradient norm: 14.80412483215332
Iteration: 14, Value: 3.5138602256774902, Gradient norm: 15.767204284667969
Iteration: 15, Value: 3.162855625152588, Gradient norm: 16.00653648376465
Iteration: 16, Value: 2.8462588787078857, Gradient norm: 9.95525074005127
Iteration: 17, Value: 2.6769323348999023, Gradient norm: 13.72292709350586
Iteration: 18, Value: 2.4160618782043457, Gradient norm: 11.994536399841309
Iteration: 19, Value: 2.057173490524292, Gradient norm: 11.88132381439209
Iteration: 20, Value: 1.8203760385513306, Gradient norm: 9.582115173339844
Iteration: 21, Value: 1.412052869796753, Gradient norm: 8.546500205993652
Iteration: 22, Value: 1.323590636253357, Gradient norm: 8.562395095825195
Iteration: 23, Value: 1.1128056049346924, Gradient norm: 12.693962097167969
Iteration: 24, Value: 0.9371380805969238, Gradient norm: 9.07295036315918
Iteration: 25, Value: 0.7151755690574646, Gradient norm: 8.560441970825195
Iteration: 26, Value: 0.5538158416748047, Gradient norm: 9.518914222717285
Iteration: 27, Value: 0.36079245805740356, Gradient norm: 7.089829921722412
Iteration: 28, Value: 0.3164284825325012, Gradient norm: 6.030513763427734
Iteration: 29, Value: 0.20977073907852173, Gradient norm: 5.676949501037598
Iteration: 30, Value: 0.15522511303424835, Gradient norm: 8.340078353881836
Iteration: 31, Value: 0.09047466516494751, Gradient norm: 4.751142978668213
Iteration: 32, Value: 0.05980892479419708, Gradient norm: 5.03424072265625
Iteration: 33, Value: 0.04211728274822235, Gradient norm: 3.312865972518921
Iteration: 34, Value: 0.026040762662887573, Gradient norm: 3.228518009185791
Iteration: 35, Value: 0.011805434711277485, Gradient norm: 2.2738325595855713
Iteration: 36, Value: 0.004367878660559654, Gradient norm: 1.0254955291748047
Iteration: 37, Value: 0.001954672858119011, Gradient norm: 1.2583403587341309
Iteration: 38, Value: 0.0005290439585223794, Gradient norm: 0.6866418719291687
Iteration: 39, Value: 8.905142021831125e-05, Gradient norm: 0.2946114242076874
Iteration: 40, Value: 6.649098395428155e-06, Gradient norm: 0.06996441632509232
Iteration: 41, Value: 2.400921175649273e-06, Gradient norm: 0.07435156404972076
Iteration: 42, Value: 2.871581372687615e-08, Gradient norm: 0.005336267873644829
Iteration: 43, Value: 2.9224929143367717e-09, Gradient norm: 0.0017461755778640509
Final value: 4.79e-11, Final gradient norm: 3.14e-04

Debugging

Accessing debug information

In some cases, L-BFGS with a linesearch as a solver will fail. Most of the times, the culprit goes down to the linesearch. To debug the solver in such cases, we provide a verbose option to the optax.scale_by_zoom_linesearch. We show below how to proceed.

To demonstrate such bug, we try to minimize the Zakharov function and set the scale_init_precond option to False (by choosing the default option scale_init_precond=True, the algorithm would actually run fine, we just want to showcase the possibility to use debugging in the linesearch here). You’ll observe that the final value is the same as the initial value which points out that the solver failed.

def fun(w):
  ii = jnp.arange(1, len(w) + 1, step=1, dtype=w.dtype)
  sum1 = (w**2).sum()
  sum2 = (0.5 * ii * w).sum()
  return sum1 + sum2**2 + sum2**4

opt = optax.lbfgs(scale_init_precond=False)

init_params = jnp.array([600.0, 700.0, 200.0, 100.0, 90.0, 1e4])
print(
    f'Initial value: {fun(init_params)} '
    f'Initial gradient norm: {optax.tree.norm(jax.grad(fun)(init_params))}'
)
final_params, _ = run_opt(init_params, fun, opt, max_iter=50, tol=1e-3)
print(
    f'Final value: {fun(final_params)}, '
    f'Final gradient norm: {optax.tree.norm(jax.grad(fun)(final_params))}'
)

Initial value: 1.0129932568095621e+18 Initial gradient norm: 609193933406208.0

Final value: 1.0129932568095621e+18, Final gradient norm: 609193933406208.0

The default implementation of the linesearch in the code is

scale_by_zoom_linesearch(max_linesearch_steps=20, initial_guess_strategy='one')

To debug we can set the verbose option of the linesearch to True.

opt = optax.chain(print_info(), optax.lbfgs(scale_init_precond=False,
  linesearch=optax.scale_by_zoom_linesearch(
      max_linesearch_steps=20, verbose=True, initial_guess_strategy='one'
  )
))

init_params = jnp.array([600.0, 700.0, 200.0, 100.0, 90.0, 1e4])
print(
    f'Initial value: {fun(init_params):.2e} '
    f'Initial gradient norm: {optax.tree.norm(jax.grad(fun)(init_params)):.2e}'
)
final_params, _ = run_opt(init_params, fun, opt, max_iter=100, tol=1e-3)
print(
    f'Final value: {fun(final_params):.2e}, '
    f'Final gradient norm: {optax.tree.norm(jax.grad(fun)(final_params)):.2e}'
)

Initial value: 1.01e+18 Initial gradient norm: 6.09e+14

Iteration: 0, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Iteration: 1, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Iteration: 2, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Iteration: 3, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Iteration: 4, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Iteration: 5, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Iteration: 6, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Iteration: 7, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Iteration: 8, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Iteration: 9, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Iteration: 10, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Iteration: 11, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Iteration: 12, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Iteration: 13, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Iteration: 14, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Iteration: 15, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Iteration: 16, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Iteration: 17, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Iteration: 18, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Iteration: 19, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Iteration: 20, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Iteration: 21, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Iteration: 22, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Iteration: 23, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Iteration: 24, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Iteration: 25, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Iteration: 26, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Iteration: 27, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Iteration: 28, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Iteration: 29, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Iteration: 30, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Iteration: 31, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Iteration: 32, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Iteration: 33, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Iteration: 34, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Iteration: 35, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Iteration: 36, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Iteration: 37, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Iteration: 38, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Iteration: 39, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Iteration: 40, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Iteration: 41, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Iteration: 42, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Iteration: 43, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Iteration: 44, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Iteration: 45, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Iteration: 46, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Iteration: 47, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Iteration: 48, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Iteration: 49, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Iteration: 50, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Iteration: 51, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Iteration: 52, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Iteration: 53, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Iteration: 54, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Iteration: 55, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Iteration: 56, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Iteration: 57, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Iteration: 58, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Iteration: 59, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Iteration: 60, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Iteration: 61, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Iteration: 62, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Iteration: 63, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Iteration: 64, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Iteration: 65, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Iteration: 66, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Iteration: 67, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Iteration: 68, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Iteration: 69, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Iteration: 70, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Iteration: 71, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Iteration: 72, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Iteration: 73, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Iteration: 74, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Iteration: 75, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Iteration: 76, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Iteration: 77, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Iteration: 78, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Iteration: 79, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Iteration: 80, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Iteration: 81, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Iteration: 82, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Iteration: 83, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Iteration: 84, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Iteration: 85, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Iteration: 86, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Iteration: 87, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Iteration: 88, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Iteration: 89, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Iteration: 90, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Iteration: 91, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Iteration: 92, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Iteration: 93, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Iteration: 94, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Iteration: 95, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Iteration: 96, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Iteration: 97, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Iteration: 98, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Iteration: 99, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
INFO: optax.scale_by_zoom_linesearch:
Value at current params: 1.0129932568095621e+18
Slope along update direction: -3.7111729481005034e+29
Stepsize reached: 1.9073486328125e-06
Decrease Error: inf
Curvature Error: inf
Length of searched interval has been reduced below threshold. Interval length: 1.9073486328125e-06.
Linesearch failed, no stepsize satisfying sufficient decrease found.
Consider augmenting the maximal number of linesearch iterations.
Very large absolute slope at stepsize=0. (|slope|=3.7111729481005034e+29). The objective is badly conditioned. Consider reparameterizing objective (e.g., normalizing parameters) or finding a better guess for the initial parameters for the solver.
Cannot even make a step without getting Inf or Nan. The linesearch won't make a step and the optimizer is stuck.
Final value: 1.01e+18, Final gradient norm: 6.09e+14

As expected, the linesearch failed at the very first step taking a stepsize that did not ensure a sufficient decrease. Multiple information is displayed. For example, the slope (derivative along the update direction) at the first step is extremely large which explains the difficulties to find an appropriate stepsize. As pointed out in the log above, the first thing to try is to use a larger number of linesearch steps.

opt = optax.chain(print_info(), optax.lbfgs(scale_init_precond=False,
  linesearch=optax.scale_by_zoom_linesearch(
      max_linesearch_steps=50, verbose=True, initial_guess_strategy='one'
  )
))

init_params = jnp.array([600.0, 700.0, 200.0, 100.0, 90.0, 1e4])
print(
    f'Initial value: {fun(init_params):.2e} '
    f'Initial gradient norm: {optax.tree.norm(jax.grad(fun)(init_params)):.2e}'
)
final_params, _ = run_opt(init_params, fun, opt, max_iter=100, tol=1e-3)
print(
    f'Final value: {fun(final_params):.2e}, '
    f'Final gradient norm: {optax.tree.norm(jax.grad(fun)(final_params)):.2e}'
)

Initial value: 1.01e+18 Initial gradient norm: 6.09e+14

Iteration: 0, Value: 1.0129932568095621e+18, Gradient norm: 609194000515072.0
Iteration: 1, Value: 1.2426169316016128e+16, Gradient norm: 22454512648192.0
Iteration: 2, Value: 6721875073826816.0, Gradient norm: 14163075137536.0
Iteration: 3, Value: 1773725661790208.0, Gradient norm: 5201816715264.0
Iteration: 4, Value: 630085191204864.0, Gradient norm: 2356343996416.0
Iteration: 5, Value: 215450709393408.0, Gradient norm: 881636474880.0
Iteration: 6, Value: 51199176867840.0, Gradient norm: 349650419712.0
Iteration: 7, Value: 21698082504704.0, Gradient norm: 187240923136.0
Iteration: 8, Value: 6098329272320.0, Gradient norm: 73784377344.0
Iteration: 9, Value: 2144722878464.0, Gradient norm: 32777279488.0
Iteration: 10, Value: 731362099200.0, Gradient norm: 11847122944.0
Iteration: 11, Value: 178632851456.0, Gradient norm: 5218198016.0
Iteration: 12, Value: 178615173120.0, Gradient norm: 5217774080.0
Iteration: 13, Value: 176890298368.0, Gradient norm: 5172149760.0
Iteration: 14, Value: 173696139264.0, Gradient norm: 5037954560.0
Iteration: 15, Value: 45631021056.0, Gradient norm: 1836248064.0
Iteration: 16, Value: 18003183616.0, Gradient norm: 890520000.0
Iteration: 17, Value: 17855059968.0, Gradient norm: 897006464.0
Iteration: 18, Value: 17611644928.0, Gradient norm: 904036480.0
Iteration: 19, Value: 4928171008.0, Gradient norm: 323001408.0
Iteration: 20, Value: 3482264064.0, Gradient norm: 263560816.0
Iteration: 21, Value: 2522250496.0, Gradient norm: 31022630.0
Iteration: 22, Value: 2507938816.0, Gradient norm: 31354602.0
Iteration: 23, Value: 132079992.0, Gradient norm: 6020654.5
Iteration: 24, Value: 28353042.0, Gradient norm: 5543275.5
Iteration: 25, Value: 28211742.0, Gradient norm: 5452383.5
Iteration: 26, Value: 10805378.0, Gradient norm: 3554045.0
Iteration: 27, Value: 6768756.0, Gradient norm: 2418408.25
Iteration: 28, Value: 6353451.0, Gradient norm: 2290546.0
Iteration: 29, Value: 5072234.5, Gradient norm: 1911489.75
Iteration: 30, Value: 1440178.75, Gradient norm: 788539.125
Iteration: 31, Value: 478457.3125, Gradient norm: 346872.53125
Iteration: 32, Value: 154086.421875, Gradient norm: 148257.46875
Iteration: 33, Value: 153831.9375, Gradient norm: 147896.921875
Iteration: 34, Value: 11854.306640625, Gradient norm: 12452.7783203125
Iteration: 35, Value: 10402.576171875, Gradient norm: 12848.263671875
Iteration: 36, Value: 9519.4765625, Gradient norm: 13108.123046875
Iteration: 37, Value: 1801.2613525390625, Gradient norm: 5236.0986328125
Iteration: 38, Value: 1773.1986083984375, Gradient norm: 5091.07470703125
Iteration: 39, Value: 1756.2581787109375, Gradient norm: 5004.54248046875
Iteration: 40, Value: 805.4383544921875, Gradient norm: 57.10791778564453
Iteration: 41, Value: 27.507734298706055, Gradient norm: 10.569019317626953
Iteration: 42, Value: 0.011515535414218903, Gradient norm: 0.5599392056465149
Iteration: 43, Value: 0.0028441993054002523, Gradient norm: 0.5186086297035217
Iteration: 44, Value: 0.0026483393739908934, Gradient norm: 0.4807297885417938
Iteration: 45, Value: 0.0024611656554043293, Gradient norm: 0.4610635042190552
Iteration: 46, Value: 0.0024114542175084352, Gradient norm: 0.45158037543296814
Iteration: 47, Value: 0.0023696222342550755, Gradient norm: 0.44357097148895264
Iteration: 48, Value: 0.0016692231874912977, Gradient norm: 0.3720560669898987
Iteration: 49, Value: 1.4261591786635108e-05, Gradient norm: 0.007942347787320614
Final value: 5.34e-09, Final gradient norm: 3.44e-04

By simply taking a maximum of 50 steps of the linesearch instead of 20, we ensured that the first stepsize taken provided a sufficient decrease and the solver worked well. Additional debugging information can be found in the source code accessible from the docs of optax.scale_by_zoom_linesearch().

Tips

• LBFGS

  • Selecting a higher memory_size in lbfgs may improve performance at a memory and computational cost. No real gains may be perceived after some value.

  • scale_init_precond=True is standard. It captures a similar scale as other well-known optimization methods like Barzilai Borwein.

• Zoom linesearch

  • Remember there are two conditions to be met (sufficient decrease and small curvature). If the algorithm takes too many linesearch steps, you may try setting curv_rtol = jnp.inf, effectively ignoring the small curvature condition. The resulting algorithm will essentially perform a backtracking linesearch where a valid stepsize is searched by minmizing a quadratic or cubic approximation of the objective (so that would be a potentially faster algorithm than the current implementation of scale_by_backtracking_linesearch).

  • As pointed above, if the solver gets stuck, try using a larger number of linesearch steps and print debugging information.

You may run the solver in double precision by setting jax.config.update("jax_enable_x64", True). If you use double precision, consider augmenting the number of linesearch steps to reach the machine precision (like using max_linesearch_steps=55).

Contributing and benchmarking

Numerous other linesearch could be implemented, as well as other solvers for medium scale problems without stochasticity. Contributions are welcome.

If you want to contribute a new solver for medium scale problems like LBFGS, benchmarks would be highly appreciated. We provide below an example of benchmark (which could also be used if you want to test some hyperparameters of the algorithm). We take here the classical Rosenbroke function, but it could be better to expand such benchmarks to e.g. the set of test functions given by Andrei, 2008.

import time
num_fun_calls = 0

def register_call():
  global num_fun_calls
  num_fun_calls += 1

def test_hparams(lbfgs_hparams, linesearch_hparams, dimension=512):
  global num_fun_calls
  num_fun_calls = 0

  def fun(x):
    jax.debug.callback(register_call)
    return jnp.sum((x[1:] - x[:-1] ** 2) ** 2 + (1.0 - x[:-1]) ** 2)

  opt = optax.chain(optax.lbfgs(**lbfgs_hparams,
    linesearch=optax.scale_by_zoom_linesearch(**linesearch_hparams)
    )
  )

  init_params = jnp.arange(dimension, dtype=jnp.float32)

  tic = time.time()
  final_params, _ = run_opt(
      init_params, fun, opt, max_iter=500, tol=5*1e-5
    )
  final_params = jax.block_until_ready(final_params)
  time_run = time.time() - tic

  final_value = fun(final_params)
  final_grad_norm = optax.tree.norm(jax.grad(fun)(final_params))
  return final_value, final_grad_norm, num_fun_calls, time_run

import copy
import matplotlib.pyplot as plt

default_lbfgs_hparams = {'memory_size': 15, 'scale_init_precond': True}
default_linesearch_hparams = {
    'max_linesearch_steps': 15,
    'initial_guess_strategy': 'one'
}

memory_sizes = [int(2**i) for i in range(7)]
times = []
calls = []
values = []
grad_norms = []
for m in memory_sizes:
  lbfgs_hparams = copy.deepcopy(default_lbfgs_hparams)
  lbfgs_hparams['memory_size'] = m
  v, g, n, t = test_hparams(lbfgs_hparams, default_linesearch_hparams, dimension=1024)
  values.append(v)
  grad_norms.append(g)
  calls.append(n)
  times.append(t)

fig, axs = plt.subplots(1, 4, figsize=(16, 4))
axs[0].plot(memory_sizes, values)
axs[0].set_ylabel('Final values')
axs[0].set_yscale('log')
axs[1].plot(memory_sizes, grad_norms)
axs[1].set_ylabel('Final gradient norms')
axs[1].set_yscale('log')
axs[2].plot(memory_sizes, calls)
axs[2].set_ylabel('Number of function calls')
axs[3].plot(memory_sizes, times)
axs[3].set_ylabel('Run times')
for i in range(4):
  axs[i].set_xlabel('Memory size')
plt.tight_layout()