Optimization in hilbert spaces

In the remainder of the course we will discuss optimization problems. In general, an optimization problem consists of picking the “best” signal according to some metric; the metric will be some functional $f:X\to Y$ and the search will be over a set of interest $D\subseteq X$ , so that the problem can be written as

We will need to extend the ideas of derivatives and gradients (which are used in optimization of single-variable real-valued functions) to arbitrary signal spaces where we can move in infinite directions on a set of interest.

Directional derivatives

Assume that we have a metric function $f:X\to Y$ and a set of interest $D\subseteq X$ . Navigating the “surface” of $f$ to find a maximum or minimum requires for us to formulate a framework for derivatives.

Definition 1 Let $x\in D\subseteq X$ and $h\in X$ be arbitrary. If the limit

exists, it is called Gâteaux differential of $f$ at $x$ with increment (or in the direction) $h$ . If the limit exists for each $h\in X$ , the transformation $f$ is said to be Gâteaux differentiable at $x$ . If $f$ is Gâteaux differentiable at all $x\in X$ , then it is called a Gâteaux differentiable functional .

This extends the concept of derivative to incorporate direction so it can be used for any signal space. Note that $\alpha$ needs to be sufficiently small so that $x+\alpha h\in D$ . Note also that for a fixed point $x$ and variable direction $h$ , the Gâteaux differential is a map from $X$ to $Y$ , i.e., $\delta f(x;·):X\to Y$ .

Fact 1 In the common case of $Y=\mathbb{R}$ ,

Example 1 Let $H$ be a Hilbert space and $L\in B(H,H)$ . Define the function $f:H\to \mathbb{R}$ by $f\left(x\right)=⟨Lx,x⟩$ . What is its Gâteaux differential? From the definition,

We compute the derivative:

Therefore,

Unfortunately, the Gâbeaux differential does not satisfy our need to connect differentiability to continuity.

Definition 2 Let $f:X\to Y$ be a transformation on $D\subseteq X$ . If for each $x\in D$ and each direction $h\in X$ there exists a function $\delta f(x;h):D\times X\to Y$ that is linear and continuous with respect to $h$ such that

then f is said to be Fréchet differentiable at $x$ and $\delta f(x;h)$ is said to be the Fréchet differential of $f$ at $x$ with increment $h$ .

One can intuitively see that there is a stronger connection between the common definition of a derivative (for functions $\mathbb{R}\to \mathbb{R}$ ) and the Fréchet derivative. There are additional connections between the derivatives and their properties.

Lemma 1 If a function $f$ is Fréchet differentiable then $\delta f(x;h)$ is unique.

Lemma 2 If the Fréchet differential of $f$ exists at $x$ , then the Gâteaux differential of $f$ exists at $x$ and they are equal.

Lemma 3 If $f$ defined on an open set $D\subseteq X$ has a Fréchet differential at $x$ then $f$ is continuous at $x$ .

For a Fréchef-differentiable function, for any $ϵ>0$ there exists a sufficiently small $h\in X$ such that

This in turn implies

as $\delta f(x;h)$ is a linear continuous functional on $h$ , implying that it is bounded. Therefore, as $\parallel h\parallel \to 0$ , we have

This implies that $f$ is continuous at $x$ .