Similar to the real line concerning two real scalars and the distance between them, $|a - b|$, vector norms allow us to get a sense of the distance or magnitude of a vector. In fact, a vector of length one is simply a scalar. Norms are often used in regularization methods and other machine learning procedures, as well as many different matrix and vector operations in linear algebra.
Formally, given a vector space $\mathbb{R}^n$, a vector norm is defined as a function $f: \mathbb{R}^n \rightarrow \mathbb{R}$. Norms are represented by double-bar notation, for example, a norm $x$ would be denoted $||x||$. Vector norms have the following properties:
- $\Vert \vec{x} \Vert > 0$ for a vector $x \in \mathbb{R}^n$
- $\Vert \vec{x} \Vert = 0$ if the vector $\vec{x} = 0$
- $\Vert \alpha \vec{x} \vert = \vert \alpha \vert \Vert \vec{x} \Vert $ for a vector $\vec{x} \in \mathbb{R}^n$ and a scalar $\alpha \in \mathbb{R}$
- $\Vert \vec{x} + \vec{y} \Vert \leq \Vert \vec{x} \Vert + \Vert \vec{y} \Vert$ for vectors $\vec{x},\vec{y} \in \mathbb{R}^n$
The most commonly occurring vector norms are the $1$, $2$, and $\infty$ norms, which are a part of the $p$-norm class of vector norms.
Vector Norms¶
The following will explore the most frequently occurring vector norms $1$, $2$, and $\infty$ and some Python implementations of the norms. We first create an array using numpy's randint function that will be used as our example vector.
import numpy as np
x = np.random.randint(-10, 10, 10)
x
1-Norm¶
The 1-norm of a vector $x$ is defined as:
$$ \large \Vert \vec{x} \Vert_1 = \vert x_1 \vert + \vert x_2 \vert + \cdots + \vert x_n \vert $$The $1$-norm is often denoted $\textit{l}_1$ or $L^1$ and is also referred to as the Taxicab norm. The following is an implementation of the calculation of the $1$-norm.
norm1 = 0
for i in x:
norm1 += np.abs(i)
print(norm1)
Rather than writing a (comparatively slow) loop, we can take advantage of numpy's vectorization to calculate the $1$-norm as a one-liner.
np.sum(np.abs(x))
We can confirm our results with Numpy's norm function.
np.linalg.norm(x, 1)
2-Norm¶
The $2$-norm of a vector $x$ is defined as:
$$ \large \Vert \vec{x} \Vert _2 = \sqrt{x_1^2 + x_2^2 + \cdots + x_n^2} = \sqrt{\vec{x}^T \vec{x}} $$The $2$-norm is often denoted as $\textit{l}_2$ or $L^2$ and is also referred to as the Euclidean length or distance. An example implementation of the $2$-norm calculation is somewhat similar to the $1$-norm computation.
norm2 = 0
for i in x:
norm2 += i ** 2
print(np.sqrt(norm2))
Another example of computing the $2$-norm of a vector as a one-liner.
np.sqrt(np.sum(np.power(x, 2)))
Given the definition of the $2$-norm above, we know the above computations is also equal to the square root of the inner product of the vector and the conjugate transpose. Numpy provides an inner function for computing the inner product.
np.sqrt(np.inner(x, x))
As before, we can verify our calculations of the $2$-norm by using numpy's norm function.
np.linalg.norm(x, 2)
P-Norm¶
The p-norm, which is considered a class of vector norms is defined as:
$$ \large \Vert \vec{x} \Vert_p = \sqrt[p]{\vert x_1 \vert^p + \vert x_2 \vert^p + \cdots + \vert x_n \vert^p} \qquad p \geq 1 $$By setting $p$ equal to $1$ or $2$, we can find the $1$ and $2$-norm of a vector without the need for separate equations and functions. Below we calculate the $2$-norm of a vector using the $p$-norm equation.
pnorm = 0
p = 2
for i in x:
pnorm += np.abs(i) ** p
pnorm ** (1. / p)
A more simple and one-liner implementation of the $p$-norm class of vector norms.
np.sum(np.abs(x) ** p) ** (1. / p)
The $p$-norm is not limited to $p = 1$ or $p = 2$, but $p$ cannot be less than $1$ or complex. Here, we compute the vector $3$-norm and compare the results with numpy's norm function.
p = 3
print(np.sum(np.abs(x) ** p) ** (1. / p))
print(np.linalg.norm(x, 3))
Inf-Norm¶
The $\infty$ norm of a vector $\vec{x}$ is defined as:
$$ \large \Vert \vec{x} \Vert_{\infty} = \underset{1 \leq i \leq n}{\max} \vert x_i \vert $$The $\infty$-norm of a vector is the maximum absolute value of the elements in the vector. Below is one possible example implementation of finding the $\infty$-norm of a vector.
x_abs = np.abs(x)
infnorm = x_abs[0]
for i in x_abs[1:]:
if infnorm < i:
infnorm = i
infnorm
A much more easy and straightforward approach to finding the $\infty$-norm using numpy's max and absolute functions.
np.max(np.abs(x))
As before, we can compare our results with numpy's norm function by setting the function's ord parameter ($p$) to numpy's inf constant.
np.linalg.norm(x, np.inf)
Lastly, we take a look at the $-\infty$-norm, which can be thought of as the opposite of the $\infty$-norm.
-Inf-Norm¶
The $-\infty$ norm of a vector $\vec{x}$ is defined as:
$$ \large \Vert \vec{x} \Vert_{\infty} = \underset{1 \leq i \leq n}{\min} \vert x_i \vert $$The $-\infty$-norm of a vector is the minimum absolute value of the elements in the vector. The following example implementation of finding the $-\infty$-norm is quite similar to the $\infty$-norm implementation example, except here we are looking for the smallest value rather than the maximum value.
neg_infnorm = x_abs[0]
for i in x_abs[1:]:
if neg_infnorm > i:
neg_infnorm = i
neg_infnorm
A much more easy and simple implementation of finding the $-\infty$-norm of a vector.
np.min(np.abs(x))
Again, setting the ord parameter to numpy's negative inf constant, we can confirm our calculations of the $-\infty$-norm above.
np.linalg.norm(x, -np.inf)
Inequalities¶
Now that the most common and important vector norms have been introduced, we turn our attention towards some interesting inequalities that arise from the study of vector norms.
Cauchy-Schwartz Inequality¶
In the linear and vector algebra, The Cauchy-Schwartz inequality, sometimes referred to as the Cauchy–Bunyakovsky–Schwarz inequality states that for any two vectors, denoted $\vec{x}$ and $\vec{y}$, the following is true:
$$ \large \vert \langle \vec{x}, \vec{y} \rangle \vert^2 \geq \langle \vec{x}, \vec{x} \rangle \cdot \langle \vec{y}, \vec{y} \rangle $$The inequality can be rewritten for vector norms as the following:
$$ \large |\vec{x}^T \vec{y}| \leq \Vert \vec{x} \Vert_2\Vert \vec{y} \Vert_2 $$Which states that for any two vectors $\vec{x}$ and $\vec{y}$, the inner product of $\vec{x}$ and $\vec{y}$ will always be less than or equal to the product of the $2$-norms of the vectors.
The proof of the Cauchy-Schwartz inequality is beyond the scope of this post; however, we can write a simple function to demonstrate the inequality. The following function creates two vectors of length $10$ with elements between $1$ and $100$ using numpy's randint function. The inner product is computed with numpy's inner function and is then compared with the right side of the inequality. The function will return True if the inequality holds and False otherwise. Thus, the function should always return True.
def cauchy_schwartz_inequality():
x = np.random.randint(1, 100, 10)
y = np.random.randint(1, 100, 10)
inner_prod = np.inner(x, y)
right = np.sqrt(np.sum(np.power(x, 2))) * np.sqrt(np.sum(np.power(y, 2)))
return inner_prod <= right
The following tests the inequality by running the cauchy_schwartz_inequality() function created above $100,000$ times and appends the results to a list. Using Python's any method, we can then see if any of the appended results in the list is False.
res = []
for i in range(0, 100000):
res.append(cauchy_schwartz_inequality())
any(x == False for x in res)
Hölder's Inequality¶
It can be shown the Cauchy-Schwartz inequality is a special case of Hölder's Inequality, which in the linear algebra setting can be defined as:
$$ \large \vert \vec{x}^T \vec{y} \vert \leq \Vert \vec{x} \Vert_p \Vert \vec{y} \Vert_q, \qquad \frac{1}{p} + \frac{1}{q} = 1 $$The variables $p$ and $q$ are known as Hölder conjugates. The inequality thus states that the inner product of two vectors $\vec{x}$ and $\vec{y}$ will always be less than equal to the product of $p$-norms of the vectors where $\frac{1}{p} + \frac{1}{q} = 1$. As before, the proof of Hölder's inequality is beyond the scope of this post, but we can write a function to demonstrate the inequality. The following function creates two vectors of length ten with random elements between $1$ and $100$ and a number $p$ between $2$ and $10$. The conjugate number $q$ is then found, and the inner product and right side of the inequality are computed. The function then returns True if the inner product is less than or greater to the right side of the inequality and False otherwise. We also return the calculation $1 / p + 1 / q$ to ensure it is always equal to $1$.
def holder_inequality():
x = np.random.randint(1, 100, 10)
y = np.random.randint(1, 100, 10)
p = np.random.randint(2, 10)
q = p / (p - 1)
inner_prod = np.inner(x, y)
right = np.sum(np.abs(x) ** p) ** (1. / p) * np.sum(np.abs(y) ** q) ** (1. / q)
return inner_prod <= right, int(1 / p + 1 / q)
We then test the inequality by running the function $100,000$ times and append the results to a list. Given the inequality's definition, the result should always be True and $1$ for the $1/p + 1/q$ calculation.
res = []
for i in range(0, 100000):
res.append(holder_inequality())
print('False values exist in result:', any(x[0] == False for x in res))
print('All conjugate calculations are equal to 1:', all(x[1] == 1 for x in res))
Minkowski's Inequality¶
The final inequality we shall investigate is known as Minkowski's inequality for norms. Minkowski's inequality for norms is also known as the triangle inequality. For two vectors, $\vec{x}$ and $\vec{y}$ of length $n$, Minkowski's inequality can be written as:
$$ \large \left( \sum^n_{i=1} \left(\vert \vec{x}_i + \vec{y}_i \vert \right)^p \right)^{\frac{1}{p}} \leq \left(\sum^n_{i=1} \vert \vec{x}_i \vert^p \right)^{\frac{1}{p}} + \left(\sum^n_{i=1} \vert \vec{y}_i \vert^p \right)^{\frac{1}{p}} $$Which can be more simply written using vector norms as:
$$ \large \Vert \vec{x} + \vec{y} \Vert_p \leq \Vert \vec{x} \Vert_p \Vert \vec{y} \Vert_p $$Similar to the other inequalities we examined previously, we can demonstrate Minkowski's inequality for norms by writing a fairly straightforward function. The function minkowski_inequality() creates two vectors of length ten with random integer elements between $1$ and $100$ and a $p$ number between $1$ and $5$. The left and right sides of the inequality are then computed, and the function returns True if the left side is less than or equal to the right side and False otherwise.
def minkowski_inequality():
x = np.random.randint(1, 100, 10)
y = np.random.randint(1, 100, 10)
p = np.random.randint(1, 5)
left = np.sum(np.abs(x + y) ** p) ** (1. / p)
right = np.sum(np.abs(x) ** p) ** (1. / p) + np.sum(np.abs(y) ** p) ** (1. / p)
return left <= right
Given the definition of Minkowski's inequality, the function should always return True. We test this by running the function $100,000$ times and appending the results to a list. Then, with Python's any function, we check that the appended list of results does not contain a False value.
res = []
for i in range(0, 100000):
res.append(minkowski_inequality())
any(x == False for x in res)
References¶
Weisstein, Eric W. "Vector Norm." From MathWorld--A Wolfram Web Resource. https://mathworld.wolfram.com/VectorNorm.