Analytic Geometry
3.2 Inner Products
Inner products allow for the introduction of intuitive geometrical con- cepts, such as the length of a vector and the angle or distance between two vectors. A major purpose of inner products is to determine whether vectors are orthogonal to each other.
3.2.1 Dot Product
We may already be familiar with a particular type of inner product, the scalar product/dot productinRn, which is given by
scalar product dot product
x⊤y=
n
X
i=1
xiyi. (3.5)
We will refer to this particular inner product as the dot product in this book. However, inner products are more general concepts with specific properties, which we will now introduce.
3.2.2 General Inner Products
Recall the linear mapping from Section 2.7, where we can rearrange the mapping with respect to addition and multiplication with a scalar. Abi-
bilinear mapping
linear mapping Ω is a mapping with two arguments, and it is linear in each argument, i.e., when we look at a vector spaceV then it holds that for allx,y,z ∈V, λ, ψ∈Rthat
Ω(λx+ψy,z) =λΩ(x,z) +ψΩ(y,z) (3.6) Ω(x, λy+ψz) =λΩ(x,y) +ψΩ(x,z). (3.7) Here, (3.6) asserts thatΩis linear in the first argument, and (3.7) asserts thatΩis linear in the second argument (see also (2.87)).
3.2 Inner Products 73 Definition 3.2. LetV be a vector space andΩ :V ×V →Rbe a bilinear mapping that takes two vectors and maps them onto a real number. Then
Ω is called symmetricif Ω(x,y) = Ω(y,x) for all x,y ∈ V, i.e., the symmetric
order of the arguments does not matter.
Ωis calledpositive definiteif positive definite
∀x∈V\{0}: Ω(x,x)>0, Ω(0,0) = 0. (3.8) Definition 3.3. LetV be a vector space andΩ :V ×V →Rbe a bilinear mapping that takes two vectors and maps them onto a real number. Then A positive definite, symmetric bilinear mappingΩ :V×V →Ris called
aninner productonV. We typically write⟨x,y⟩instead ofΩ(x,y). inner product
The pair(V,⟨·,·⟩)is called aninner product spaceor (real)vector space inner product space vector space with inner product
with inner product. If we use the dot product defined in (3.5), we call (V,⟨·,·⟩)aEuclidean vector space.
Euclidean vector space
We will refer to these spaces as inner product spaces in this book.
Example 3.3 (Inner Product That Is Not the Dot Product) ConsiderV =R2. If we define
⟨x,y⟩:=x1y1−(x1y2+x2y1) + 2x2y2 (3.9) then⟨·,·⟩is an inner product but different from the dot product. The proof will be an exercise.
3.2.3 Symmetric, Positive Definite Matrices
Symmetric, positive definite matrices play an important role in machine learning, and they are defined via the inner product. In Section 4.3, we will return to symmetric, positive definite matrices in the context of matrix decompositions. The idea of symmetric positive semidefinite matrices is key in the definition of kernels (Section 12.4).
Consider ann-dimensional vector spaceV with an inner product⟨·,·⟩: V ×V →R(see Definition 3.3) and an ordered basisB= (b1, . . . ,bn)of V. Recall from Section 2.6.1 that any vectorsx,y∈ V can be written as linear combinations of the basis vectors so thatx = Pni=1ψibi ∈ V and y =Pnj=1λjbj ∈V for suitableψi, λj ∈R. Due to the bilinearity of the inner product, it holds for allx,y∈V that
⟨x,y⟩=
* n X
i=1
ψibi,
n
X
j=1
λjbj
+
=
n
X
i=1 n
X
j=1
ψi⟨bi,bj⟩λj= ˆx⊤Aˆy, (3.10) whereAij :=⟨bi,bj⟩andx,ˆ yˆare the coordinates ofxandywith respect to the basisB. This implies that the inner product⟨·,·⟩is uniquely deter- mined throughA. The symmetry of the inner product also means thatA
is symmetric. Furthermore, the positive definiteness of the inner product implies that
∀x∈V\{0}:x⊤Ax>0. (3.11) Definition 3.4(Symmetric, Positive Definite Matrix). A symmetric matrix A ∈ Rn×n that satisfies (3.11) is called symmetric, positive definite, or
symmetric, positive
definite justpositive definite. If only⩾holds in (3.11), thenAis calledsymmetric,
positive definite symmetric, positive semidefinite
positive semidefinite.
Example 3.4 (Symmetric, Positive Definite Matrices) Consider the matrices
A1 = 9 6
6 5
, A2= 9 6
6 3
. (3.12)
A1is positive definite because it is symmetric and x⊤A1x=x1 x2
9 6
6 5 x1
x2
(3.13a)
= 9x21+ 12x1x2+ 5x22= (3x1+ 2x2)2+x22 >0 (3.13b) for allx∈ V\{0}. In contrast,A2 is symmetric but not positive definite becausex⊤A2x = 9x21+ 12x1x2+ 3x22 = (3x1+ 2x2)2−x22can be less than0, e.g., forx= [2,−3]⊤.
IfA∈Rn×nis symmetric, positive definite, then
⟨x,y⟩= ˆx⊤Ayˆ (3.14) defines an inner product with respect to an ordered basisB, wherexˆ and ˆ
yare the coordinate representations ofx,y∈V with respect toB. Theorem 3.5. For a real-valued, finite-dimensional vector spaceV and an ordered basisBofV, it holds that⟨·,·⟩:V ×V →Ris an inner product if and only if there exists a symmetric, positive definite matrixA∈Rn×nwith
⟨x,y⟩= ˆx⊤Aˆy. (3.15) The following properties hold ifA ∈ Rn×n is symmetric and positive definite:
The null space (kernel) ofAconsists only of0becausex⊤Ax >0for allx̸=0. This implies thatAx̸=0ifx̸=0.
The diagonal elementsaii ofAare positive becauseaii =e⊤i Aei >0, whereeiis theith vector of the standard basis inRn.
3.3 Lengths and Distances 75 3.3 Lengths and Distances
In Section 3.1, we already discussed norms that we can use to compute the length of a vector. Inner products and norms are closely related in the
sense that any inner product induces a norm Inner products induce norms.
∥x∥:=q⟨x,x⟩ (3.16)
in a natural way, such that we can compute lengths of vectors using the in- ner product. However, not every norm is induced by an inner product. The Manhattan norm (3.3) is an example of a norm without a corresponding inner product. In the following, we will focus on norms that are induced by inner products and introduce geometric concepts, such as lengths, dis- tances, and angles.
Remark(Cauchy-Schwarz Inequality). For an inner product vector space
(V,⟨·,·⟩)the induced norm∥ · ∥satisfies theCauchy-Schwarz inequality Cauchy-Schwarz inequality
| ⟨x,y⟩ |⩽∥x∥∥y∥. (3.17)
♢
Example 3.5 (Lengths of Vectors Using Inner Products)
In geometry, we are often interested in lengths of vectors. We can now use an inner product to compute them using (3.16). Let us takex= [1,1]⊤∈ R2. If we use the dot product as the inner product, with (3.16) we obtain
∥x∥=√
x⊤x=√
12+ 12 =√
2 (3.18)
as the length ofx. Let us now choose a different inner product:
⟨x,y⟩:=x⊤
1 −12
−12 1
y=x1y1−1
2(x1y2+x2y1) +x2y2. (3.19) If we compute the norm of a vector, then this inner product returns smaller values than the dot product ifx1andx2have the same sign (andx1x2>
0); otherwise, it returns greater values than the dot product. With this inner product, we obtain
⟨x,x⟩=x21−x1x2+x22= 1−1 + 1 = 1 =⇒ ∥x∥=√
1 = 1, (3.20) such thatxis “shorter” with this inner product than with the dot product.
Definition 3.6 (Distance and Metric). Consider an inner product space (V,⟨·,·⟩). Then
d(x,y) :=∥x−y∥=q⟨x−y,x−y⟩ (3.21)
is called the distancebetween x andy for x,y ∈ V. If we use the dot distance
product as the inner product, then the distance is calledEuclidean distance. Euclidean distance
The mapping
d:V ×V →R (3.22)
(x,y)7→d(x,y) (3.23)
is called ametric.
metric
Remark. Similar to the length of a vector, the distance between vectors does not require an inner product: a norm is sufficient. If we have a norm induced by an inner product, the distance may vary depending on the
choice of the inner product. ♢
A metricdsatisfies the following:
1. dispositive definite, i.e., d(x,y) ⩾ 0 for allx,y ∈ V andd(x,y) =
positive definite
0 ⇐⇒ x=y.
2. dissymmetric, i.e.,d(x,y) =d(y,x)for allx,y∈V.
symmetric
triangle inequality 3. Triangle inequality:d(x,z)⩽d(x,y) +d(y,z)for allx,y,z ∈V. Remark. At first glance, the lists of properties of inner products and met- rics look very similar. However, by comparing Definition 3.3 with Defini- tion 3.6 we observe that⟨x,y⟩andd(x,y)behave in opposite directions.
Very similarxandywill result in a large value for the inner product and
a small value for the metric. ♢