Acknowledgments
4.4 Linear algebra
4.4.1 Linear systems
In mathematics, a system described by a function f is linear if it is addi- tive:
f(x+y) = f(x) + f(y) (4.44) and if it is homogeneous,
f(αx) =αf(x) (4.45)
In simpler words, we can say that the output is proportional to the input.
As discussed in the introduction to this chapter, one of the simplest tech- niques for approximating any unknown system consists of approximating it with a linear system (and this approximation will be correct for some system and not for others).
When we try to model a new system, approximating the system with a linear system is often the first step in describing it in a quantitative way, even if it may turn out that this is not a good approximation.
This is the same as approximating the function f describing the system with the first-order Taylor expansions f(x+h)− f(x) = f0(x)h.
For a multidimensional system with inputx (now a vector) and outputy (also a vector, not necessarily of the same size asx), we can still approxi- matey= f(x)with f(y+h)−y' Ah, yet we need to clarify what this latter equation means.
Given
x≡
x0
x1
. . . xn−1
y≡
y0
y1
. . . ym−1
(4.46)
A≡
a00 a01 . . . a0,n−1
a10 a11 . . . a1,n−1
. . . . am−1,0 am−1,1 . . . am−1,n−1
(4.47)
the following equation means
y= f(x)' Ax (4.48)
which means
y0= f0(x) 'a00x0+a01x1+· · ·+a0,n−1xn−1 (4.49) y1= f1(x) 'a10x0+a11x1+· · ·+a1,n−1xn−1 (4.50) y2= f2(x) 'a20x0+a21x1+· · ·+a2,n−1xn−1 (4.51)
. . .=. . . '. . . (4.52)
ym−1= fm−1(x)'am−1,0x0+am−1,1x1+. . .am−1,n−1xn−1 (4.53) which says that every output variableyj is approximated with a function proportional to each of the input variablesxi.
A system is linear if the' relations turn out to be exact and can be re- placed by=symbols.
As a corollary of the basic properties of a linear system discussed earlier, linear systems have one nice additional property. If we combine two linear systemsy =Axandz= By, the combined system is also a linear system z= (BA)x.
Elementary algebrais defined as a set of numbers (e.g., real numbers) en- dowed with the ordinary four elementary operations (+,−,×,/).
Abstract algebra is a generalization of the concept of elementary algebra to other sets of objects (not necessarily numbers) by definition operations among them such as addition and multiplication.
Linear algebrais the extension of elementary algebra to matrices (and vec- tors, which can be seen as special types of matrices) by defining the four elementary operations among them.
We will implement them in code using Python. In Python, we can imple- ment a matrix as a list of lists, as follows:
1 >>> A = [[1,2,3],[4,5,6],[7,8,9]]
But such an object (a list of lists) does not have the mathematical proper- ties we want, so we have to define them.
First, we define a class representing a matrix:
Listing4.12: in file:nlib.py
1 class Matrix(object):
2 def __init__(self,rows,cols=1,fill=0.0):
3 """
4 Constructor a zero matrix
5 Examples:
6 A = Matrix([[1,2],[3,4]])
7 A = Matrix([1,2,3,4])
8 A = Matrix(10,20)
9 A = Matrix(10,20,fill=0.0)
10 A = Matrix(10,20,fill=lambda r,c: 1.0 if r==c else 0.0)
11 """
12 if isinstance(rows,list):
13 if isinstance(rows[0],list):
14 self.rows = [[e for e in row] for row in rows]
15 else:
16 self.rows = [[e]for e in rows]
17 elif isinstance(rows,int) and isinstance(cols,int):
18 xrows, xcols = xrange(rows), xrange(cols)
19 if callable(fill):
20 self.rows = [[fill(r,c) for c in xcols] for r in xrows]
21 else:
22 self.rows = [[fill for c in xcols] for r in xrows]
23 else:
24 raise RuntimeError("Unable to build matrix from %s" % repr(rows))
25 self.nrows = len(self.rows)
26 self.ncols = len(self.rows[0])
Notice that the constructor takes the number of rows and columns (cols) of the matrix but also a fill value, which can be used to initialize the matrix elements and defaults to zero. It can be callable in case we need to initialize the matrix with row,col dependent values.
The actual matrix elements are stored as a list or array into thedatamem- ber variable. If optimize=True, the data are stored in an array of double precision floating point numbers (“d”). This optimization will prevent you from building matrices of complex numbers or matrices of arbitrary precision decimal numbers.
Now we define a getter method, a setter method, and a string representa- tion for the matrix elements:
Listing4.13: in file: nlib.py
1 def __getitem__(A,coords):
2 " x = A[0,1]"
3 i,j = coords
4 returnA.rows[i][j]
5
6 def __setitem__(A,coords,value):
7 " A[0,1] = 3.0 "
8 i,j = coords
9 A.rows[i][j] = value
10
11 def tolist(A):
12 " assert(Matrix([[1,2],[3,4]]).tolist() == [[1,2],[3,4]]) "
13 returnA.rows
14
15 def __str__(A):
16 return str(A.rows)
17
18 def flatten(A):
19 " assert(Matrix([[1,2],[3,4]]).flatten() == [1,2,3,4]) "
20 return[A[r,c] for r in xrange(A.nrows) for c in xrange(A.ncols)]
21
22 def reshape(A,n,m):
23 " assert(Matrix([[1,2],[3,4]]).reshape(1,4).tolist() == [[1,2,3,4]]) "
24 if n*m != A.nrows*A.ncols:
25 raise RuntimeError("Impossible reshape")
26 flat = A.flatten()
27 returnMatrix(n,m,fill=lambda r,c,m=m,flat=flat: flat[r*m+c])
28
29 def swap_rows(A,i,j):
30 " assert(Matrix([[1,2],[3,4]]).swap_rows(1,0).tolist() == [[3,4],[1,2]])
"
31 A.rows[i], A.rows[j] = A.rows[j], A.rows[i]
We also define some convenience functions for constructing the identity matrix (given its size) and a diagonal matrix (given the diagonal ele- ments). We make these methods static because they do not act on an existing matrix.
Listing4.14: in file:nlib.py
1 @staticmethod
2 def identity(rows=1,e=1.0):
3 returnMatrix(rows,rows,lambda r,c,e=e: e if r==c else 0.0)
4
5 @staticmethod
6 def diagonal(d):
7 returnMatrix(len(d),len(d),lambda r,c,d=d:d[r] if r==c else 0.0)
Now we are ready to define arithmetic operations among matrices. We start with addition and subtraction:
Listing4.15: in file: nlib.py
1 def __add__(A,B):
2 """
3 Adds A and B element by element, A and B must have the same size
4 Example
5 >>> A = Matrix([[4,3.0], [2,1.0]])
6 >>> B = Matrix([[1,2.0], [3,4.0]])
7 >>> C = A + B
8 >>> print C
9 [[5, 5.0], [5, 5.0]]
10 """
11 n, m = A.nrows, A.ncols
12 if not isinstance(B,Matrix):
13 if n==m:
14 B = Matrix.identity(n,B)
15 elif n==1 or m==1:
16 B = Matrix([[B for c in xrange(m)] for r in xrange(n)])
17 if B.nrows!=n or B.ncols!=m:
18 raise ArithmeticError('incompatible dimensions')
19 C = Matrix(n,m)
20 for r in xrange(n):
21 for c in xrange(m):
22 C[r,c] = A[r,c]+B[r,c]
23 return C
24
25 def __sub__(A,B):
26 """
27 Adds A and B element by element, A and B must have the same size
28 Example
29 >>> A = Matrix([[4.0,3.0], [2.0,1.0]])
30 >>> B = Matrix([[1.0,2.0], [3.0,4.0]])
31 >>> C = A - B
32 >>> print C
33 [[3.0, 1.0], [-1.0, -3.0]]
34 """
35 n, m = A.nrows, A.ncols
36 if not isinstance(B,Matrix):
37 if n==m:
38 B = Matrix.identity(n,B)
39 elif n==1 or m==1:
40 B = Matrix(n,m,fill=B)
41 if B.nrows!=n or B.ncols!=m:
42 raise ArithmeticError('Incompatible dimensions')
43 C = Matrix(n,m)
44 for r in xrange(n):
45 for c in xrange(m):
46 C[r,c] = A[r,c]-B[r,c]
47 return C
48 def __radd__(A,B): #B+A
49 returnA+B
50 def __rsub__(A,B): #B-A
51 return(-A)+B
52 def __neg__(A):
53 returnMatrix(A.nrows,A.ncols,fill=lambda r,c:-A[r,c])
With the preceding definitions, we can add matrices to matrices, subtract matrices from matrices, but also add and subtract scalars to and from matrices and vectors (scalars are interpreted as diagonal matrices when added to square matrices and as constant vectors when added to vectors).
Here are some examples:
Listing4.16: in file:nlib.py
1 >>> A = Matrix([[1.0,2.0],[3.0,4.0]])
2 >>> print A + A # calls A.__add__(A)
3 [[2.0, 4.0], [6.0, 8.0]]
4 >>> print A + 2 # calls A.__add__(2)
5 [[3.0, 2.0], [3.0, 6.0]]
6 >>> print A - 1 # calls A.__add__(1)
7 [[0.0, 2.0], [3.0, 3.0]]
8 >>> print -A # calls A.__neg__()
9 [[-1.0, -2.0], [-3.0, -4.0]]
10 >>> print 5 - A # calls A.__rsub__(5)
11 [[4.0, -2.0], [-3.0, 1.0]]
12 >>> b = Matrix([[1.0],[2.0],[3.0]])
13 >>> print b + 2 # calls b.__add__(2)
14 [[3.0], [4.0], [5.0]]
The classMatrixworks with complex numbers as well:
Listing4.17: in file:nlib.py
1 >>> A = Matrix([[1,2],[3,4]])
2 >>> print A + 1j
3 [[(1+1j), (2+0j)], [(3+0j), (4+1j)]]
Now we implement multiplication. We are interested in three types of multiplication: multiplication of a scalar by a matrix (__rmul__), multipli- cation of a matrix by a matrix (__mul__), and scalar product between two vectors (also handled by__mul__):
Listing4.18: in file:nlib.py
1 def __rmul__(A,x):
2 "multiplies a number of matrix A by a scalar number x"
3 importcopy
4 M = copy.deepcopy(A)
5 for r in xrange(M.nrows):
6 for c in xrange(M.ncols):
7 M[r,c] *= x
8 return M
9
10 def __mul__(A,B):
11 "multiplies a number of matrix A by another matrix B"
12 if isinstance(B,(list,tuple)):
13 return (A*Matrix(len(B),1,fill=lambda r,c:B[r])).nrows
14 elif not isinstance(B,Matrix):
15 return B*A
16 elif A.ncols == 1 and B.ncols==1 and A.nrows == B.nrows:
17 # try a scalar product ;-)
18 return sum(A[r,0]*B[r,0] for r in xrange(A.nrows))
19 elif A.ncols!=B.nrows:
20 raise ArithmeticError('Incompatible dimension')
21 M = Matrix(A.nrows,B.ncols)
22 for r in xrange(A.nrows):
23 for c in xrange(B.ncols):
24 for k in xrange(A.ncols):
25 M[r,c] += A[r,k]*B[k,c]
26 return M
This allows us the following operations:
Listing4.19: in file: nlib.py
1 >>> A = Matrix([[1.0,2.0],[3.0,4.0]])
2 >>> print(2*A) # scalar * matrix
3 [[2.0, 4.0], [6.0, 8.0]]
4 >>> print(A*A) # matrix * matrix
5 [[7.0, 10.0], [15.0, 22.0]]
6 >>> b = Matrix([[1],[2],[3]])
7 >>> print(b*b) # scalar product
8 14