From A First Course in Linear Algebra
Version 1.21
© 2004.
Licensed under the GNU Free Documentation License.
http://linear.ups.edu/
In any linearly dependent set there is always one vector that can be written as a
linear combination of the others. This is the substance of the upcoming
Theorem DLDS. Perhaps this will explain the use of the word “dependent.” In a
linearly dependent set, at least one vector “depends” on the others (via a linear
combination).
Indeed, because Theorem DLDS is an equivalence (Technique E) some authors use this condition as a definition (Technique D) of linear dependence. Then linear independence is defined as the logical opposite of linear dependence. Of course, we have chosen to take Definition LICV as our definition, and then follow with Theorem DLDS as a theorem.
If we use a linearly dependent set to construct a span, then we can always create the same infinite set with a starting set that is one vector smaller in size. We will illustrate this behavior in Example RSC5. However, this will not be possible if we build a span from a linearly independent set. So in a certain sense, using a linearly independent set to formulate a span is the best possible way — there aren’t any extra vectors being used to build up all the necessary linear combinations. OK, here’s the theorem, and then the example.
Theorem DLDS
Dependency in Linearly Dependent Sets
Suppose that is a
set of vectors. Then
is a linearly dependent set if and only if there is an index
,
such that
is a linear combination
of the vectors .
Proof () Suppose that is linearly dependent, so there exists a nontrivial relation of linear dependence by Definition LICV. That is, there are scalars, , , which are not all zero, such that
Since the cannot all be zero, choose one, say , that is nonzero. Then,
Since the values of are again scalars, we have expressed as a linear combination of the other elements of .
() Assume that the vector is a linear combination of the other vectors in . Write this linear combination, denoting the relevant scalars as , , …, , , …, as
Then we have
So the scalars provide a nontrivial linear combination of the vectors in , thus establishing that is a linearly dependent set (Definition LICV).
This theorem can be used, sometimes repeatedly, to whittle down the size of a set of vectors used in a span construction. We have seen some of this already in Example SCAD, but in the next example we will detail some of the subtleties.
Example RSC5
Reducing a span in
Consider the set of
vectors from ,
and define .
To employ Theorem LIVHS, we form a coefficient matrix, ,
and row-reduce to understand solutions to the homogeneous system ,
We can find infinitely many solutions to this system, most of them nontrivial, and we choose any one we like to build a relation of linear dependence on . Let’s begin with , to find the solution
So we can write the relation of linear dependence,
Theorem DLDS guarantees that we can solve this relation of linear dependence for some vector in , but the choice of which one is up to us. Notice however that has a zero coefficient. In this case, we cannot choose to solve for . Maybe some other relation of linear dependence would produce a nonzero coefficient for if we just had to solve for this vector. Unfortunately, this example has been engineered to always produce a zero coefficient here, as you can see from solving the homogeneous system. Every solution has !
OK, if we are convinced that we cannot solve for , let’s instead solve for ,
We now claim that this particular equation will allow us to write
in essence declaring as surplus for the task of building as a span. This claim is an equality of two sets, so we will use Definition SE to establish it carefully. Let and . We want to show that .
First show that . Since every vector of is in , any vector we can construct in as a linear combination of vectors from can also be constructed as a vector in by the same linear combination of the same vectors in . That was easy, now turn it around.
Next show that . Choose any from . Then there are scalars so that
This equation says that can then be written as a linear combination of the vectors in and hence qualifies for membership in . So and we have established that .
If was also linearly dependent (its not), we could reduce the set even further. Notice that we could have chosen to eliminate any one of , or , but somehow is essential to the creation of since it cannot be replaced by any linear combination of , or .
In Example RSC5 we used four vectors to create a span. With a relation of linear dependence in hand, we were able to “toss-out” one of these four vectors and create the same span from a subset of just three vectors from the original set of four. We did have to take some care as to just which vector we tossed-out. In the next example, we will be more methodical about just how we choose to eliminate vectors from a linearly dependent set while preserving a span.
Example COV
Casting out vectors
We begin with a set
containing seven vectors from ,
and define . The set is obviously linearly dependent by Theorem MVSLD, since we have vectors from . So we can slim down some, and still create as the span of a smaller set of vectors. As a device for identifying relations of linear dependence among the vectors of , we place the seven column vectors of into a matrix as columns,
By Theorem SLSLC a nontrivial solution to will give us a nontrivial relation of linear dependence (Definition RLDCV) on the columns of (which are the elements of the set ). The row-reduced form for is the matrix
so we can easily create solutions to the homogeneous system using the free variables . Any such solution will correspond to a relation of linear dependence on the columns of . These solutions will allow us to solve for one column vector as a linear combination of some others, in the spirit of Theorem DLDS, and remove that vector from the set. We’ll set about forming these linear combinations methodically. Set the free variable to one, and set the other free variables to zero. Then a solution to is
which can be used to create the linear combination
This can then be arranged and solved for , resulting in expressed as a linear combination of ,
This means that is surplus, and we can create just as well with a smaller set with this vector removed,
Technically, this set equality for requires a proof, in the spirit of Example RSC5, but we will bypass this requirement here, and in the next few paragraphs.
Now, set the free variable to one, and set the other free variables to zero. Then a solution to is
which can be used to create the linear combination
This can then be arranged and solved for , resulting in expressed as a linear combination of ,
This means that is surplus, and we can create just as well with a smaller set with this vector removed,
Do it again, set the free variable to one, and set the other free variables to zero. Then a solution to is
which can be used to create the linear combination
This can then be arranged and solved for , resulting in expressed as a linear combination of ,
This means that is surplus, and we can create just as well with a smaller set with this vector removed,
Set the free variable to one, and set the other free variables to zero. Then a solution to is
which can be used to create the linear combination
This can then be arranged and solved for , resulting in expressed as a linear combination of ,
This means that is surplus, and we can create just as well with a smaller set with this vector removed,
You might think we could keep this up, but we have run out of free variables. And not coincidentally, the set is linearly independent (check this!). It should be clear how each free variable was used to eliminate the corresponding column from the set used to span the column space, as this will be the essence of the proof of the next theorem. The column vectors in were not chosen entirely at random, they are the columns of Archetype I. See if you can mimic this example using the columns of Archetype J. Go ahead, we’ll go grab a cup of coffee and be back before you finish up.
For extra credit, notice that the vector
is the vector of constants in the definition of Archetype I. Since the system is consistent, we know by Theorem SLSLC that is a linear combination of the columns of , or stated equivalently, . This means that must also be a linear combination of just the three columns . Can you find such a linear combination? Did you notice that there is just a single (unique) answer? Hmmmm.
Example COV deserves your careful attention, since this important example motivates the following very fundamental theorem.
Theorem BS
Basis of a Span
Suppose that is a set of
column vectors. Define
and let
be the matrix whose columns are the vectors from
. Let
be the reduced
row-echelon form of ,
with
the set of column indices corresponding to the pivot columns of
.
Then
Proof To prove that is linearly independent, begin with a relation of linear dependence on ,
and we will try to conclude that the only possibility for the scalars is that they are all zero. Denote the non-pivot columns of by . Then we can preserve the equality by adding a big fat zero to the linear combination,
By Theorem SLSLC, the scalars in this linear combination (suitably reordered) are a solution to the homogeneous system . But notice that this is the solution obtained by setting each free variable to zero. If we consider the description of a solution vector in the conclusion of Theorem VFSLS, in the case of a homogeneous system, then we see that if all the free variables are set to zero the resulting solution vector is trivial (all zeros). So it must be that , . This implies by Definition LICV that is a linearly independent set.
The second conclusion of this theorem is an equality of sets (Definition SE). Since is a subset of , any linear combination of elements of the set can also be viewed as a linear combination of elements of the set . So . It remains to prove that .
For each , , form a solution to by setting the free variables as follows:
By Theorem VFSLS, the remainder of this solution vector is given by,
From this solution, we obtain a relation of linear dependence on the columns of ,
which can be arranged as the equality
Now, suppose we take an arbitrary element, , of and write it as a linear combination of the elements of , but with the terms organized according to the indices in and ,
From the above, we can replace each by a linear combination of the ,
This mess expresses the vector as a linear combination of the vectors in
thus saying that . Therefore, .
In Example COV, we tossed-out vectors one at a time. But in each instance, we rewrote the offending vector as a linear combination of those vectors that corresponded to the pivot columns of the reduced row-echelon form of the matrix of columns. In the proof of Theorem BS, we accomplish this reduction in one big step. In Example COV we arrived at a linearly independent set at exactly the same moment that we ran out of free variables to exploit. This was not a coincidence, it is the substance of our conclusion of linear independence in Theorem BS.
Here’s a straightfoward application of Theorem BS.
Example RSSC4
Reducing a span in
Begin with a set of five vectors from
,
and let . To arrive at a (smaller) linearly independent set, follow the procedure described in Theorem BS. Place the vectors from into a matrix as columns, and row-reduce,
Columns 1 and 3 are the pivot columns () so the set
is linearly independent and . Boom!
Since the reduced row-echelon form of a matrix is unique (Theorem RREFU), the procedure of Theorem BS leads us to a unique set . However, there is a wide variety of possibilites for sets that are linearly independent and which can be employed in a span to create . Without proof, we list two other possibilities:
Can you prove that and are linearly independent sets and ?
Example RES
Reworking elements of a span
Begin with a set of five vectors from
,
It is easy to create elements of — we will create one at random,
We know we can replace by a smaller set (since it is obviously linearly dependent by Theorem MVSLD) that will create the same span. Here goes,
So, if we collect the first, second and fourth vectors from ,
then is linearly independent and by Theorem BS. Since we built as an element of it must also be an element of . Can we write as a linear combination of just the three vectors in ? The answer is, of course, yes. But let’s compute an explicit linear combination just for fun. By Theorem SLSLC we can get such a linear combination by solving a system of equations with the column vectors of as the columns of a coefficient matrix, and as the vector of constants. Employing an augmented matrix to solve this system,
So we see, as expected, that
A key feature of this example is that the linear combination that expresses as a linear combination of the vectors in is unique. This is a consequence of the linear independence of . The linearly independent set is smaller than , but still just (barely) big enough to create elements of the set . There are many, many ways to write as a linear combination of the five vectors in (the appropriate system of equations to verify this claim has two free variables in the description of the solution set), yet there is precisely one way to write as a linear combination of the three vectors in .
Write one vector from as a linear combination of the other two (you should be able to do this on sight, rather than doing some computations). Convert this expression into a relation of linear dependence on .
C20 Let be the set of columns of the matrix below. Define . Find a set so that (1) has 3 vectors, (2) is a subset of , and (3) .
Contributed by Robert Beezer Solution [455]
C40 Verify that the set
at the end of Example RSC5 is linearly independent.
Contributed by Robert Beezer
C50 Consider the set of vectors from , , given below. Find a linearly independent set that contains three vectors from and such that .
Contributed by Robert Beezer Solution [456]
C51 Given the set below, find a linearly independent set so that .
Contributed by Robert Beezer Solution [459]
C55 Let be the set
of vectors . Find two
different subsets of ,
named
and , so
that and
each contain three
vectors, and so that
and . Prove
that both
and
are linearly independent.
Contributed by Robert Beezer Solution [457]
C70 Reprise Example RES by creating a new version of the vector
. In
other words, form a new, different linear combination of the vectors in
to create
a new vector
(but do not simplify the problem too much by choosing any
of the five new scalars to be zero). Then express this new
as a combination
of the vectors in .
Contributed by Robert Beezer
M10 At the conclusion of Example RSSC4 two alternative solutions, sets
and
,
are proposed. Verify these claims by proving that
and
.
Contributed by Robert Beezer
T40 Suppose that and are any two vectors from . Prove the following set equality.
Contributed by Robert Beezer Solution [460]
C20 Contributed by Robert Beezer Statement [451]
Let . The
vector
is a solution to the homogeneous system with the matrix
as the coefficient matrix (check this!). By Theorem SLSLC it
provides the scalars for a linear combination of the columns of
(the vectors
in )
that equals the zero vector, a relation of linear dependence on
,
We can rearrange this equation by solving for ,
This equation tells us that the vector is superfluous in the span construction that creates . So . The requested set is .
C50 Contributed by Robert Beezer Statement [451]
To apply Theorem BS, we formulate a matrix
whose columns
are . Then we
row-reduce .
After row-reducing, we obtain
From this we that the pivot columns are . Thus
is a linearly independent set and . Compare this problem with Exercise LI.M50.
C55 Contributed by Robert Beezer Statement [452]
Let
be the matrix whose columns are the vectors in
. Then
row-reduce ,
From Theorem BS we can form by choosing the columns of that correspond to the pivot columns of . Theorem BS also guarantees that will be linearly independent.
That was easy. To find will require a bit more work. From we can obtain a solution to , which by Theorem SLSLC will provide a nontrivial relation of linear dependence on the columns of , which are the vectors in . To wit, choose the free variable to be 1, then , , , and so
this equation can be rewritten with the second vector staying put, and the other three moving to the other side of the equality,
We could have chosen other vectors to stay put, but may have then needed to divide by a nonzero scalar. This equation is enough to conclude that the second vector in is “surplus” and can be replaced (see the careful argument in Example RSC5). So set
and then . is also a linearly independent set, which we can show directly. Make a matrix whose columns are the vectors in . Row-reduce and you will obtain the identity matrix . By Theorem LIVRN, the set is linearly independent.
C51 Contributed by Robert Beezer Statement [452]
Theorem BS says we can make a matrix with these four vectors as
columns, row-reduce, and just keep the columns with indices in the set
. Here
we go, forming the relevant matrix and row-reducing,
Analyzing the row-reduced version of this matrix, we see that the firast two columns are pivot columns, so . Theorem BS says we need only “keep” the first two columns to create a set with the requisite properties,
T40 Contributed by Robert Beezer Statement [453]
This is an equality of sets, so Definition SE applies.
The “easy” half first. Show that .
Choose .
Then for
some scalars
and .
Then,
which qualifies for membership in , as it is a linear combination of .
Now show the opposite inclusion, .
Choose . Then
there are scalars
such that .
Rearranging, we obtain,
This is an expression for as a linear combination of and , earning membership in . Since is a subset of , and vice versa, we see that , as desired.