Next: 11.2 Counting Sort and Up: 11. Sorting Algorithms Previous: 11. Sorting Algorithms Contents Index
In this section, we present three sorting algorithms: merge-sort,
quicksort, and heap-sort. Each of these algorithms takes an input array
and sorts the elements of
into non-decreasing order in
(expected) time. These algorithms are all comparison-based.
These algorithms don't care what type
of data is being sorted; the only operation they do on the data is
comparisons using the
method. Recall, from Section 1.2.4,
that
returns a negative value if
, a positive
value if
, and zero if
.
The merge-sort algorithm is a classic example of recursive divide
and conquer:
If the length of
is at most 1, then
is already
sorted, so we do nothing. Otherwise, we split
into two halves,
![$ \ensuremath{\ensuremath{\ensuremath{\mathit{a_0}}}}=\ensuremath{\ensuremath{\e...
...,\ensuremath{\ensuremath{\ensuremath{\mathit{a}}[\ensuremath{\mathit{n}}/2-1]}}$](img4090.png){kind=small}
and
![$ \ensuremath{\ensuremath{\ensuremath{\mathit{a_1}}}}=\ensuremath{\ensuremath{\e...
...ts,\ensuremath{\ensuremath{\ensuremath{\mathit{a}}[\ensuremath{\mathit{n}}-1]}}$](img4091.png){kind=small}
.
We recursively sort
and
, and then we merge (the now sorted)
and
to get our fully sorted array
:
An example is shown in Figure 11.1.
Compared to sorting, merging the two sorted arrays
and
is
fairly easy. We add elements to
one at a time. If
or
is empty, then we add the next elements from the other (non-empty)
array. Otherwise, we take the minimum of the next element in
and
the next element in
and add it to
:
Notice that the
algorithm performs at most
comparisons before running out of elements in one of
or
.
To understand the running-time of merge-sort, it is easiest to think
of it in terms of its recursion tree. Suppose for now that
is a
power of two, so that
, and
is an integer.
Refer to Figure 11.2. Merge-sort turns the problem of
sorting
elements into two problems, each of sorting
elements.
These two subproblem are then turned into two problems each, for a total
of four subproblems, each of size
. These four subproblems become eight
subproblems, each of size
, and so on. At the bottom of this process,
subproblems, each of size two, are converted into
problems,
each of size one. For each subproblem of size
, the time
spent merging and copying data is
. Since there are 
subproblems of size
, the total time spent working on problems
of size 
, not counting recursive calls, is
The proof of the following theorem is based on preceding analysis,
but has to be a little more careful to deal with the cases where
is not a power of 2.
time and
performs at most
comparisons.
. The base case, in which
,
is trivial; when presented with an array of length 0 or 1 the algorithm
simply returns without performing any comparisons.
Merging two sorted lists of total length
requires at most
comparisons. Let
denote the maximum number of comparisons performed by
on an array
of length
. If
is even, then we apply the inductive hypothesis to
the two subproblems and obtain
 |
 |
|
 |
||
 |
||
 |
||
 |
is odd is slightly more complicated. For this case,
we use two inequalities that are easy to verify:
and
. Inequality (11.1) comes from the fact that
while (11.2) follows from the fact that 
is a concave function. With these tools in hand we have, for odd
,
 |
 |
|
 |
||
 |
||
 |
||
 |
||
 |
||
 |
||
 |
The quicksort algorithm is another classic divide and conquer algorithm. Unlike merge-sort, which does merging after solving the two subproblems, quicksort does all of its work upfront.
Quicksort is simple to describe: Pick a random pivot element,
, from
; partition
into the set of elements less than
, the
set of elements equal to
, and the set of elements greater than
;
and, finally, recursively sort the first and third sets in this partition.
An example is shown in Figure 11.3.
method only sorts the
subarray
![$ \ensuremath{\ensuremath{\ensuremath{\mathit{a}}[\ensuremath{\mathit{i}}]}},\ld...
...th{\ensuremath{\mathit{a}}[\ensuremath{\mathit{i}}+\ensuremath{\mathit{n}}-1]}}$](img4176.png){kind=small}
. Initially, this method is invoked
with the arguments
.
At the heart of the quicksort algorithm is the in-place partitioning
algorithm. This algorithm, without using any extra space, swaps elements
in
and computes indices
and
so that
loop in the code, works
by iteratively increasing
and decreasing
while maintaining the
first and last of these conditions. At each step, the element at position
is either moved to the front, left where it is, or moved to the back.
In the first two cases,
is incremented, while in the last case,
is not incremented since the new element at position
has not yet been
processed.
Quicksort is very closely related to the random binary search trees
studied in Section 7.1. In fact, if the input to quicksort consists
of
distinct elements, then the quicksort recursion tree is a random
binary search tree. To see this, recall that when constructing a random
binary search tree the first thing we do is pick a random element
and
make it the root of the tree. After this, every element will eventually
be compared to
, with smaller elements going into the left subtree
and larger elements into the right.
In quicksort, we select a random element
and immediately compare
everything to
, putting the smaller elements at the beginning of
the array and larger elements at the end of the array. Quicksort then
recursively sorts the beginning of the array and the end of the array,
while the random binary search tree recursively inserts smaller elements
in the left subtree of the root and larger elements in the right subtree
of the root.
The above correspondence between random binary search trees and quicksort means that we can translate Lemma 7.1 to a statement about quicksort:
, the expected number of times element
is compared
to a pivot element is at most
.A little summing up of harmonic numbers gives us the following theorem about the running time of quicksort:
distinct
elements, the expected number of comparisons performed is at most
.
be the number of comparisons performed by quicksort when sorting
distinct elements. Using Lemma 11.1 and linearity of
expectation, we have:
![$\displaystyle \mathrm{E}[T]$](img4202.png){kind=small} |
 |
|
 |
||
 |
||
 |
Theorem 11.3 describes the case where the elements being sorted are
all distinct. When the input array,
, contains duplicate elements,
the expected running time of quicksort is no worse, and can be even
better; any time a duplicate element
is chosen as a pivot, all
occurrences of
get grouped together and do not take part in either
of the two subproblems.
method runs in
expected
time and the expected number of comparisons it performs is at most
.
The heap-sort algorithm is another in-place sorting algorithm.
Heap-sort uses the binary heaps discussed in Section 10.1.
Recall that the BinaryHeap data structure represents a heap using
a single array. The heap-sort algorithm converts the input array
into a heap and then repeatedly extracts the minimum value.
More specifically, a heap stores
elements in an array,
, at array locations
![$ \ensuremath{\ensuremath{\ensuremath{\mathit{a}}[0]}},\ldots,\ensuremath{\ensuremath{\ensuremath{\mathit{a}}[\ensuremath{\mathit{n}}-1]}}$](img4216.png){kind=small}
with the smallest value stored at the root,
![$ \ensuremath{\ensuremath{\mathit{a}}[0]}$](img4217.png){kind=small}
. After transforming
into a BinaryHeap, the heap-sort
algorithm repeatedly swaps
![$ \ensuremath{\ensuremath{\mathit{a}}[0]}$](img4219.png){kind=small}
and
![$ \ensuremath{\ensuremath{\mathit{a}}[\ensuremath{\mathit{n}}-1]}$](img4220.png){kind=small}
, decrements
, and
calls
so that
![$ \ensuremath{\ensuremath{\ensuremath{\mathit{a}}[0]}},\ldots,\ensuremath{\ensuremath{\ensuremath{\mathit{a}}[\ensuremath{\mathit{n}}-2]}}$](img4223.png){kind=small}
once again are
a valid heap representation. When this process ends (because
)
the elements of
are stored in decreasing order, so
is reversed
to obtain the final sorted order.11.1Figure 11.4 shows an example of the execution of
.
![\includegraphics[scale=0.90909]{figs-python/heapsort}](img4229.png)
|
A key subroutine in heap sort is the constructor for turning
an unsorted array
into a heap. It would be easy to do this
in
time by repeatedly calling the BinaryHeap
method, but we can do better by using a bottom-up algorithm.
Recall that, in a binary heap, the children of
![$ \ensuremath{\ensuremath{\mathit{a}}[\ensuremath{\mathit{i}}]}$](img4237.png){kind=small}
are stored at
positions
![$ \ensuremath{\ensuremath{\mathit{a}}[2i+1]}$](img4238.png){kind=small}
and
![$ \ensuremath{\ensuremath{\mathit{a}}[2i+2]}$](img4239.png){kind=small}
. This implies that the elements
![$ \ensuremath{\ensuremath{\ensuremath{\mathit{a}}}}[\lfloor\ensuremath{\ensurema...
...ts,\ensuremath{\ensuremath{\ensuremath{\mathit{a}}[\ensuremath{\mathit{n}}-1]}}$](img4240.png){kind=small}
have no children. In other
words, each of
![$ \ensuremath{\ensuremath{\ensuremath{\mathit{a}}}}[\lfloor\ensuremath{\ensurema...
...ts,\ensuremath{\ensuremath{\ensuremath{\mathit{a}}[\ensuremath{\mathit{n}}-1]}}$](img4241.png){kind=small}
is a sub-heap
of size 1. Now, working backwards, we can call
for
each
. This works, because by
the time we call
, each of the two children of
![$ \ensuremath{\ensuremath{\mathit{a}}[\ensuremath{\mathit{i}}]}$](img4245.png){kind=small}
are the root of a sub-heap, so calling
makes
![$ \ensuremath{\ensuremath{\mathit{a}}[\ensuremath{\mathit{i}}]}$](img4247.png){kind=small}
into the root of its own subheap.
The interesting thing about this bottom-up strategy is that it is more
efficient than calling
times. To see this, notice that,
for
elements, we do no work at all, for
elements, we call
on a subheap rooted at
![$ \ensuremath{\ensuremath{\mathit{a}}[\ensuremath{\mathit{i}}]}$](img4253.png){kind=small}
and whose height is one, for
elements, we call
on a subheap whose height is two,
and so on. Since the work done by
is proportional to
the height of the sub-heap rooted at
![$ \ensuremath{\ensuremath{\mathit{a}}[\ensuremath{\mathit{i}}]}$](img4257.png){kind=small}
, this means that the total
work done is at most
is equal, by definition of expected value,
to the expected number of times we toss a coin up to and including the
first time the coin comes up as heads and applying Lemma 4.2.
The following theorem describes the performance of
.
method runs in
time and performs at
most
comparisons.
into a heap,
(2) repeatedly extracting the minimum element from
, and (3) reversing
the elements in
. We have just argued that step 1 takes
time and performs
comparisons. Step 3 takes
time and
performs no comparisons. Step 2 performs
calls to
.
The 
th such call operates on a heap of size
and performs
at most
comparisons. Summing this over 
gives
We have now seen three comparison-based sorting algorithms that each run
in
time. By now, we should be wondering if faster
algorithms exist. The short answer to this question is no. If the
only operations allowed on the elements of
are comparisons, then no
algorithm can avoid doing roughly
comparisons. This is
not difficult to prove, but requires a little imagination. Ultimately,
it follows from the fact that
We will start by focusing our attention on deterministic algorithms like
merge-sort and heap-sort and on a particular fixed value of
. Imagine
such an algorithm is being used to sort
distinct elements. The key
to proving the lower-bound is to observe that, for a deterministic
algorithm with a fixed value of
, the first pair of elements that are
compared is always the same. For example, in
, when
is even, the first call to
is with
and the
first comparison is between elements
![$ \ensuremath{\ensuremath{\mathit{a}}[\ensuremath{\mathit{n}}/2-1]}$](img4289.png){kind=small}
and
![$ \ensuremath{\ensuremath{\mathit{a}}[\ensuremath{\mathit{n}}-1]}$](img4290.png){kind=small}
.
Since all input elements are distinct, this first comparison has only
two possible outcomes. The second comparison done by the algorithm may
depend on the outcome of the first comparison. The third comparison
may depend on the results of the first two, and so on. In this way,
any deterministic comparison-based sorting algorithm can be viewed
as a rooted binary comparison tree.
Each internal node,
,
of this tree is labelled with a pair of indices
and
.
If
![$ \ensuremath{\ensuremath{\ensuremath{\mathit{a}}[\ensuremath{\mathit{u}}.\ensur...
...math{\ensuremath{\mathit{a}}[\ensuremath{\mathit{u}}.\ensuremath{\mathit{j}}]}}$](img4294.png){kind=small}
the algorithm proceeds to the left subtree,
otherwise it proceeds to the right subtree. Each leaf
of this
tree is labelled with a permutation
![$ \ensuremath{\ensuremath{\ensuremath{\mathit{w}}.\ensuremath{\mathit{p}}[0]}},\...
...th{\ensuremath{\mathit{w}}.\ensuremath{\mathit{p}}[\ensuremath{\mathit{n}}-1]}}$](img4296.png){kind=small}
of
. This permutation represents the one that is
required to sort
if the comparison tree reaches this leaf. That is,
![$\displaystyle \ensuremath{\ensuremath{\ensuremath{\mathit{a}}[\ensuremath{\math...
...h{\mathit{w}}.\ensuremath{\mathit{p}}[\ensuremath{\mathit{n}}-1]}]} \enspace .
$](img4299.png){kind=small}
is shown in
Figure 11.5.
The comparison tree for a sorting algorithm tells us everything about
the algorithm. It tells us exactly the sequence of comparisons that
will be performed for any input array,
, having
distinct elements
and it tells us how the algorithm will reorder
in order to sort it.
Consequently, the comparison tree must have at least
leaves;
if not, then there are two distinct permutations that lead to the same
leaf; therefore, the algorithm does not correctly sort at least one of
these permutations.
For example, the comparison tree in Figure 11.6 has only
leaves. Inspecting this tree, we see that the two input arrays
and 
both lead to the rightmost leaf. On the input 
this leaf correctly outputs
![$ \ensuremath{\ensuremath{\ensuremath{\mathit{a}}[1]}}=1,\ensuremath{\ensuremath...
...emath{\mathit{a}}[2]}}=2,\ensuremath{\ensuremath{\ensuremath{\mathit{a}}[0]}}=3$](img4312.png){kind=small}
. However, on the
input 
, this node incorrectly outputs
![$ \ensuremath{\ensuremath{\ensuremath{\mathit{a}}[1]}}=2,\ensuremath{\ensuremath...
...emath{\mathit{a}}[2]}}=1,\ensuremath{\ensuremath{\ensuremath{\mathit{a}}[0]}}=3$](img4314.png){kind=small}
.
This discussion leads to the primary lower-bound for comparison-based
algorithms.
and any integer
, there exists an input array
of
length
such that
performs at least
comparisons when sorting
.
must have at least
leaves. An easy inductive proof shows that
any binary tree with 
leaves has a height of at least 
.
Therefore, the comparison tree for
has a leaf,
,
with a depth of at least
and there is an input array
that leads to this leaf. The input array
is an input for which
does at least
comparisons.
Theorem 11.5 deals with deterministic
algorithms like merge-sort and heap-sort, but doesn't tell us anything
about randomized algorithms like quicksort. Could a randomized algorithm
beat the
lower bound on the number of comparisons?
The answer, again, is no. Again, the way to prove it is to think
differently about what a randomized algorithm is.
In the following discussion, we will assume that our decision
trees have been ``cleaned up'' in the following way: Any node that can not
be reached by some input array
is removed. This cleaning up implies
that the tree has exactly
leaves. It has at least
leaves
because, otherwise, it could not sort correctly. It has at most
leaves since each of the possible
permutation of
distinct
elements follows exactly one root to leaf path in the decision tree.
We can think of a randomized sorting algorithm,
, as a
deterministic algorithm that takes two inputs: The input array
that should be sorted and a long sequence
of random real numbers in the range ![$ [0,1]$](img4345.png){kind=small}
. The random numbers provide
the randomization for the algorithm. When the algorithm wants to toss a
coin or make a random choice, it does so by using some element from 
.
For example, to compute the index of the first pivot in quicksort,
the algorithm could use the formula
.
Now, notice that if we fix 
to some particular sequence 
then
becomes a deterministic sorting algorithm,
, that has an associated comparison tree,
. Next, notice that if we select
to be a random
permutation of
, then this is equivalent to selecting
a random leaf,
, from the
leaves of
.
Exercise 11.12 asks you to prove that, if we select
a random leaf from any binary tree with 
leaves, then the expected
depth of that leaf is at least 
. Therefore, the expected
number of comparisons performed by the (deterministic) algorithm
when given an input array containing a random
permutation of
is at least
. Finally,
notice that this is true for every choice of 
, therefore it holds even for
. This completes the proof of the lower-bound for randomized algorithms.
and any (deterministic or randomized)
comparison-based sorting algorithm
, the expected number
of comparisons done by
when sorting a random permutation
of
is at least
.
function so that the
heap sort algorithm stores the elements directly in ascending order.
![\includegraphics[width=\textwidth ]{figs-python/mergesort}](img4098.png)
![\includegraphics[width=\textwidth ]{figs-python/mergesort-recursion}](img4129.png)
![\includegraphics[scale=0.90909]{figs-python/quicksort}](img4174.png)
![$ \ensuremath{\ensuremath{\ensuremath{\mathit{a}}[0]}},\ensuremath{\ensuremath{\ensuremath{\mathit{a}}[1]}},\ensuremath{\ensuremath{\ensuremath{\mathit{a}}[2]}}$](img4302.png){kind=small}
![\includegraphics[width=\textwidth ]{figs-python/comparison-tree}](img4301.png)
![\includegraphics[width=\textwidth ]{figs-python/comparison-tree-b}](img4315.png)