Chapter 1 Introduction
3.3 Theoretical Comparison
3.3.2 Parallelization
The parallelization considered here is the simplest: for every method, we determine the minimum number, M, of machines that can fit the storage required in their main memory, assuming machines with 50 GB of memory. Then we split the images evenly across these machines and each will take a copy of the probe image and search its own share of images.
Finally, all the machines will communicate their ranked list of images (of length L) and produce a final list of candidate matches that is further geometrically checked. We call this simple scheme Independent Kd-Trees (IKdt), since the Kd-Trees on the different machines are built independently.
31
Figure 3.2: Kd-Tree Parallelizations. (Left)Independent Kd-Trees (IKdt). The images are divided onto the machines, which build independent Kd-Trees for the images they contain.
At query time, all the machines are searched in parallel. (Right) Distributed Kd-Trees (DKdt). The root machine stores the top of the tree, while the leaf machines store the leaves of the tree. At query time, the root machine directs features to a small subset of the leaf machines, which leads to higher throughput. See Section 3.3.2
100 101 102 103 104 105
101 102 103 104
Number of Machines
Time per image (msec)
kd−tree (IKdt) kd−tree (DKdt) lsh−l2
inv−file
Figure 3.3:Parallelization Run Time. Time per image vs. min. no. of machines required M for different dataset sizes. Each point represents a dataset size from 106 to 109 images, going left to right. IKdt is the Independent Kd-Tree, while DKdt is the Distributed Kd-Tree parallelization. See Section 3.3.2.
More sophisticated parallelization techniques are possible, that can take advantage of the properties of the method at hand. For example, in the case of Kd-trees, one such ad- vanced approach is to store the top part of the Kd-tree on one machine, and divide the bottom subtrees across other machines (see Figure 3.2). We call this approach Distributed Kd-Trees (DKdt), since different parts of the same Kd-Tree are distributed among different machines. For 1012 features (109 images), we have 40 levels in the Kd-tree, and so we can store up to 30 levels in onerootmachine, and split the bottom 10 levels evenly across theleaf machines (see Section 3.4). Given a probe image, we first query the root machine and get the list of branches (and hence subtrees) that we need to search with the backtrack- ing. Then the appropriate leaf machines will process these query features and update their ranked list of images.
The motivations for this distinction between root and leaf machines are:
• Most of the storage for the tree is in the leaves. Therefore we can store most of the top of the tree in a single machine, which will then dispatch jobs to the leaf machines.
• The time spent searching the top of the tree is smaller than that spent searching the lower parts of the tree, so we can gain more speed up by splitting the bottom part of the tree instead of splitting the whole tree.
• Searching the tree will generally not activate all of the leaf machines, i.e., the list of explored nodes will not span all the node machines. Therefore, the root machine can process more features, and the leaf machines can interleave computations of differ- ent features simultaneously. This offers more speed up over the first parallelization approach above. So, even though the search time per feature is similar to that in the
33
first case, interleaving computations will make this much smaller.
This approach has the advantage of significant speed up and better utilization of the ma- chines, since not all the machines will be working on the same query feature at the same time, rather they will be processing different features from the probe image concurrently (see figure 3.3). However, a drawback with this approach is the extra storage requirements, because the lower parts of the trees will generally not store the same features for different trees, and therefore each machine will have to store the features in the subtrees it is holding, and as an upper bound the extra storage will be proportional to the number of Kd-Trees, i.e., we will need to store each featureKtimes, whereKis the number of trees. We provide more details on the implementations of Independent and Distributed Kd-Trees in Chapter 7.
Figure 3.2 shows a schematic of the Distributed Kd-Trees compared to the Independent Kd-Trees, while Figure 3.3 shows the time per image vs. the minimum number of machines Mfor different dataset sizes from 106to 109. We note the following:
• Distributed Kd-Trees provide significant speed ups starting at 108images. It might seem paradoxical that increasing the dataset size decreases the run time, however it makes sense when we notice that adding more machines allows us to interleave processing of more features concurrently, and thus allows faster processing. This however comes at a cost of more storage used (see Figure 3.3).
• We also note that many of the FR methods have parameters that affect the trade off between run time and accuracy, e.g., number of hash functions or bin size for LSH. These parameters have to be tuned for every database size under consideration,
and we should not use the same settings when enlarging the database. This poses a problem for these methods, which have to be re-tuned as the database size changes, as opposed to Kd-Trees which adapt to the current database size and need very little tuning.