Modern Algorithms of Cluster Analysis

(1)

Studies in Big Data 34

Sławomir T. Wierzchoń Mieczysław A. Kłopotek

Modern

Algorithms of Cluster

Analysis

(2)

Studies in Big Data

Volume 34

Series editor

Janusz Kacprzyk, Polish Academy of Sciences, Warsaw, Poland e-mail: [email protected]

(3)

The series“Studies in Big Data”(SBD) publishes new developments and advances in the various areas of Big Data- quickly and with a high quality. The intent is to cover the theory, research, development, and applications of Big Data, as embedded in theﬁelds of engineering, computer science, physics, economics and life sciences.

The books of the series refer to the analysis and understanding of large, complex, and/or distributed data sets generated from recent digital sources coming from sensors or other physical instruments as well as simulations, crowd sourcing, social networks or other internet transactions, such as emails or video click streams and other. The series contains monographs, lecture notes and edited volumes in Big Data spanning the areas of computational intelligence incl. neural networks, evolutionary computation, soft computing, fuzzy systems, as well as artiﬁcial intelligence, data mining, modern statistics and Operations research, as well as self-organizing systems. Of particular value to both the contributors and the readership are the short publication timeframe and the world-wide distribution, which enable both wide and rapid dissemination of research output.

More information about this series at http://www.springer.com/series/11970

(4)

S ł awomir T. Wierzcho ń

^•

Mieczys ł aw A. K ł opotek

Modern Algorithms of Cluster Analysis

123

(5)

Sławomir T. Wierzchoń Institute of Computer Science Polish Academy of Sciences Warsaw

Poland

Mieczysław A. Kłopotek Institute of Computer Science Polish Academy of Sciences Warsaw

Poland

ISSN 2197-6503 ISSN 2197-6511 (electronic)

Studies in Big Data

ISBN 978-3-319-69307-1 ISBN 978-3-319-69308-8 (eBook)

https://doi.org/10.1007/978-3-319-69308-8

Library of Congress Control Number: 2017955662

The herein contained material cannot be reproduced nor distributed, in its entirety as well as with respect to the fragments thereof, by means of electronic, mechanical, copying, registering and other devices, without the written consent from the Authors.

©Springer International Publishing AG 2018

This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the material is concerned, speciﬁcally the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microﬁlms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed.

The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a speciﬁc statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use.

The publisher, the authors and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the editors give a warranty, express or implied, with respect to the material contained herein or for any errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional claims in published maps and institutional afﬁliations.

Printed on acid-free paper

This Springer imprint is published by Springer Nature The registered company is Springer International Publishing AG

The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland

(6)

Symbols

X Set of objects (observations) x An element of the setX

X2R^mn Matrix, representing the set of objects,X¼ ðx1 xmÞ^T m Cardinality of the setX

n Dimensionality (number of coordinates) of the vector of features of x2X

C Set of clusters

k Number of clusters

diagðvÞ Diagonal matrix having the main diagonal equal tov

G¼ ðV;EÞ Graph spanned over the set of verticesV linked with the edges indicated in the setE

NðvÞ Set of neighbours of the vertexvin graphG

NkðvÞ Set ofknearest neighbours of the vertexvin graphG

@ðCÞ Edge separator in graphG:@ðCÞ ¼ ffu;vg 2E:u2C;v62Cg A Neighbourhood matrix having elementsaij2 f0;1g,A¼ ½aij_mm S Similarity matrix having elementssij2 ½0;1,S¼ ½sij_m_m m Number of edges in graphG

di Degree (weight) of thei-th object, di¼Pm j¼1sij

D Matrix of degrees,D¼diagð½d1;. . .;dmÞ L Combinatorial Laplacian,L¼DS L Normalised Laplacian,L¼D¹⁼²LD¹⁼² L A variant of the Laplacian,L¼IP

P Column-stochastic transition matrix, describing walk on graphG, P¼SD¹

eP Row-stochastic transition matrix, describing walk on graphG, Pe¼D¹S

bP Column-stochastic transition matrix in a lazy walk on graphG, Pb¼¹₂ðIþSD¹Þ

p Stationary distribution of the Markov chain having transition matrix P, that is, a vector such thatp¼Pp

ix

(11)

qðs;aÞ Global PageRank with positive preference vectorsand the coefﬁcient of teleportationa:qðs;aÞ ¼asþ ð1aÞPqðs;aÞ pðs;aÞ Personalised PageRank vector with nonnegative preference vectors

and the coefﬁcient of teleportationa e Column vector having elements equal 1

I Unit matrix

x Symbols

(12)

List of Figures

Fig. 2.1 The task of cluster analysis: to break down the set of observations intokdisjoint subsets, composed of similar

elements. . . 15 Fig. 2.2 Grouping of the set of randomly generated objects.a100 points

randomly selected from the set½0;1 ½0;1.bDivision of the objects into three groups using thek-means algorithm. Bigger

marks indicate the geometrical centers of groups. . . 15 Fig. 2.3 Unit circles for the selected values of the parameterp:

(a)p¼1;2;1, (b)p¼2:5 (Hein’s super-ellipse). . . 20 Fig. 2.4 The influence of the parameterp on the properties of

Minkowski distance.aAverage values of the difference between the most distant points from a 100-point set in dependence upon the number of dimensions,n, and the value

of the exponent,p.b Unit circles forp\1 . . . 21 Fig. 2.5 The exemplary data seta and the corresponding dendrogram,

bobtained from the complete link method. . . 32 Fig. 2.6 Voronoi tessellation, that is, the boundaries of clusters

determined by the gravity centres marked by dark dots. . . 45 Fig. 2.7 Identiﬁcation of clusters by examining mutual similarity

between objects:aa set of points. In illustrations like this we follow the intuitive convention: The points are meant to lie in a plane, the horizontal axis reflects theﬁrst coordinate and the vertical one the second coordinate,ba graph obtained by

joiningﬁve nearest neighbors of each node. . . 49 Fig. 2.8 An example of clusters with irregular shapes and various

densities. The data set contains alsooutliers . . . 51 Fig. 2.9 Diagram ofk-distances between the points of an exemplary data

setX:aAn exemplary data set composed of 90 points, bincreasingly ordered values of the average distance between each of the 90 points of the data setXand theirk¼4 nearest

neighbours. . . 53

xi

(13)

Fig. 2.10 aThe set2spiralsis composed of two spirals, situated one inside the other.bGrouping produced by the k-means

algorithm. . . 60 Fig. 2.11 Unordered (a) and ordered (b) concordance matrices, generated

with the use of the method, described in the text. Experiments were carried out fork¼2;3;4;5 groups. . . 63 Fig. 2.12 Values of the cophenetic correlation coefﬁcient for the

partitions obtained. The horizontal axis represents the number of groups, the vertical axis shows the cophenetic correlation

coefﬁcient. . . 63 Fig. 2.13 Exemplary data having:aspherical andbnon-spherical normal

distribution. . . 64 Fig. 3.1 Local optima in the problem of partitioning of the (one

dimensional) setX, consisting of 10 points. Figure to the left:

values of the indicatorJ(vertical axis) for the gravity centre of one cluster placed at locations from the interval½3;9. Figure to the right: a detailed look at these values when the centre

belongs to the set½5:4686;5:7651. . . 70 Fig. 3.2 The influence of initialisation and scaling on the results of

clustering of a set of 450 observations. Initial location of cluster centres is marked with red squares, and respective symbols denote the assignment of objects to clusters. In case

axi1 2 f1;. . .;450g, and in casebxi1 22:22218510³þ

2:03470510¹⁰ f1;. . .;450g,i¼1;. . .;450. . . 74 Fig. 3.3 The distribution of the gravity centres (denoted with squares) in

two methods of initialisation.ðiÞ—initialisation described in point (c),ðiiÞ—initialisation described in point (d). In this last case the coordinates were initially normalised in such a way that xij2 ½1;1,i¼1;. . .;m,j¼1;. . .;n, and afterwards—as required by the method—each row of the matrix Xwas so normalized thatkxik ¼1. The lines connecting the coordinate system originð0;0Þwith gravity centres play here an

explanatory role only—they allow to imagine the angles

between the centres . . . 76 Fig. 3.4 Contour diagrams of the function:a minða1;a2Þ, and

bmhða1;a2Þ. It was assumed in both cases that

a1;a22 ð0;400. Horizontal axis:a1 value, vertical axis:a2

value, the contour lines were drawn for equidistant values . . . 87 Fig. 3.5 Results of application of the kernel-based variant of the

k-means algorithm. Object-to-cluster allocation is indicated by

different colours.r¼1 was assumed. . . 91 Fig. 3.6 Clustering of separable data sets with thek-medoids algorithm:

athe setdata3_2,bthe setdata6_2. Cluster examples are

marked with red dots. . . 93

xii List of Figures

(14)

Fig. 3.7 Clustering of data with the use of the FCM algorithm:aSet data3_2(blue points) with added noise (black points).

bAssignments to groups generated by the FCM algorithm. The bigger squares denote typical objects, smaller circles—the other objects; the colour indicates the class membership. Same convention applies to Figure (d) representing the results obtained for the setdata5_2from Figure (c).a¼2 was assumed in computations. Objectxiis treated as typical if

maxjuij0:9 . . . 102 Fig. 3.8 Influence of the value of fuzziness exponentaon the number of

iterations (picture to the left) and the quality of prototypes (picture to the right), expressed in terms of the average distance davg according to the Eq. (3.62). Solid line indicates the mean valuemof the respective quantity (averaged over 100 runs), and the dotted lines mark the valuesms, withsbeing the standard deviation.¼10⁸ was assumed. Experiments were

performed for the data setiris.txt. . . 105 Fig. 3.9 Influence of theaexponent (abscissa axis) on the reconstruction

error (ordinate axis). Test data:ﬁleiris.txt. . . 108 Fig. 3.10 Test setsdata_6_2(Figure to the left) anddata_4_2

(Figure to the right). . . 109 Fig. 3.11 Dependence of the values of quality criteria for the sets

data_6_2(to the left) anddata_4_2(to the right) on the

assumed number of classes . . . 109 Fig. 3.12 Diagrams of the function (3.75) for three values of the

parameterp. Function is of the formfðxÞ ¼0:6âj4xj^p þ0:5âj1xj^pþ0:8âj3xj^pþ0:7^mj6xj^p, wherex

meansljl, anda¼2:5. . . 113 Fig. 3.13 Comparison of partitions generated by theaFCM algorithm

andbGK algorithm for a synthetic data set consisting of three clusters with different shapes. . . 116 Fig. 4.1 Relation between an average cluster radius (leftﬁgure) and the

total distance of objects from the prototypes depending on the

number of clusters (rightﬁgure).. . . 167 Fig. 5.1 Compact and well separated clusters (a) vs. clusters

characterized by a similarity relationship (b). . . 182 Fig. 5.2 A simple graph . . . 189 Fig. 5.3 Graph from Fig. 5.2 with (arbitrarily) oriented edges. . . 192 Fig. 5.4 Eigenvectors of the combinatorial Laplacian of the path graph

consisting of 22 nodes. . . 199 Fig. 5.5 The number of zero crossings of:a the combinatorial and

bnormalized Laplacian eigenvectors for the jazz musician

network. . . 200

(15)

Fig. 5.6 An application of the Fiedler vector to visualization of the structure of a dataset:a an exemplary similarity matrix,bthe same matrix after its rows and columns have been reordered according to the order obtained by sorting the entries of the

Fiedler vector. . . 212 Fig. 5.7 First 20 eigenvalues of the Laplacian characterizing: linearly

separable clusters (left panel), and moon-shaped clusters that are not linearly separable (right panel). In theﬁrst case Gaussian kernel has been used withr¼0:63, and in the second case the

K-NN grap has been constructed withK¼10 and r¼1. . . 213 Fig. 5.8 Exemplary graph embedding:aoriginal graph andb its

embedding into spectral the space spanned by the eigenvectors ðw2;w3Þ. . . 222 Fig. 5.9 The NJW algorithm in action. Left column: data in spectral

coordinates determined by the eigenvectorsUk. Right column:

the same data in the coordinates described by the matrixY.

Panel (a) corresponds to the data consisting of three linearly

separable groups, and panel (c) representsiris data set . . . 222 Fig. 5.10 Examples of data with clear structure, (a) and (c), and similarity

matrices representing these data, (b) and (d). . . 223 Fig. 5.11 An example of unbalanced clusters:a Original data with three

clusters of size 2000 (red), 1000 (green), and 100 (blue).

bClustering obtained by thek-means algorithm, and

cclustering returned by spectral clustering algorithm. . . 224 Fig. 5.12 Eigenvalues of the 6060 grid graph. . . 239 Fig. 5.13 Sign pattern of the eigenvectorsw2,w3,w4 of the 4-by-4

grid graph. Circle (square) denotes positive (negative) sign of the eigenvector at particular node. Note that

herek2=k3= 0.58579. . . 240 Fig. 5.14 Partition of the set of nodes ofkarate network into two

groups. Black square denotes the starting node used to compute the personalized PageRank. Grey squares denote natural

neighbors of the black square. . . 246 Fig. 5.15 The values of the ordering functionq(left panel), and the values

of conductance (right panel) computed for thekarate

network. . . 247 Fig. 6.1 Result of 10-fold application of the NJW algorithm for a

neighbourhood matrix describing relations between jazz musician (Jazznetwork). The maximal value ofQ¼0:438591

is obtained each time when splitting intok¼4 groups. . . 284 Fig. 6.2 Distribution of the values of the modularityQas a function

of the number of clusters for the setsC.elegans(to the left) and

e-mail(to the right) . . . 285

xiv List of Figures

(16)

Fig. 6.3 Fiedler vector values for a graph representing friendships

among members of a karate club [525] . . . 286 Fig. 7.1 Test data sets from the Web sitehttp://www.isical.ac.in/

*sanghami/data.html:a data3_2,b data5_2,

c data6_2 . . . 316 Fig. 7.2 Test data sets:a 2rings,b 3rings,c 2spirals . . . 317 Fig. D.1 A mixture of 5 normal distributions as clustered byk-means

algorithm (Voronoi diagram superimposed). . . 360 Fig. D.2 Data from Fig. D.1 after Kleinberg’sC-transformation clustered

byk-means algorithm into two groups. . . 360 Fig. D.3 Data from Fig. D.1 after a correctedC-transformation, clustered

byk-means algorithm into 5 groups. . . 365 Fig. D.4 Impact of contraction towards cluster centre by a

factor lambda—local optimum maintained. . . 366

(17)

List of Tables

Table 2.1 Bregman divergences generated by various convex functions

[50]. . . 23 Table 3.1 Table to the left: Parameters of Gaussian distributions used to

generate samples belonging to the three groups. Table to the right: Quality indicators of the partitions generated by the

algorithms FCM and GK (Gustafson-Kessel) . . . 116 Table D.1 Variance explained (in percent) when applyingk-means

algorithm withk¼2;. . .;6 to data from Figs. D.1 (Original),

D.2 (Kleinberg) and D.3 (Centric). . . 360 Table D.2 Data points to be clustered using a ridiculous clustering

quality function. . . 362 Table D.3 Partition of the best quality (the lower the value the better)

after includingnﬁrst points from Table D.2. . . 363

xvii

(18)

List of Algorithms

Algorithm 2.1 Algorithm of agglomerative cluster analysis. . . 29

Algorithm 2.2 “Robust”Single Link. . . 34

Algorithm 2.3 Iterative algorithm of cluster analysis (generalised Lloyd’s heuristics). . . 44

Algorithm 2.4 DBSCAN algorithm. . . 52

Algorithm 2.5 Anchor grouping algorithm, [361]. . . 54

Algorithm 3.1 Incremental clustering algorithm BIMSEC. . . 73

Algorithm 3.2 k-means++ algorithm, [32]. . . 77

Algorithm 3.3 On line version of thek-means algorithm. . . 81

Algorithm 3.4 Bisectional k-means algorithm . . . 83

Algorithm 3.5 Sphericalk-means algorithm. . . 85

Algorithm 3.6 Kernel-basedk-means algorithm. . . 90

Algorithm 3.7 k-medoids algorithm. . . 92

Algorithm 3.8 k-modes algorithm, [263]. . . 95

Algorithm 3.9 EM algorithm for cluster analysis. . . 99

Algorithm 3.10 Afﬁnity propagation, [191]. . . 130

Algorithm 3.11 k-means# algorithm, [11]. . . 144

Algorithm 3.12 Streaming divide-and-conquer clustering, version after [11] . . . 144

Algorithm 5.1 Repeated bisection spectral clustering [427]. . . 218

Algorithm 5.2 Ak-way partitional spectral clustering. . . 218

Algorithm 5.3 Pivoted QR decomposition used to extract clusters from the matrixY[529] . . . 219

Algorithm 5.4 Spectral clustering proposed by Ng et al. in [374]. . . 220

Algorithm 5.5 Spectral clustering algorithmMNCut [352] . . . 226

Algorithm 5.6 k-way kernel spectral algorithm [17] . . . 238

Algorithm 5.7 Nibble—algorithm for local identiﬁcation of clusters [440]. . . 243

Algorithm 5.8 Landmark-based Spectral Clustering [111]. . . 252

Algorithm 5.9 Randomized SVD [226]. . . 254

xix

(19)

Algorithm 5.10 Approximating eigenvectors of a normalized similarity

matrix via incomplete Cholesky decomposition [316] . . . . 255 Algorithm 5.11 Compressive spectral clustering [465] . . . 259 Algorithm 6.1 Newman [366] algorithm of community detection. . . 278 Algorithm 6.2 Louvain [77] algorithm of community detection . . . 279 Algorithm 6.3 Bisectional spectral algorithm of community

detection [370]. . . 289 Algorithm B.1 Power method returning the principal eigenpair

of the matrixA. . . 330 Algorithm B.2 Applying power method to determine

theﬁrstp meigenvectors of the Laplacian. . . 332 Algorithm C.1 Algorithm of determination of the approximation

to the personalised PageRank vector on the basis

of Eq. (C.5) . . . 343 Algorithm C.2 Fast algorithm of determining an-approximation of the

personalised PageRank vector [118]. . . 344

xx List of Algorithms

(20)

Chapter 1 Introduction

Abstract This chapter characterises the scope of this book. It explains the reasons why one should be interested in cluster analysis, lists major application areas, basic theoretical and practical problems, and highlights open research issues and challenges faced by people applying clustering methods.

The methods of data analysis can be roughly classified into two categories¹: (a) the descriptive (exploratory) ones, which are recommended when, having no initial models or hypotheses, we try to understand the general nature and structure of the high-dimensional data, and (b) the confirmatory (inferential) methods, applied in order to confirm the correctness of the model or of the working hypotheses, concerning the data collected. In this context, a particular role is played by various statistical methods, such as, for instance, analysis of variance, linear regression, discriminant analysis, multidimensional scaling, factor analysis, or, finally, the subject of our considerations here—cluster analysis²[237,303].

It is really difficult to list all the domains of theoretical and practical applications of cluster analysis. Jain [272] mentions three basic domains: image segmentation [273], information retrieval [347], and bio-informatics [46].

The task of cluster analysis or grouping is to divide the set of objects into homogeneous groups: two arbitrary objects belonging to the same group are more similar to each other than two arbitrary objects belonging to different groups. If we wish to apply this recipe in practice, we must find the answers to two basic questions: (a) how to define the similarity between the objects, and (b) in what manner should one make use of the thus defined similarity in the process of grouping. The fact that the answers to these questions are provided independently one from another results in the multiplicity of algorithms.

A number of natural processes can give rise to clusters in the data [195]. One may have collected data from multiple experiments on same or intersecting sets of objects, or one obtained longitudinal data (same objects measured at different time

1See, e.g., J. W. Tukey,Exploratory Data Analysis. Addison-Wesley, 1977.

2This area is also referred to as Q-analysis, typology, grouping, partitioning, clumping, and taxonomy, [273], depending upon the domain of application.

S. T. Wierzcho´n and M. A. Kłopotek,Modern Algorithms of Cluster Analysis, Studies in Big Data 34, https://doi.org/10.1007/978-3-319-69308-8_1

1

(21)

2 1 Introduction

points), data may have been collected at different experimental sites, data may stem from subjects (human or animal) related to one another like in families etc. Season- ality, climate, geography, measurement equipment manufacturer are further possible hidden variables, inducing cluster formation among objects of the study. If one is unaware of such subtleties of the data collection process (no explicit information is available) but one is suspicious of such a situation, then one is advised to apply a clustering method prior to further analysis because the existence of clusters may impact the results of further analyses as an intra-cluster correlation may exist so that the frequent independence assumptions are violated. For example, if one intends to perform regression analysis on the data, and there exist clusters, then one is advised to add a factor for the cluster effect into the standard regression model (e.g. ANOVA analysis). A number of statistical tests, likeχ²,t-test, Wilcoxon test have also been adjusted for clusters. In image analysis, clustering may be performed prior to applying more advanced methods, like object recognition.

Division of data into clusters may also by itself be a subject of interest.

An interesting example of application of cluster analysis in modern information retrieval is constituted by the CLUSTY search engine (clusty.com). The methods of cluster analysis play an important role in the development of the recommender systems [258,416], in various kinds of economic, medical etc. analyses. Application of these methods results from at least three reasons:

(a) To gain insight into the nature of the data, and, first of all, to indicate the typical and the atypical (outlying) data items, to uncover the potential anomalies, to find hidden features, or, finally—to be able to formulate and verify the hypotheses concerning the relations between the observations;

(b) To obtain a compact description of data, to select the most representative objects;

here, the classical application is image compression;

(c) To obtain a natural classification of data, for instance—by determining the similarities between pairs of objects to establish the respective hierarchical structures.

Methods of cluster analysis are being applied there, where we wish to understand the nature of the phenomenon, represented by the set of observations, to get a sort of summary of the content of large data sets, and to process such sets effectively.

Jain and Dubes [273] list the following challenges, linked with the practical use of clustering:

(a) What is a cluster (group, module)?

(b) What features ought to be used to analyse the data collected?

(c) Should the data be normalised?

(d) Are there outliers in the analysed dataset, and if so—how should they be treated?

(e) How to define the similarity between the pairs of objects?

(f) How many clusters are really there in the data set? Do we deal, at all, with clusters?

(g) What method should be used in a given situation?

(h) Is the obtained partition of data justified?

(22)

Some of these questions find at least partial answers in a multiplicity of books, devoted to various aspects of cluster analysis, like [12,19,70,164,273], or [173].

The recent years, though, brought a number of new challenges. First, cluster analysis is being applied nowadays to processing of huge sets of data, which makes it necessary to develop specialised algorithms. Second, the necessity of analysing the data sets having complex topologies (data being situated in certain submanifolds) leads to the requirement of applying more refined tools. We mean here the spectral methods, the methods making use of kernel functions, and the relations between these two groups of methods.

The starting point for these methods is constituted by the matrix, describing the strength of interrelations between the objects making up the data set. In the case of the kernel-based methods, considerations are being transferred to the highly dimensional and nonlinear space of features for the purpose of strengthening of the separability of data. Due to application of the so-calledkernel trick to determine the distance between objects in this new space, the knowledge of a certain kernel function suffices.

The elements of the matrix, mentioned before, are the values of the kernel function, calculated for the respective pairs of objects. On the other hand, in the case of spectral methods, the elements of the matrix correspond to the values of similarity of the pairs of objects. Eigenvalues and eigenvectors of this matrix, or a matrix, being its certain transformation (usually a form of the Laplacian of the graph, corresponding to the similarity matrix), provide important information on relations between objects.

An interesting example is constituted by the problem of ordering of documents, collected by the crawlers, cooperating with a search engine. We deal in such cases with enormous collections of objects (of the order of 10⁹), with the issue of their effective representation, and, finally, with the task of indicating the important documents. The common sense postulate that the important websites (documents) are the ones that are referred to (linked to) by other important websites allows for the reformulating of the problem of assigning ranks in the ordering into the problem of determining the dominating eigenvector of the stochastic matrix P, being a slight modification of the original matrix A, representing connections between the websites, that isai j =1 when on theith website there is a link to the jth one. Vectorr, which satisfies the matrix equationr= P^Tr, is called the PageRank vector [379].

Both this vector and its diverse variants (TotalRank, HillTop, TrustRank, GeneRank, IsoRank, etc.) are the examples of the so-called spectral ranking, which find application not only in the ordering of documents (or, more generally, in various aspects of bibliometrics), but also in graph theory (and, consequently, also in the analysis of social networks), in bio-informatics, and so on.

A development over the latter idea is constituted by the methods based on random walk on graphs. Namely, the task of grouping is here considered as an attempt of finding such a division of the graph that a randomly moving wanderer shall remain for a long time in a single cluster (subgraph), only rarely jumping between the clusters.

An interesting work on this subject is presented in, e.g., [211], while a number of theoretical results have been shown by Chung [23,115,116], and by Meila [352].

The transformation of the matrices, representing relations (e.g. similarity) between the elements of the set of objects, into the stochastic Markovian matrices leads to

(23)

4 1 Introduction

yet another interesting formalism. As we treat the eigenvectors of these matrices as forming a new system of coordinates, we transform the multidimensional data into a cloud of points in a space with a much lower number of dimensions. This new representation reflects the essential properties of the original multidimensional structure. In this manner we enter the domain of the so-called diffusion maps [125], which have two significant features, distinguishing them from the classical methods:

they are nonlinear, and, besides this, they do faithfully map the local structure of data.

From the point of view of, for instance, processing of text documents, an essential advantage of such an approach is a relatively straightforward adaptation to the semi- supervised clustering problems and thecoclusteringproblems—the important, quite recently developing, directions of machine learning.

These most up-to-date trends in grouping have not been, until now, systematically presented nor considered in an integrated manner against the background of the classical methods of cluster analysis. Such a survey would be very welcome indeed from the point of view of students with an interest in the techniques of knowledge extrac- tion from data and computer-based modelling, in the context of their application in pattern recognition, signal analysis, pattern and interdependence identification, and establishment of other interesting characteristics in the large and very large collections of data, especially those put together automatically in real time.

That is why we prepared this monograph, which:

(a) Contains the survey of the basic algorithms of cluster analysis, along with pre- sentation of their diverse modifications;

(b) Makes apparent the issue of evaluation and proper selection of the number of clusters;

(c) Considers the specialisation of selected algorithms regarding the processing of enormous data sets;

(d) Presents in a unified form a homogeneous material, constituting the basis for developing new algorithms of spectral and diffusion data analysis;

(e) Comments upon the selected solutions for the semi-supervised learning;

(f) Provides a rich bibliography, which might also be a starting point to various individual research undertakings.

This book has been divided into 7 chapters. Chapter2outlines the major steps of cluster analysis. Chapters3and5are devoted to two distinct brands of clustering algorithms. While Chap.3focusses on clustering of objects that are embedded in some metric space, Chap.5concentrates on ones abandoning the embedding space and concentrating on the similarities between objects alone. The former are referred to as combinatorial algorithms, the latter are called spectral algorithms. Chapter4 addresses the issues of parameter tuning for clustering algorithms and the assessment of the resultant partitions. Chapter6discusses a specific, yet quite important application area of cluster analysis that is of community detection in large empirical graphs.

Finally, in Chap.7we present the data sets used throughout this book in illustrative examples.

From the point of view of knowledge discovery task the clustering can be split into the following stages:

(24)

• problem formulation,

• collecting data,

• choice and application of a clustering algorithm,

• evaluation of its results.

While the problem of clustering seems to be obvious in general terms, its formali- sation is not as obvious in concrete case. It requires first a formal framework for the data to be learned from, as outlined in Sect.2.1. Furthermore, the user is requested to define similarity between the objects (the geometry of the sample space) as well as a cost function (a measure of value of the obtained clusters). The definition of the geometry must be application-specific, tuned to the specific needs of the user.

Section2.2provides with a guidance in this process, listing many possibilities of measuring similarity both in numerical and non-numerical data and explaining some special properties of these measures. Section2.4.1provides with a set of possible cost functions to be used in the clustering process. Note, however, that each concrete algorithm targets at some specific cost function.

Advices on the choice of data for cluster analysis can be found in Sect.4.1.

To enable the choice of an appropriate clustering algorithm, Sect.2.3provides with an overview of hierarchical clustering methods, Sect.2.4presents partitional (called also combinatorial) clustering methods, Sect.2.5hints at other categories of clustering procedures like relational, spectral, density, kernel and ensemble methods.

Some of these types of algorithms are reflected on more deeply in Chaps.3,5and6, as already mentioned.

The results of the clustering may be looked at from two perspectives: whether or not the choice of parameters of the algorithm was appropriate (see Sect.4.2for the choice of the number of clusters), and if the results meet the quality expectations (Sect.4.3ff). These quality expectations may encompass:

• the cost function reached sufficiently low value,

• the homogeneity of clusters and their separation are sufficient (aesthetically or from business point of view),

• the geometry of the clusters corresponds to what the algorithm is expected to provide,

• the clusters correspond to some external criteria, like manually clustered subsets etc.

User’s intuition and/or expectations with respect to the geometry of clusters is essential for the choice of the clustering algorithm. One shall therefore always inspect the description of the respective algorithm if it can properly respond to our expectations. In fact the abundance of clustering algorithms is a derivative of the fact that these expectations may be different, depending on actual application.

So if the geometry of the data makes them suitable for representation in a feature space, then visit first the methods described in Chap.3. If they are conveniently represented as a graph, then either Chap.5or6are worth visiting.

If in the feature space, you may expect that ideally your clusters should form clearly separated (hyper)balls. In this case k-means algorithm (Sect.3.1) may be

(25)

6 1 Introduction

most appropriate. If the shape should turn out to be more ellipsoidal, see Sect.3.3.5.4.

Should the clusters take forms like line segments, circles etc., visit Sect.3.3.5.3. If you need to cluster text documents in vector space representation (which lie essentially on a sphere in this representation), then you may prefer spherical-k-means algorithm (Sect.3.1.5.3). It may happen that you do not expect any particular shape but rather oddly shaped clouds of points of constant density, you may prefer DBSCAN or HDBSCAN algorithm (Sect.2.5.4). If you expect that the border line between the clusters is not expressible in simple terms, you may try kernel methods (Sects.2.5.7 and3.3.5.6). Last not least you may seek a sharp split of objects into clusters or a more fuzzy one, using e.g. Fuzzy-C-Means algorithm.

There exist also multiple geometry choices for graph-based data representation.

You may seek dense subgraphs, subgraphs, where a random walker would be trapped, subgraphs separable by removal of a low number of edges or subgraphs where connections evidently are different from random.

The main topic of this book is to provide with a basic understanding of the formal concepts of the cluster, clustering, partition, cluster analysis etc. Such an understanding is necessary in the epoch of Big Data, because due to the data volume and characteristics it is no more feasible to rely solely on viewing the data when fac- ing the clustering problem. Automated processing to the largest possible extent is required and it can only be achieved if the concepts are defined in a clear-cut way. But this is not the only contribution of this book to the solving problems in the realm of Big Data.Big Datais usually understood as large amounts of data in terms of over- all volume measured in (peta)bytes, large number of objects described by the data and lengthy object descriptions in a structured, unstructured or semi-structured way.

Therefore a user interested in Big Data issues may find it helpful to get acquainted with various measures of object similarity as described in Sect.2.2, where measures based on quantitative (like numerical measurement results) and on qualitative features (like text) as well as on their mixtures are described. As Big Data encompass hypertext data or other linked objects, graph-based similarity measures described in Sects.6.2and6.4.4may be also of interest. For the discussion of other complex structures in the data, like multilayered graphs, see Sect.6.10. Various variants, how such similarity measures can be exploited when defining clustering cost functions are described in Sect.2.4.1. Alternatively, they can be used in spectral analysis of graphs, described in Chap.5.

Last not least we present a number of approaches to handle large number of objects in reasonable time. The diversity of such algorithms encompasses so-called grid-based methods, see Sect.2.5.5, methods based on sampling, see Sect.3.10, par- allelisation via map-reduce, usage of tree-structures, see Sect.3.1.4, to various heuris- tic approaches like those used for community detection, see Sect.6.7. For particular algorithms we provide also with references to publications of variants devoted to huge data sets. Note that in some cases, like spectral clustering, accommodation to big data sets requires some new theoretical developments, enabling for example to sample linked data in a reasonable way, see Sect.5.5.

(26)

Alternatively, especially with large bodies of textual data, data projection onto a smaller feature space can accelerate processing, like e.g. random projection, see Sect.3.9.

In order to concentrate the narrative of the book on the essentials of the clustering problem, some more general considerations have been moved to Appendices A to E.

The limited volume of this book did not allow to cover all the developments in the field. On the one hand, we do not discuss algorithms elaborated for theoretical purposes of clusterability theory. On the other hand, we do not consider algorithms developed for particular applications only, like medical imaging. We concentrated on the clustering paradigm of a cluster as group of objects that are more similar to one another than to objects from other clusters. But other concepts are possible, based e.g.

on classification/regression capabilities. Some researchers consider clusters as sets of objects where the attributes are (more) easily predictable from one another than in general population (mutual prediction). We mention only in passing that one may seek clusters either in a given space or in its subspaces (projections of the original space), but this is by itself a separate research area, answering questions what is the optimal number of features and which subspace (possibly linearly transformed) should be used. Still another research area concentrates around the so-called optimal modularity, where we seek clusters in which similarity pattern does not occur to be random. Still another area of investigation are special forms of optimised cluster quality function which ensure better convergence, like sub-modular functions.

(27)

Chapter 2 Cluster Analysis

Abstract This chapter outlines the major steps of cluster analysis. It starts with an informal introduction to clustering, its tools, methodology and applications. Then it proceeds with formalising the problem of data clustering. Diverse measures of object similarity, based on quantitative (like numerical measurement results) and on qualitative features (like text) as well as on their mixtures, are described. Various variants, how such similarity measures can be exploited when defining clustering cost functions are presented. Afterwards, major brands of clustering algorithms are explained, including hierarchical, partitional, relational, graph-based, density-based and kernel methods. The underlying data representations and interrelationships between various methodologies are discussed. Also possibilities of combining several algorithms for analysis of the same data (ensemble clustering) are presented. Finally, the issues related to easiness/hardness of the data clustering tasks are recalled.

Cluster analysis consists in distinguishing, in the set of analysed data, the groups, called clusters. These groups are disjoint¹ subsets of the data set, having such a property that data belonging to different clusters differ among themselves much more than the data, belonging to the same cluster. The role of cluster analysis is, therefore, to uncover a certain kind of natural structure in the data set. The means enabling performing that task is constituted usually by a certain measure of similarity or dissimilarity—the issue is discussed further on in Sect.2.2. Cluster analysis is not only an important cognitive tool, but, as well, a method for reducing large sets of data, since it allows for the replacement of a group of data by its compact characterisation, like, e.g. the centre of gravity of the given group.

The task of cluster analysis can be perceived as a problem of grouping of objects according to their mutual similarity. Objects, which are mutually similar in a sufficiently high degree, form a homogeneous group (a cluster). It is also possible to consider the similarity of objects to certain characteristic entities (called prototypes) of the classes. In this case we deal more with the problem of classification, that is—of finding the model patterns—see [164]. Yet, if the characteristics corresponding to

1The requirement of disjoint subsets is used in the classical data analysis. In the general case, the groups distinguished might constitute thecoverageof the data set. This is the case, for instance, in the fuzzy data analysis.

S. T. Wierzcho´n and M. A. Kłopotek,Modern Algorithms of Cluster Analysis, Studies in Big Data 34, https://doi.org/10.1007/978-3-319-69308-8_2

9

(28)

classes are not given a priori, they should be established. And this is exactly what cluster analysis is about.

The termdata clustering(meaning grouping of data) appeared for the first time in 1954 in the title of a paper concerning the analysis of anthropological data [272, p. 653]. Other equivalent names, given cluster analysis, areQ-analysis,typology, clumping, andtaxonomy [273], depending on the domain, in which clustering is applied. There is a number of very good books, devoted to cluster analysis. The classical ones include²: [19,164,173,273,284,436,463]. Among the more recent and specialised monographs we can mention:[9, 70, 73, 75, 140, 297, 387, 402, 453]. A reader, interested in survey works may wish to have a look at, e.g., [66,272, 274,515]. More advanced techniques of cluster analysis are considered, in particular, in [176].

It is worth noting that both in common language and in computer science there exists a bunch of synonyms or closely related concepts for the termcluster analysis which reveal various aspects of this term. One of them is theunsupervised learning orlearning without a teacher. This term suggests that we are looking for a hidden structure behind the data that we want to reveal. It suggests also that it must pay off to have recovered such a structure that is there must exist a criterion saying whether or not a useful structure was discovered. Last not least the hidden structure has to be learnable from the data in the sense of learnability theory. One speaks frequently about client segmentation,image segmentationetc. Image segmentation means that we look for a particular structure behind the image data—a trace of one or more physical objects, to be separated from the background. Criteria of belonging to the same object may include local properties, like continuity of colour, shading, texture, or global ones, like alignment along a line of limited curvature etc. Client segmentation on the other hand tries to split the population into homogeneous sections concentrated around some centric concepts so that predominantly global quality criteria will be of interest. And of course one asks the representativeness of the client population via the split into categories (generalization capability), easiness of assignment of new client to an existent category (mutual prediction of attributes within a category) and about profitability of cluster assignment from the business point of view (a kind of the cluster separation criterion). The frequently used term taxonomy formation points at the need of not only splitting the objects into classes or categories, but also of providing with a simple, compact description. Hence it is important to note that we are not talking just about clustering or grouping of objects, but rather the term analysisshould be stressed because various cluster analysis methods reveal in the data structures of different types and the user of such methods either should be aware of what kind of hidden information he is looking for so that he chooses appropriate methods or he should be aware of what type of output he got if he applied a bunch of various methods for exploratory purposes.

For these reasons, the authors of this book do not try to point out the best method for clustering, neither explicitly nor implicitly, but rather concentrate on pointing

2Many of those listed have been modified several times over and successive editions have been published.

(29)

2 Cluster Analysis 11 at particular advantages of individual methods and methodologies while covering a broad spectrum of potential and actual applications of cluster analysis.

It is the researcher himself who needs to know his goals and to choose the appropriate clustering approach that fits best his purposes. In order to help to understand the importance of user’s goals, let us recall the once famous “Ugly Duckling Theo- rem” of Watanabe³Assume objects in a collection are described by binary attributes taking on values “true” and “false”. Assume that two objects are more similar if they share more “true”-valued attributes. Furthermore, let us include in object description not only the original attributes but also their derivatives (obtained via logical operations “and”, “or”, “not”). Under these circumstances any object is similar to any other so that no clustering makes sense. This fact has a number of other con- sequences, which are frequently ignored: dropping/selecting/weighting appropriate subset of these attributes may lead to any similarity structure we wish to have. It means also that the very same set of objects may have different meaningful clus- terings depending on the purpose of clustering and that application of an irrelevant clustering algorithm may lead to completely useless cluster sets. Therefore before starting cluster analysis, we need to have clear criteria on how we evaluate the quality of a single cluster (homogeneity) and/or the quality of the interrelationships between clusters (separation), and of course, what similarity/dissimilarity of objects means in our application area.

So e.g. when applying cluster analysis for purposes of delimitation of species of plants or animals in biology, the choice of algorithms and attributes will be driven by the assumed definition of species, e.g. via the requirement of (direct or indirect) interbreeding capability. While there is a limited amount of experimental data on interbreeding, the clustering performed on other available attributes is considered correct if it matches the known interbreeding possibilities and/or restrictions. Med- ical classification of diseases would be validated on/driven by a compromise between common symptoms, known causes and treatment possibilities. Image segmentation will understand clusters of pixels as homogeneous if they belong to the same organ, blood vessel, building, connected area covered with same plants etc. Market segmentation will be successful if we discover groups of clients worthy/not worthy addressing in marketing campaigns. Geographical client segmentation for purposes of store chains may be evaluated on the discovery of new profitable locations of stores etc. Clustering in databases aims at efficient organization of data for query processing.

Whatever the domain-specific expectations, one may apply cluster analysis with diverse goals of later usage of the results. One may perform exploratory data analysis seeking interesting patterns just to identify hypotheses worthy further investigation.

One may be interested in data reduction (compression) and structuring for efficient storage and access. Or one may look for a clustering that aligns with other groupings to investigate their pertinence or universality.

3Watanabe, Satosi (1969). Knowing and Guessing: A Quantitative Study of Inference and Informa- tion. New York: Wiley, pp. 376–377.

(30)

Note also that depending on the application, the criteria for homogeneity and separation may be diverse. Let us just mention a few:

• homogeneity criteria

– small within-cluster dissimilarities,

– clusters fitting some homogeneous probability models like the Gaussian or a uniform distribution on a convex set, or some functional, time series or spatial process models,

– members of a cluster well represented by its centroid,

– clusters corresponding to connected areas in data space with high density, – clusters fitting into convex shapes (e.g. balls)

– clusters characterisable by a small number of variables.

– features approximately independent within clusters;

• separation criteria

– large between-cluster dissimilarities

– dissimilarity matrix of the data reflected by the clustering

– constraints on co-occurrence/not co-occurrence in the same cluster matched – cluster stability (e.g. under resampling, bootstrapping)

– low number of clusters

– clusters of roughly the same size

– balanced distances between cluster centres along a minimum weight spanning tree

– clusters corresponding well to an externally given partition or values of one or more variables that were not used for computing the clustering

– clusters fitting partial labelling

– learnability of cluster membership from data

– for hierarchical clusters the distance between cluster members reflected by the hierarchy level when they join the same cluster

The above criteria serve only as examples of possible cluster/clustering quality and whether or not any, some or large portion of them is actually used in a concrete application case will depend on the interests of the researcher.

We would further need to specify, in what sense a given criterion should be satis- fied, e.g. by just passing a threshold or by reaching a local/global optimum or deviate only by some percentage from such an optimum. If we define multiple optimality criteria, we would need to reconcile them by stating a strategy of compromise between them. These and various other aspects of clustering philosophy are discussed in-depth in the paper [250] by Hennig.

(31)

2.1 Formalising the Problem 13

2.1 Formalising the Problem

It is usually assumed in cluster analysis, that a set ofmobjectsX= {x1, . . . ,xm}is given.

For purposes of analysis the set of objects may be characterised by an embedded or relational representation.

In theembedded representation, every object is described by ann-dimensional vectorxi=(xi1. . .xi n)^T, wherexi jdenotes the value of thej-th feature of the object xi. The vectorxi is being called feature vector or image.⁴

The subject of cluster analysis is constituted, therefore, not so much by the original set of objectsX, as by its representation, given through the matrixX =(x1. . .xm)^T, whosei-th row is a vector of features, describing thei-th object. In view of the fact that to objectxicorresponds thei-th row of matrixX, the term “object” shall be used to denote both the elementxi ∈Xand the vector of feature valuesxi, characterising this object. In statistics, vectorxiis called (n-dimensional) observation. Even though the values of individual measurements of feature values might be expressed in different scales (nominal, ordinal or quotient), the majority of practical and theoretical results have been obtained under the assumption that the components of vectorsxiare real numbers. Thus, for a vast majority of cases, considered in the present book, we shall be assuming that the observations are given by the vectorsxi∈Rⁿ.

This kind of data representation will be explored in the current chapter as well as in Chaps.3and4.

It is alternatively assumed (in therelationalrepresentation) that information on the data set is provided in the form of the matrixSof dimensionm×m, the elements of this matrixsi j represent similarities (or dissimilarities) for the pairs of objectsxi, xj. When making use of such a representation, we can give up the requirement of having the features, describing the objects, measured on the quantitative scales. The pioneer of such a perspective was Polish anthropologist, ethnographer, demogra- pher and statistician—Jan Czekanowski.⁵The method, developed by Czekanowski, and presented in the first handbook of modern methods of data analysis and inter- pretation of its results [129], consists in replacing numbers in the matrix S by the appropriately selected graphical symbols. In this manner, an unordered diagram (called Czekanowski’s diagram) arises, which, after an adequate reordering of rows and columns of the matrix, makes apparent the existence of groups of objects mutually similar. Extensive description of his idea provide Graham and Hell in [213]. To illustrate this approach consider a small matrix

4The latter term is particularly justified, when we treat the measurements as mappingsf:X→Rⁿof the set of objects into a certain set of values. Then,x_i=f(xi), and in the mathematical nomenclature x_iis the image of the object_xi.

5See J. Gajek. Jan Czekanowski. Sylwetka uczonego.Nauka Polska, 6(2), 1958, 118–127.

(32)

∇ =

⎡

⎢⎢

⎣

1 2 3 4 5 6

1 1.0 0.2 0.1 0.3 0.8 0.4

2 0.2 1.0 0.1 0.9 0.3 0.3

3 0.1 0.1 1.0 0.2 0.2 0.7

4 0.3 0.9 0.2 1.0 0.4 0.1

5 0.8 0.3 0.2 0.4 1.0 0.2

6 0.4 0.3 0.7 0.1 0.2 1.0

⎤

⎥⎥

⎦

and let us denote bythe degrees smaller than 0.5 and bythe degrees greater than 0.5. Then the above matrix can be represented graphically as in left part of Eq.

(2.1). After reordering its rows and columns we obtain the matrix depicted in right part of the Eq. (2.1) revelling three clusters:{1,5},{2,4}and{3}.

1 2 3 4 5 6

1

2

3

4

5

6

1 5 2 4 3 6

1

5

2

4

3

6

(2.1)

Although the method was developed originally more than 100 years ago, it is still in use, mainly in archaeology,⁶in economic sciences,⁷and even in musicology.

Using the method of Czekanowski, the mathematicians from Wrocław: K. Florek, J. Łukaszewicz, J. Perkal, H. Steinhaus and S. Zubrzycki, elaborated the so-called

“Wrocław taxonomy”, which they presented in 1957 in 17th issue of the journal Przegl¸ad Antropologiczny(see also [515]). In further parts of the book we present other methods of analysing the matrix of similarities. More advanced considerations of application of the similarity matrix in cluster analysis have been presented in the references [44, 45]. This type of representation is addressed primarily in Chaps.5 and6.

The role of the classical cluster analysis is to split the set of objects (observations) into k<m groupsC= {C1, . . . ,Ck}, where eachi-th groupCi is called cluster.

Such a division fulfils three natural requirements:

(i) Each cluster ought to contain at least one object,Cj = ∅,j =1, . . . ,k.

(ii) Each object ought to belong to a certain cluster,k

j=1Cj =X.

(iii) Each object ought to belong to exactly one cluster,Cj₁∩Cj₂= ∅,j1= j2.

6See, e.g., A. Soltysiak, and P. Jaskulski. Czekanowski’s Diagram: A method of multidimensional clustering. In: J.A. Barceló, I. Briz and A. Vila (eds.)New Techniques for Old Times. CAA98. Com- puter Applications and Quantitative Methods in Archaeology. Proc. of the 26th Conf., Barcelona, March 1998 (BAR International Series 757). Archaeopress, Oxford 1999, pp. 175–184.

7See, e.g., A. Wójcik. Zastosowanie diagramu Czekanowskiego do badania podobie´nstwa krajów Unii Europejskiej pod wzgl¸edem pozyskiwania energii ze ´zródeł odnawialnych.Zarz¸adzanie i Finanse(J. of Management and Finance), 11(4/4), 353–365, 2013,http://zif.wzr.pl/pim/2013_4_

4_25.pdf.

(33)

2.1 Formalising the Problem 15 In particular, whenk=m, each cluster contains exactly one element from the set X. This partition is trivial, and so we shall be considering the cases, in whichkis much smaller thanm.

An exemplary illustration of the problem that we deal with in cluster analysis, is provided in the Fig.2.1. The “clouds” of objects, situated in the lower left and upper right corners constitute distinctly separated clusters. The remaining objects form three, two, or one cluster, depending on how we define the notion of similarity between the objects.

Cluster analysis is also referred to as unsupervised learning. We lack here, namely, the information on the membership of the objects in classes, and it is not known, how many classes there should really be. Even though the mechanical application of the algorithms, which are presented in the further parts of the book allows for the division of any arbitrary set into a given number of classes, the partition thus obtained may not have any sense. Assume thatXis a set of points selected conform to the uniform distribution from the set[0,1] × [0,1]—see Fig.2.2a. Mechanical application of an algorithm of grouping, with the predefined parameterk=3 leads to the result, shown in Fig.2.2b. It is obvious that although the partition obtained fulfils the conditions set before, it has no sense.

In informal terms, the presence of a structure in a data set is manifested through the existence of separate areas, and hence of clusters, enjoying such a property that any two objects, belonging to the common clusterCiare more mutually similar than any two objects, picked from two different clusters, i.e.

Fig. 2.1 The task of cluster analysis: to break down the set of observations intok disjoint subsets, composed of similar elements

Fig. 2.2 Grouping of the set of randomly generated objects.a100 points randomly selected from the set[0,1] × [0,1].bDivision of the objects into three groups using thek-means algorithm. Bigger marks indicate the geometrical centers of groups