CSC662 Data Mining, Data Warehouse and Visualization

(1)

8 ก ก !ก "#

$%. . ก& "##'"(

)*#)+# %" ' )+*#%" ' &,ก+'*#

CSC662 Data Mining, Data Warehouse and

Visualization

July 1, 2007 การจัดจําแนกประเภท ตนไมการตัดสินใจ 2

)* ก -*-ก กก

". กก ก

#ก

ก ก !ก "#

**กก/+

0(กกก#!กก

$12' !ก "# - 3

)-

CRISP-DM

ก*ก))*

ก ก

ก ก (Classification):

4*#2กกก4 ก/+ 5

)" 67กกก/+ )-!- -

$12'! * ก(Classifier) 1 !ก *- () !-)"

ก (Prediction):

4กก) *+)- ก 1 )-ก กก ก/+ )- - -ก )-!

$12'! * (Predictor) 1 !+)- *- () "

ก กกก

(2)

." *: กก ก/+ 4)"

--7)"-.

*"ก training set 4ก8 !"

. *: %ก/+ 9กก!--)"

) *+)-)" *-" *

ก*)*9ก * ก)#) train-test set train-validate-test set * ก!ก: - *- ก test set )*9ก $-ก+;'ก

0(ก ก". ก

Training Data

N A M E R A N K Y E A R S T E N U R E D Mike Assistant Prof 3 no Mary Assistant Prof 7 yes

Bill Professor 2 yes

Jim Associate Prof 7 yes

Dave Assistant Prof 6 no Anne Associate Prof 3 no

. *#2 Classification

IF rank = ‘professor’

OR years > 6

THEN tenured = ‘yes’

Classifier ( *)

." *

Classifier

Test

N A M E R A N K Y E A R S T E N U R E D

Tom Assistant Prof 2 no

Merlisa Associate Prof 7 no

George Professor 5 yes

Joseph Assistant Prof 7 yes

!-)1ก-

(Jeff, Professor, 4)

Tenured?

. *

ก ก* * "9!*-4ก*)

(Machine learning) กก&- *-ก *!- ก8 ! )-

ก+ * กกกก train-test set ! *"9 train set !-!-$#1 - *ก-*!ก test set ก!$12'!- ก8ก+' ก-*ก*- 0(กกก#! (overfitting problem)

ก #*#2กก0(ก-*)กก7& validate set

กก *กก1# ' - 3 1! * !ก *-) !

#ก

(3)

ก supervised (classification)

กก ก/+ 5 " ก)"5

ก*-$#9ก 1#+ก)-)"5 ก unsupervised (clustering)

!-กก ก/+ 5

กก#กกก *-ก4ก&-" *- )ก-ก&-*ก - *- -ก- -ก&-

ก*)*))* -ก #ก6' (Metric) ) *+ก)- ก/+ &กก/+ - *-

)* ก -*-

supervised

ก

unsupervised

)*9ก ก *-!-)1ก- (Accuracy from test set) *" * (Model building time)

)* -$#ก # *!-!กก&- ก1#$#ก # (Robustness to noisy data)

* *(-!-กก !ก

#+ก (Model size)

)*)* * *) 2#".ก)*

"ก*- *) 2#* (Model interpretability)

**1ก * ก

!ก "#: )ก< !ก&

& ก/+ !-)"

)-$ก"

)-)"กก)"

ก" !ก "#ก* 2 .

. ก" !

#กกก& training data .

กก&- *- ก/+ ก *ก6.

. ก* -ก#

*ก ก#$ก -)*9ก ก

ก ก !ก "#

no yes

attribute?

age income student credit_rating buys_computer

<=30 high no fair no

<=30 high no excellent no

31…40 high no fair yes

>40 medium no fair yes

>40 low yes fair yes

>40 low yes excellent no

31…40 low yes excellent yes

<=30 medium no fair no

<=30 low yes fair yes

>40 medium yes fair yes

<=30 medium yes excellent yes

31…40 medium no excellent yes

31…40 high yes fair yes

>40 medium no excellent no

*-ก

Quinlan’s ID3

*-ก6.)1#* '

18

(4)

!ก "#ก6.)1#* '

no yes excellent fair

<=30 >40

no yes no yes

yes

31..40

age?

student? credit rating?

กก (Greedy approach)

4 top-down recursive divide-and-conquer # ก&ก *-ก

กก/+ !- - categorical -ก. *- )-ก/+ กก6.

ก/+ 9กกก ** - information gain !ก&ก-

*-&ก *)-)"ก

!-ก/+ ก- ก)-)" majority voting !- *- )-)"&-7.!

*)#" )(ก" !

)-ก""% Information gain (ID3) กก/+ 4categorical

ก/+ 4)- -4)-!- -ก- )-""-*ก Gain ratio (C4.5)

!ก)-ก""%*)-""%ก/+

)- Gini index (IBM IntelligentMiner)

"9กก/+ 4)-!- -)- -ก:!

กก-ก/+ )- ก -กก

**กก/+

กก/+ )-ก""%""&

1#+ก&--4")") P N

* *-)" P ) p * * *-)"N ) n *

)-""%ก&-))-))ก&- *- *#

กก)" PN#

ก) *+)-""% Information

Ip , n= p

pnlog₂



pn^p



pnⁿ log₂



pⁿn



(5)

ก A)ก/+ - S ก4{S₁, S₂ , …, S_v}

^Sⁱ *-ก)" P * p_i *-ก)" N * n_i, )-:= (entropy) )-))กกก/+

A )

)-ก""% (Information gain) !กกก*

ก/+ A

Gain(A) = I(p, n) - E(A)

ก) *+)-ก""% Information gain

EA=

∑

i=1

v p_in_i

pn Ip_i, n_i

g)"P: buys_computer = “yes”

g)"N: buys_computer = “no”

I(p, n) = I(9, 5) =0.94029

g) *+)-:=" age: .

Gain(age) = I(p, n) - E(age)

*ก

Gain(age) = 0.2467 Gain(income) = 0.0292 Gain(student) = 0.1518 Gain(credit_rating) = 0.0481 age p_i n_i I(p_i, n_i)

<=30 2 3 0.971

30…40 4 0 0

>40 3 2 0.971

กกก/+ )-ก""%

Information gain

Eage= 5

14 I2, 3 4

14I4, 0

 5

14I3, 2=0.69354

12

g)"P: buys_computer = “yes”

g)"N: buys_computer = “no”

I(p, n) = I(9, 5) =0.94029

g) *+)-:=" age:

.

Gain(age) = I(p, n) - E(age)

)-""% age ) 1.5774

. GainRatio(age) = 0.1564 GainRatio(income) = 0.0188 GainRatio(student) = 0.1518 GainRatio(credit_rating) = 0.0488 age pi ni I(pi, ni)

<=30 2 3 0.971

30…40 4 0 0

>40 3 2 0.971

กกก/+ )- "-*ก Gain ratio

Eage= 5

14I2, 34

14I4, 0

 5

14I3, 2=0.69354

12

*-. S n )", 6 gini #

p_j ))-)*9"12')" j S

6 S 9ก-ก/+ A ก4 S₁ S₂ 67 n₁

n₂ , )- gini 9ก-*ก/+ A )

ก/+ )- gini_split(A) :ก"&9กก ( ก)-&ก)-4

&-)

Gini Index (IBM IntelligentMiner)

giniS=1

∑

j=1 n

p²_j

gini_splitA=n₁

n giniS₁n₂

n giniS₂

(6)

ก *-ก6.)1#* ' S 2 )") buys_computer = yes ก buys_computer = no .

gini(S) = 1 - (9/14)² – (5/14)² = 0.45918 1#+ก/+ age - S ก4""-* )- gini

ก/+ age )

gini(S, age) = 0.34286 gini(S, income) = 0.44048 gini(S, student) = 0.36735 gini(S, credit_rating) = 0.42857

ก) *+)- Gini Index

ก8ก ก"9ก !ก "# IF-THEN*#2 ก)กก8ก -*#9

)- attribute-value *#9ก8ก8* and (conjunction)

ก "-*"&ก8 67)-)"

ก")*ก8 &/'"9 !!-ก*- *-

IF age = “<=30” AND student = “no” THEN buys_computer = “no”

IF age = “<=30” AND student = “yes” THEN buys_computer = “yes”

IF age = “31…40” THEN buys_computer = “yes”

IF age = “>40” AND credit_rating = “excellent” THEN buys_computer = “no”

IF age = “>40” AND credit_rating = “fair” THEN buys_computer = “yes”

ก"ก8ก ก

0(กกก#!)ก *!กก training data )- )*9ก ก-ก)"5" - !ก test data !)-)*9ก

ก-*ก7) *!4กก training data ก -!-"9 !ก!-)1ก-!

0(.ก1 *)-"กก * hidden

node กก#)* 4 training 9ก !* hidden

node )-")-9-*. ก!)-)*9ก "

ก

0(กกก#! Overfitting problem

!ก "#"ก/+0(กกก#!ก4 ! ก กก#ก -ก# *-#+

*#2กก0(กกก#! !ก "# !"ก/+

ก -ก#*-" !: &ก" ! **ก ก ก#)- ก*-ก+;'ก - *&ก#!

ก -ก#" !: ก *ก# ก ก#))*9ก ก ก!

*#2ก*#2- validation data 1"ก -ก#ก !"& “best pruned tree” " validation data

ก0(กกก#!

(7)

1.กก4 training set (4/10), validation set (3/10)

test set (3/10)

2. k-fold cross validation -4^k"-*- 3 ก¹"-*4

test data⁹"-*4 training data

3.. training data statistical test (- chi-

square) ก+*-ก ก#-*$กก9ก

ก7.!-

4.ก minimum description length (MDL) &ก กก#

*#9*ก#)-^MDLก

กกก !"&

!ก "#!กก/+ )- - ก 4)-!- -*-ก" ! ก ก-ก" !

กกก/+ !

)-ก8-"&ก/+

) *+)-)*-4 *กกก-*ก )-!

"ก/+ -

*)-ก/+ ก!-9ก"4ก/+ -

-* ก กก- (fragmentation), กก#6. (repetition),

ก6. 6 (replication)

ก)*"9 !ก "#

*#2ก ก (Classification) ก%7ก/*#." "9# #

*#ก)1#* ' )""%

กก. *#2"9กก(-40("

(Scalability)

*#2ก !ก "#!)*"1

*ก".ก*-*#2

$12'!4ก8!-

กก ^SQL . *#2"#2#17.

)*9ก ก)-"ก*#2

ก กก>(-

4ก* generalization ก decision-tree induction (Kamber

et al’97)

Classification >>

- &+#4% → )*. → "1ก%

ก-กกก)"ก !41&-

0(ก)*1"ก$$#

Cube-based multi-level classification ก*#)' Relevance analysis -

*#)' Information-gain analysis *ก# #.

ก !ก "#ก data cube

(8)

*-$12' !ก "#

ตัวอยางตนไม J48 จาก Weka

"&

ก ก4ก supervised learning 7 *ก ก") !ก "# 67"

. *#2*ก#

กกก/+ 1ก- "9 **กก ** - Information gain, Gain Ratio, Gini index

1ก0(กกก#! !ก "#!9ก -ก#

ก -ก#*-ก" ก -ก#ก"

$12'!"9" ก8 1 ก<Tก

(9)

C. Apte and S. Weiss. Data mining with decision trees and decision rules.

Future Generation Computer Systems, 13, 1997.

U. M. Fayyad. Branching on attribute values in decision tree generation.

In Proc. 1994 AAAI Conf., pages 601-606, AAAI Press, 1994.

J. Gehrke, R. Ramakrishnan, and V. Ganti. Rainforest: A framework for fast decision tree construction of large datasets. In Proc. 1998 Int. Conf.

Very Large Data Bases, pages 416-427, New York, NY, August 1998.

M. Kamber, L. Winstone, W. Gong, S. Cheng, and J. Han. Generalization and decision tree induction: Efficient classification in data mining. In Proc. 1997 Int. Workshop Research Issues on Data Engineering (RIDE'97), pages 111-120, Birmingham, England, April 1997.

ก"# U

M. Mehta, R. Agrawal, and J. Rissanen. SLIQ : A fast scalable classifier for data mining. In Proc. 1996 Int. Conf. Extending Database Technology (EDBT'96), Avignon, France, March 1996.

S. K. Murthy, Automatic Construction of Decision Trees from Data: A Multi-Diciplinary Survey, Data Mining and Knowledge Discovery 2(4):

345-389, 1998

R. Rastogi and K. Shim. Public: A decision tree classifer that integrates building and pruning. In Proc. 1998 Int. Conf. Very Large Data Bases, 404-415, New York, NY, August 1998.

J. Shafer, R. Agrawal, and M. Mehta. SPRINT : A scalable parallel classifier for data mining. In Proc. 1996 Int. Conf. Very Large Data Bases, 544-555, Bombay, India, Sept. 1996.

S. M. Weiss and C. A. Kulikowski. Computer Systems that Learn:

Classification and Prediction Methods from Statistics, Neural Nets, Machine Learning, and Expert Systems. Morgan Kaufman, 1991.

ก"# V