Pengolahan Database Parallel

(1)

Pengolahan Database Parallel

Imam Machdi

Graduate School of Systems and Information Engineering

University of Tsukuba

Presentasi PPI Ibaraki

Tsukuba, 7 Pebruari 2010

Daftar Isi

• Pendahuluan

• Arsitektur Pengolahan Parallel

• Parallel Relational Database

• Parallel XML Database

(2)

Pendahuluan

Database

Sistem Pengolahan Data

Data

Fasilitas Komputasi

(3)

Database

Sistem Pengolahan Data

-Problem

Data

Fasilitas Komputasi

Pengguna

Database

Data

Pengguna

Sistem Pengolahan Data –

Solusi

Database

Data

Kolaborasi

(4)

Distribusi workload

Pengolahan Parallel

• Data Parallelism

• Partisi data

• Distribusi data

• Task Parallelism

• Partisi task

• Distribusi task

dikerjakan secara simultan

Meningkatkan Performa

Data 1

Task 1

Data 2

(5)

Contoh

• Mengolah 1 TB data

• Kecepatan pengolahan 1MB/det.

• Waktu yang diperlukan

• 10.048.576 detik atau

• 174.760 menit atau

• 2910 jam atau

• 120 hari

• Diperlukan

Pengolahan Parallel

untuk

meningkatkan performa pengolahan.

(6)

Multi-Computers

• Menggunakan beberapa PC yang terkoneksi

dengan LAN.

• PC relatif murah dan mudah didapat.

shared-nothing on

CPU, memory, disk

komunikasi

Multi-Computers

• Biaya komunikasi antar PC tinggi.

• Data ditransfer via network perlu waktu lebih

lama dibandingkan dengan transfer di memory.

• Library

• Parallel Virtual Machine (PVM)

• Message Passing Interface (MPI)

• Bahasa pemrograman C/C++

(7)

Multi-Core Computer

• 1 processor terdiri dari beberapa core.

• Dual core, quad core.

• Shared-everything (something)

• Cache, memory, disk

Core 1

Core 2

Memory

Multi-Core Computer

• Biaya komunikasi lebih rendah.

• Perlu menghindari memory contention

• Library

• Threads

• OpenMP

• Peningkatan performa

• Exploitasi cache

• Kurangi synchronization

Core

0 Core

1 CPU

Memory

Cache

(8)

Parallel Relational Database

Tabel Pegawai

No Peg Nama

Tgl Lahir Alamat

Kota

Gol Darah Jabatan

1001 Asep

1-Jan-01 Ichinoya

Tsukuba

A

Staf

1002 Bram

2-Feb-02 Oikoshi

Tsukuba

B

Staf

1003 Asep

3-Mar-03 Hirasuna

Ami

O

Supervisor

1004 Cici

4-Apr-04 Ichinoya

Tsukuba

AB

Supervisor

….

…..

….

1005 Zazang

12-Dec-02 Hirasuna

Ami

A

Staf

Relational Model

• Data direpresentasikan dalam bentuk tabel

2-dimensi (terstruktur)

Atribut

Key

(9)

Query

• Structured Query Language (SQL)

• Cari Asep yang tinggal di Tsukuba

• SELECT NoPeg, Nama, Kota FROM Pegawai

WHERE Nama = ‘Asep’ AND Kota = ‘Tsukuba’.

Tabel Pegawai

No Peg Nama

Tgl Lahir Alamat

Kota

Gol Darah Jabatan

1001 Asep

1-Jan-01 Ichinoya

Tsukuba

A

Staf

1002 Bram

2-Feb-02 Oikoshi

Tsukuba

B

Staf

1003 Asep

3-Mar-03 Hirasuna

Ami

O

Supervisor

1004 Cici

4-Apr-04 Ichinoya

Tsukuba

AB

Supervisor

….

…..

….

1005 Zazang

12-Dec-02 Hirasuna

Ami

A

Staf

Partisi Data

• Partisi data untuk menciptakan parallelism.

• Partisi horisontal

• Partisi vertikal

No Peg

Nama

Tgl Lahir Alamat

Kota

Gol Darah

Jabatan

1001 Asep

1-Jan-01 Ichinoya

Tsukuba

A

Staf

1002 Bram

2-Feb-02 Oikoshi

Tsukuba

B

Staf

….

…..

….

No Peg

Nama

Tgl Lahir Alamat

Kota

Gol Darah

Jabatan

1003 Asep

3-Mar-03 Hirasuna

Ami

O

Supervisor

SELECT NoPeg, Nama, Kota

FROM Pegawai

WHERE Nama = ‘Asep’

AND Kota = ‘Tsukuba’

SELECT NoPeg, Nama, Kota

FROM Pegawai

WHERE Nama = ‘Asep’

(10)

Partisi Data

• Partisi data untuk menciptakan parallelism.

• Partisi horisontal

• Partisi vertikal

No Peg Nama Tgl Lahir Alamat

1001 Asep 1-Jan-01 Ichinoya 1002 Bram 2-Feb-02 Oikoshi 1003 Asep 3-Mar-03 Hirasuna 1004 Cici 4-Apr-04 Ichinoya …. ….. …. ….

1005 Zazang 12-Dec-02 Hirasuna

No Peg Kota Gol Darah Jabatan

1001 Tsukuba A Staf 1002 Tsukuba B Staf 1003 Ami O Supervisor 1004 Tsukuba AB Supervisor …. …. …. …. 1005 Ami A Staf

SELECT NoPeg, Nama, Kota

FROM Pegawai

WHERE Nama = ‘Asep’

AND Kota = ‘Tsukuba’

SELECT NoPeg, Nama, Kota

FROM Pegawai

WHERE Nama = ‘Asep’

AND Kota = ‘Tsukuba’

1001 Asep

1003 Asep

1001 Tsukuba

1002 Tsukuba

1004 Tsukuba

Query Parallelism

• Intra-query parallelism

• Inter-query parallelism

SELECT NoPeg, Nama, Kota

FROM Pegawai

WHERE Nama = ‘Asep’

AND Kota = ‘Tsukuba’

SELECT NoPeg, Nama, Kota

FROM Pegawai

WHERE Nama = ‘Asep’

AND Kota = ‘Tsukuba’

SELECT NoPeg, Nama, Kota

FROM Pegawai

SELECT NoPeg, Nama, Kota

FROM Pegawai

SELECT NoPeg, Nama, Kota

FROM Pegawai

WHERE Nama = ‘Asep’

SELECT NoPeg, Nama, Kota

FROM Pegawai

WHERE Nama = ‘Asep’

Q1

Q1.1

(11)

Query Parallelism

• Intra-query parallelism

• Inter-query parallelism

SELECT NoPeg, Nama, Kota

FROM Pegawai

WHERE Nama = ‘Asep’

AND Kota = ‘Tsukuba’

SELECT NoPeg, Nama, Kota

FROM Pegawai

WHERE Nama = ‘Asep’

AND Kota = ‘Tsukuba’

SELECT DISTINCT Kota

FROM Pegawai

WHERE Nama = ‘Asep’

SELECT DISTINCT Kota

FROM Pegawai

WHERE Nama = ‘Asep’

Q1

Q2

(12)

XML

• Extensible Markup Language

• Memberi anotasi pada teks

• Data semi-terstruktur dimodelkan sebagai Tree

club

clubname

member

name

address

email

lname

fname

city

state

soccer

kita

nishi tsukuba IB

dj@co.jp

club

clubname

member

name

address

email

lname

fname

city

state

soccer

kita

nishi tsukuba IB

dj@co.jp

<club>

<clubname>soccer</clubname>

<member>

<name>

<lname>kita</lname>

<fname>nishi</fname>

</name>

<address>

<city>tsukuba</city>

<state>ibaraki</state>

</address>

</member>

</club>

XML Model

• Query memiliki pattern

• Relasi Parent-Child (/)

• Relasi Ancestor-Descendant (//)

• Mencari pattern dari query pada XML data.

club

clubname

member

name

address

email

lname

fname

city

state

member

name

address

lname

fname

city state

zip

soccer

kita

nishi tsukuba IB

dj@co.jp

minami higashi chiba CB 260-0000

club

clubname

member

name

address

email

lname

fname

city

state

member

name

address

lname

fname

city state

zip

soccer

kita

nishi tsukuba IB

dj@co.jp

minami higashi chiba CB 260-0000

Query

member

name

fname

city

kita

tsukuba

lname

nishi

member

name

fname

city

kita

tsukuba

lname

nishi

Query

member

name

fname

city

kita

tsukuba

lname

nishi

member

name

fname

city

kita

tsukuba

lname

nishi

(13)

Partisi Data

• Berbasis XML Tree

• General Graph

club

clubname

member

name

address

email

lname

fname

city

state

member

name

address

lname

fname

city state

zip

soccer

kita

nishi tsukuba IB

dj@co.jp

minami higashi chiba CB 260-0000

club

clubname

member

name

address

email

lname

fname

city

state

member

name

address

lname

fname

city state

zip

soccer

kita

nishi tsukuba IB

dj@co.jp

minami higashi chiba CB 260-0000

Query

member

name

fname

city

kita

tsukuba

lname

nishi

member

name

fname

city

kita

tsukuba

lname

nishi

Query

member

name

fname

city

kita

tsukuba

lname

nishi

member

name

fname

city

kita

tsukuba

lname

nishi

Kontribusi Riset

(14)

Pengolahan XML Query Secara Parallel

• Novelty – Kontribusi 1

• Cara baru mempartisi XML Tree

• Berbasis stream

• Konsep logis berbentuk Grid (kotak-kotak)

• Best Paper di Austria

• Novelty – Kontribusi 2

• Berbasis stream

• Propagasi partisi

• Best Paper di Fukushima

Grid Metadata Model for XML

D

p1

cf (doc1, p1) cf (doc2, q1)

p2

cf (doc1, p2) cf (doc2, p2)

p3

cf (doc2, p3)

p4

cf (doc2, p4)

p5

cf (doc2, p5)

p6

cf (doc2, p6)

p7

cf (doc3, p7)

doc1

doc2

doc3

P

cf (doc2, p1)

Grid Metadata for XML

Query

Paths

Query

Paths

p1

p2

p3

p4

p5

p6

p7

S

e

S

b

S

c

S

a

S

f

S

d

S

g

S

h

Streams of Elements

a1

b1 b2

c1 c2

d1 d2

e1 e2

e1 e2 e3

f1

g1 g2

g1 g2 g3

h1

doc1

doc2

doc3

q3

q1

q2

q4

Queries

b

c

d

g

c

d

f

e

d

g

h

e

cf (doc1, q1) cf (doc2, q1)

q1

doc1

a1 b1 c1 d1 b2 c2 d2

doc1

a1 b1 c1 d1 b2 c2 d2 a1 b1 c1 d1 b2 c2 d2

doc2

g1 b1 b2 g2 e1 e2 f1

doc2

g1 b1 b2 g2 e1 e2 f1 g1 b1 b2 g2 e1 e2 f1

doc3

h1 g1 e1 g2 e2 g3 e3

doc3

h1 g1 e1 g2 e2 g3 e3

doc1

a1 b1 c1 d1 b2 c2 d2

doc1

a1 b1 c1 d1 b2 c2 d2 a1 b1 c1 d1 b2 c2 d2

doc1

a1 b1 c1 d1 b2 c2 d2 a1 b1 c1 d1 b2 c2 d2

doc1

a1 b1 c1 d1 b2 c2 d2 a1 b1 c1 d1 b2 c2 d2

doc2

g1 b1 b2 g2 e1 e2 f1

doc2

g1 b1 b2 g2 e1 e2 f1 g1 b1 b2 g2 e1 e2 f1

doc2

g1 b1 b2 g2 e1 e2 f1 g1 b1 b2 g2 e1 e2 f1

doc2

g1 b1 b2 g2 e1 e2 f1 g1 b1 b2 g2 e1 e2 f1

doc3

h1 g1 e1 g2 e2 g3 e3

doc3

h1 g1 e1 g2 e2 g3 e3

doc3

h1 g1 e1 g2 e2 g3 e3

doc3

h1 g1 e1 g2 e2 g3 e3

(15)

Teknik Partisi

• Document Clustering

a1

S

a

b1 b2

S

b

c1 c2

S

c

d1 d2

S

d

e1 e2

e1 e2 e3

S

e

f1

S

f

g1 g2

g1 g2 g3

S

g

q4

cf (doc3, q4)

h1

S

h

doc1

doc2

doc3

cf (doc2, q2)

cf (doc2, q3)

q3

doc1

doc2

doc3

q1

q2

D

Q

Streams

Grid Metadata for XML

a1

S

a

b1 b2

S

b

c1 c2

S

c

d1 d2

S

d

e1 e2

e1 e2 e3

S

e

f1

S

f

g1 g2

g1 g2 g3

S

g

q4

cf (doc3, q4)

h1

S

h

doc1

doc2

doc3

cf (doc2, q2)

cf (doc2, q3)

q3

doc1

doc2

doc3

q1

q2

D

Q

a1

S

a

b1 b2

S

b

c1 c2

S

c

d1 d2

S

d

e1 e2

e1 e2 e3

S

e

f1

S

f

g1 g2

g1 g2 g3

S

g

q4

cf (doc3, q4)

h1

S

h

doc1

doc2

doc3

cf (doc2, q2)

cf (doc2, q3)

q3

doc1

doc2

doc3

q1

q2

a1

S

a

b1 b2

S

b

c1 c2

S

c

d1 d2

S

d

e1 e2

e1 e2 e3

S

e

f1

S

f

g1 g2

g1 g2 g3

S

g

q4

cf (doc3, q4)

h1

S

h

doc1

doc2

doc3

cf (doc2, q2)

cf (doc2, q3)

q3

doc1

doc2

doc3

q1

q2

D

Q

Streams

Grid Metadata for XML

similar

a1

S

a

b1 b2

S

b

c1 c2

S

c

d1 d2

S

d

e1 e2

e1 e2 e3

S

e

f1

S

f

g1 g2

g1 g2 g3

S

g

q4

cf (doc3, q4)

h1

S

h

doc1

doc2

doc3

cf (doc2, q2)

cf (doc2, q3)

q3

doc1

doc2

doc3

q1

q2

D

Q

Streams

Grid Metadata for XML

a1

S

a

b1 b2

S

b

c1 c2

S

c

d1 d2

S

d

e1 e2

e1 e2 e3

S

e

f1

S

f

g1 g2

g1 g2 g3

S

g

q4

cf (doc3, q4)

h1

S

h

doc1

doc2

doc3

cf (doc2, q2)

cf (doc2, q3)

q3

doc1

doc2

doc3

q1

q2

D

Q

a1

S

a

b1 b2

S

b

c1 c2

S

c

d1 d2

S

d

e1 e2

e1 e2 e3

S

e

f1

S

f

g1 g2

g1 g2 g3

S

g

q4

cf (doc3, q4)

h1

S

h

doc1

doc2

doc3

cf (doc2, q2)

cf (doc2, q3)

q3

doc1

doc2

doc3

q1

q2

a1

S

a

b1 b2

S

b

c1 c2

S

c

d1 d2

S

d

e1 e2

e1 e2 e3

S

e

f1

S

f

g1 g2

g1 g2 g3

S

g

q4

cf (doc3, q4)

h1

S

h

doc1

doc2

doc3

cf (doc2, q2)

cf (doc2, q3)

q3

doc1

doc2

doc3

q1

q2

D

Q

Streams

Grid Metadata for XML

similar

Teknik Partisi

• Query Clustering

a1 Sa b1 b2 b1 b2 Sb c1 c2 c1 c2 Sc d1 d2 d1 d2 Sd g1 g2 Sg d1 d2 d1 d2 Sd e1 e2 Se f1 Sf e1 e2 e3 Se g1 g2 g3 Sg h1 Sh

D

Q

Grid Metadata for XML

Streams

q4

cf (doc3, q4)

doc1

doc2

doc3

cf (doc1, q1) cf (doc2, q1) cf (doc2, q2) cf (doc2, q3)

q1

q2

q3

D

Q

Grid Metadata for XML

Streams

q4

cf (doc3, q4)

doc1

doc2

doc3

q1

q2

q3

similar

duplicate

D

Q

Grid Metadata for XML

Streams

q4

cf (doc3, q4)

doc1

doc2

doc3

q1

q2

q3

D

Q

Grid Metadata for XML

Streams

q4

cf (doc3, q4)

doc1

doc2

doc3

q1

q2

q3

similar

duplicate

D

Q

Grid Metadata for XML

Streams

q4

cf (doc3, q4)

doc1

doc2

doc3

q1

q2

q3

D

Q

Grid Metadata for XML

Streams

q4

cf (doc3, q4)

doc1

doc2

doc3

q1

q2

q3

similar

D

Q

Grid Metadata for XML

Streams

q4

cf (doc3, q4)

doc1

doc2

doc3

q1

q2

q3

D

Q

Grid Metadata for XML

Streams

q4

cf (doc3, q4)

doc1

doc2

doc3

q1

q2

q3

similar

duplicate

(16)

Stream-based Partitioning for XML

5

4

3

2

1

4

3

2

3

2

1 club

clubname

member

name

lname

fname

member

name

lname

soccer

kita

nishi

kita

(1, 1:55, 1) (1, 2:4, 2) (1, 3, 3) (1, 6:13, 3) (1, 8, 5) (1, 7:9, 4) (1, 11, 5) (1, 5:14, 2)

member

name

lname

fname

minami

nishi

member

name

fname

kita

member

name

lname

fname

higashi

(1, 10:12, 4) (1, 15:24, 2) (1, 16:23, 3) (1, 18, 5) (1, 17:19, 4) (1, 25:34, 2) (1, 26:33, 3) (1, 28, 5) (1, 27:29, 4) (1, 31, 5) (1, 30:32, 4) (1, 35:44, 2) (1, 36:43, 3) (1, 41, 5) (1, 40:42, 4) (1, 45:54, 2) (1, 46:53, 3) (1, 48, 5) (1, 47:49, 4) (1, 50:52, 4)