Deep Learning Menggunakan

(1)

Deep Learning Menggunakan

Ibnu Daqiqil ID

Doctoral Student

Graduate School of Interdisciplinary Science and Engineering in Health Systems

Okayama University

(2)

❑

Memberikan intuisi pengembangan Deep

Learning menggunakan PyTorch

(3)

Konsep Dasar

Neural Network Pengenalan Python dan PyTorch

Proses Learning _{Demo PyTorch di Google}

(4)

Konsep Dasar

Deep Learning

(5)

(6)

Penelitian si Barra

Warna Panjang Lebar 3cm 1.5cm 3cm 1.5cm 1.5cm 4 cm 1.8cm 3.5cm Panjang Le bar

(7)

Penelitian si Barra

Warna Panjang Lebar 2cm 0.5cm 3cm 1.5cm 1.5cm 4 cm 1.8cm 3.5cm 0.5 cm 2 cm 0.3 cm 2.5cm 1 cm 1.8cm Panjang Le bar

(8)

Tebakan si Aisha

Abang Barra, aisha punya bunga dengan Panjang kelopak 3.2 cm dan lebar kelopak 2cm. Coba tebak apa

warna bunganya???

Hmmm… gmn ya cara menebak warnanya????

(9)

Warna Panjang Lebar 3cm 1.5cm 1.5cm 4 cm 1.8cm 3.5cm 0.5 cm 2 cm 0.3 cm 2.5cm 1 cm 1.8cm

Plot Data Bunga Barra

3.2 cm 2 cm

Data Training Data Uji

Panjang

Le

(10)

Deep Learning

byIan Goodfellow, Yoshua Bengio, Aaron Courville

(11)

Deep Learning

(12)

Deep Learning

(13)

Rule Based vs ML vs DL

(14)

(15)

(16)

(17)

Komputasi Data

#cores

max clock

speed

memory

2 x Xeon CPUs

2 x 20

3.9 GHz

384 GB

4 x V100 GPUs

4 x 5120

1.455 GHz

4 x 32 GB

(18)

(19)

(20)

Loss

Target Data

Node Node Node

(21)

Loss

Target Data

Node Node Node

Bagaimana melakukan

penyesuaian input

jika output harus di

ubah?

(22)

Loss

Target Data

Node Node Node

Turunan terhadap input

Output

(23)

Fundamental

Neural

(24)

- Terkoneksi - Representasi komputasi sederhana Memiliki mekanisme transmisi data (

eksitasi &Inhibisi)

Memiliki state & Outputs spikes

Want to learn more?

Hodgkin AL, Huxley AF A quantitative

description of membrane current and its application to conduction and excitation in nerve. The Journal of

Physiology. 117 (4): 500–44. (1952)

Soma

Axon Dendrite

Diperkirakan otak manusia berisi sekitar

86,000,000,000 Neuron.

(25)

- Mudah untuk dibuat Merepresentasikan komputasi

sederhana

Merepresentasikan exitasi dan inhibisi

- Stateless

- Output berupa real

McCulloch, Warren S.; Pitts, Walter

A logical calculus of the ideas immanent in nervous activity

Bulletin of Mathematical Biophysics. 5 (4): 115–133. (1943)

“Soma”

“Axon” “Dendrite”

Tujuan utama model artiﬁcial neurons adalah untuk

merefleksikan beberapa neurophysiological observations, tetapi tidak mereproduksi

kedimanisannya

(26)

McCulloch, Warren S.; Pitts, Walter

A logical calculus of the ideas immanent in nervous activity

Bulletin of Mathematical Biophysics. 5 (4): 115–133. (1943)

- Mudah untuk dibuat Merepresentasikan komputasi

sederhana

Merepresentasikan exitasi dan inhibisi

- Stateless

- Output berupa real

Tujuan utama model artiﬁcial neurons adalah untuk

merefleksikan beberapa neurophysiological observations, tetapi tidak mereproduksi

kedimanisannya

(27)

- Mudah diimplementasi Merupakan gabungan dari artifisial neuron Mudah divektorisasi Dapat di optimaliasi dengan GPU/TPU

Jouppi, Norman P. et al. In-Datacenter

Performance Analysis of a Tensor Processing Unit™ 44th International

Symposium on Computer Architecture (ISCA) (2017)

Neurons pada layer biasa disebut units. Parameters biasa disebut weights.

(28)

Loss

Target Data

Node Linear

(29)

Loss

Target Data

Node Linear

(30)

- Menghadirkan non-linear behaviour - Output berupa

estimasi probabilitas - Memiliki turunan yang

sederhana - Saturates

- Derivatives vanish

Hinton G. Deep belief networks. Scholarpedia. 4 (5): 5947. (2009)

Activation functions menghasilkan non-linearities.

(31)

Loss

Target Data

Sigmoid Linear

(32)

- Biasanya digunakan untuk kasus klasifikasi - Encodes negation of logarithm of probability of correct classiﬁcation Composable with sigmoid - Numerically unstable

Murphy, Kevin Machine Learning: A Probabilistic Perspective (2012)

Cross entropy loss is also called negative log likelihood or logistic loss.

(33)

Encodes negation of logarithm of probability of entirely correct classiﬁcation Equivalent to logistic regression model Numerically unstable

Cramer, J. S. The origins of logistic regression (Technical report). 119. Tinbergen Institute. pp. 167–178(2002)

Cross entropy loss biasa disebut negative log likelihood atau logistic loss. Cross entropy Target Data Linear Sigmoid

(34)

- Multi-dimensional generalisation dari sigmoid - Digunakan untuk probability estimate - Memiliki turunan sederhana - Saturates - Derivatives vanish

Goodfellow, Ian; Bengio, Yoshua; Courville, Aaron Softmax Units for Multinoulli Output Distributions. Deep Learning. MIT Press. pp. 180–184. (2016)

Softmax biasa digunakan pada kasus classiﬁcation.

(35)

Mengencode negasi dari log probability dari klasifikasi Setara dengan multinomial logistic regression model Numerically stable combination

Martins, Andre, and Ramon Astudillo. From softmax to sparsemax: A sparse model of attention and multi-label classiﬁcation. International Conference on Machine Learning. (2016)

Dapat digunakan secara luas, tidak hanya diklasifikasi

classiﬁcation tetapi juga RL. Does not scale dengan jumlah k.

Cross entropy

Target Data

(36)

PyTorch memberikan seperangkat tool yang cepat, Fleksibel dan

Efisien melalui sebuah front-end yang user-friendly, distributed

training, dan sejumlah ecosystem yang berisi tool yang sangat

membantu produktifitas.

● Tensor computation (seperti numpy) dengan GPU acceleration

● Deep Neural Networks berdasarkan autograd system

http://pytorch.org/about/

END-TO-END

(37)

● Lebih Pythonic (imperative)

○ Flexible

○ Intuitive dan cleaner code

○ Easy to debug

● Lebih Neural Networkic

○ Write code as the network works

○ forward/backward

(38)

Perbandingan Code Numpy, TF dan PyTorch

(39)

Install

(40)

https://towardsdatascience.com/is-pytorch-catching-tensorflow-ca88f9128304

(41)

(42)

S I M P L E S T

N E U R A L N E T W O R K

BCE Target Data Sigmoid Linear

(43)

S I M P L E S T

N E U R A L N E T W O R K

(44)

S I M P L E S T

N E U R A L N E T W O R K

(45)

(46)

Linear Softmax

Linear Sigmoid Cross Entropy

(47)

Linear Softmax

(48)

Linear Softmax

(49)

Linear Softmax

(50)

Linear Softmax

(51)

Linear Softmax

(52)

Linear Softmax

(53)

Dengan

menggunakan 1 hidden layer dapat memecahkanXOR Hidden layer

memungkinkan kita untuk memutar dan membengkokkan input space

Blum, E. K. Approximation of Boolean functions by sigmoidal networks: Part I: XOR and other two-variable functions Neural computation 1.4 532-540. (1989)

Linear Sigmoid Linear Softmax Cross Entropy

Target Data

Hidden layer memungkinkan kita untuk melakukannon-linear input space

transformation sehingga pada layer yang terakhir

(54)

(55)

(56)

Linear Node Linear Node

Data

Linear Node Loss

Target Linear Node

(57)

- Introduces non-linear behaviour - Creates piecewise linear functions - Derivatives do not vanish

- Dead neurons can occur

- Technically not differentiable at 0

Hahnloser, R.; Sarpeshkar, R.; Mahowald, M. A.; Douglas, R. J.; Seung, H. S. Digital selection and analogue ampliﬁcation coexist in a cortex-inspired silicon circuit. Nature. 405: 947–951 (2000)

Salah satu fungsi yang paling umum digunakan.

(58)

Lines/corners detection Object detection Linear ReLU Shapes detection Linear ReLU Data

Linear ReLU Loss

Target

Class detection

(59)

- Expressing

symmetries and regularities is much easier with deep model than wide one. Deep model means many non-linear

composition and thus harder learning

Guido Montúfar, Razvan Pascanu, Kyunghyun Cho, Yoshua Bengio. On the Number of Linear Regions of Deep Neural Networks Arxiv (2014)

Jumlah linear regions bertambah secara exponentially dengan depth, dan polynomially polynomial dengan

(60)

Linear ReLU Linear ReLU

Data

Linear ReLU Cross

Entropy

Target Linear Sotfmax

(61)

(62)

(63)

(64)

(65)

(66)

Proses

(67)

Linear algebrarecap

(68)

Kingma, Diederik P., and Jimmy Ba.Adam: A method for stochastic optimization arXiv preprint arXiv:1412.6980 (2014).

Pemilihan metode oprimalisasi = critical. Contoh: SGD, Adam, RMSProp,...

(69)

Forward pass

(70)

Forward pass

(71)

Abadi, Martín, et al. Tensorﬂow: A system for large-scale machine learning. 12th Symposium on Operating Systems Design and Implementation (2016)

(72)

Abadi, Martín, et al. Tensorﬂow: A system for large-scale machine learning. 12th Symposium on Operating Systems Design and Implementation (2016)

(73)

(74)

(75)

(76)

(77)

(78)

PyTorch membantu peneliti untuk focus kepada pengembangan arsitektor NN

(79)

(80)

(81)

(82)

(83)