AML.tt#t2o9 - Reinforcement learning

AML.tt#t2o9

Reinforcement learning

Agent

^{chooses actions ,}

get

^a

( probabilistic)

^reward

Determine ^a

strategy

^{to choose aehnis to maximize}

longterm

^reward

Exploitation

^vs

Exploration

I

b

choosing greedily

^{Learn more about}

system

Non^- associative model

Rewards ^{are not based on current state}

(

^ie.

only

one state

)

-

Possibility of

^time

varying

^rewards

k^{- bandit}

problem

choose

among

^{k actions}

Associative

^model

Markov

^Decision

Process

Markov

^chain _Sr

. - Sj ^{- - S.}

States

I

Transition

probability

^{matrix si}

[ Pij ]

Pg

^:

prob of

^{si →}

Sj

§ Pig

^{-- 1}

Finite

state

,

one next state distribution

per

^state

Generator

At S_, choose ^{a -}

Agent

^{'s choice}

\

Associated probability pls !

^r

f

^s,

a)

=?

s -9 ^ru

Sn

pls

^'

Is

^,

a)

⁼

Er pls

^',

rls

^,

a)

§

^,

pls

^'

Is ,a7

^=L

Expected

^reward

r

(

^{s,a, s '}

)

^--

g

^{r -}

Pga pls

^'_Is

)

,

a)

Idealized setting for

^'' associative^"

reinforcement

learning

"

State information

^is

available

^to

agent

What

^{does it mean}

to

maximise

long

^{term reward}

?

Finite

trajectory

-

Play

^{a game till it ends in win}

floss (

^draw

- T ^moves

,

t ^-- r _. ^{. -} ,T

I

Expected

^{return at time f}

"

Episode

^"

At

⁼

Rt

# trttzt ^{- - - t}

RT t

action At ^{at t}

generates

^reward

Rtt

_,

Cst

^,

they

^→

Stat

, Reti

Many

^{situations - no}

finite episode boundary Infinite sequence of

^rewards

Discounted ^reward

Gt defy Rtt

^{, t 8}

Rttz tf

^{Retz t - - -}

O # El

=

E 8k Rttkti

k=o

T ^--O ^{- "}

myopic

^{" r→1}

GE

^{= Rtt I t r Retz t 82 Retz t - -}

= Rtt , t 8

(

^{Retz tr}

Retz trktth

^--

)

-

htt

GE

^--

Reti

^t

VA

Eti

tf

^{R -- ti}

always

a

⁼

I

I

^-r

To

_reconcile

fmte

^&

infinite

^case

-

Episodic

^{tasks , assume that}

- terminal ^state have ^a

single self loop

- Reward ^{is O}

for self loop

-

D @

^r=o

Use

_same

discounted

^reward

defn

^{in both}

cases

Actual

^"value"

depends

^on

policy

Vets )

^{- value}

of

^{state s art}

policy

^to

Eta @

^t

Ist

^{-- s}

]

^--

Ee [ E. orkrttun Is

State

^-value

function

Eels

^,

a) Et Ea [ at Isles

^,

At =D

Vela

^--

Gillet

^sees

]

=

fit [ Rent Thet

^,

I

^Sees

]

-

→ -

ay.sk#a.--Eaitlals7sE.EpCstrls.a7f

O

rifle

^'

[ rtrEEGttilstt.es 'D

si

^.- se^'

vacs )

^-_

Eautalssrpcsirls

^,

a) frtrritcs 'D

vacs )

⁼

Ea Hals ) sq

^,

P' Girls

^,

a) ft

^r

re GD ]

Bellman

equation for

^'

hots )

Find

a

solution to

^these

( simultaneous ) equations

-

Given

^a

policy

^{, we have Bellman}

equations

^to

solve

for

^ht

LD

What _we

really

^{want to do is to}

find

^"

best

^{" I}

Best ^it

?

IT, Eltz

if VI

^-

Ve

^,

(

^s

)

^E

Vitals )

There

_exist

optimal policies

Suppose

^{it, ik are both}

optimal

-

Hs

^.

Yo

_, ⁼

Vitals )

-

Vrs

_,a

gig

_,

G.

^{a) =}

Eads

^,

a)

Hypothetically

^{, there are}

optimum 4×67

^,

9*6

^.

a)

We must have

¥67

⁼

maax

^{9 *}

(

^s,

a)

=

imaax

^E

[ at 1 Sees

^{, At}

=D

=

maax

^E

[ Rtt

^{t r}

Gotti 1st

^--

site

^--

D

=

may E [ Rtt

ⁱ

try (

^Seti

) ( St

^.-as,

AED

vids )

^--

may § pls

^'

.ir/s.a)frtrrv*Ls 'D

✓_*

G)

^--

maax fr pls ! rls

^,

a) [

^rtrrv*

Cs 'D

±

In ^vs

vacs )

⁼

Ea Hals ) sq

^,

P' Girls

^,

a) (

^{r t rreCs}

'D

-

Similarly

9.

^*

Is

^,

a)

⁼

fr pls

^',

rls

^,

a) fr

^to

maay 9*61

^a

'D