CS4803/7643: Deep Learning Problem Set 4 Solution

• Optimal Policy and Value Function [15 points]

Consider a simple 2-state MDP shown in Figure 1, S = fS₁; S₂g. From each state, there are two available actions A = fstay; gog. The reward received for taking an action at a state is shown in the figure for each (s; a) pair. Also, taking action go from state S₂ ends the episode. All transitions are deterministic.

Figure 1: 2-state MDP

(a) Consider a policy that always chooses the stay action at every state. Compute the sum of discounted rewards obtained by this policy starting at state S₁, assuming a discount of i.e.

1

compute ^{P t}r_t(s_t; a_t). [4 points]

t=0

1. What is the optimal policy? Write it as a tuple (a₁; a₂) of the optimal actions at (s₁; s₂) respectively. Compute the sum of discounted rewards obtained by this policy, given that the start state is S₁, with discount . [4 points]

1. Recall that one update of the value iteration algorithm performs the following operation, where r(s; a) denotes the reward for taking action a at state s.

For each s 2 S :





V ⁱ⁺¹(s) max ^X p(s⁰js; a) r(s; a) + V ⁱ(s⁰)

(1)


a





s⁰

Note: Since taking action go at s₂ will terminate the episode, the value of taking the go action at s₂ is just r(s₂; go) instead of r(s₂; go) + V (s₁) (since you don’t care about future values after the episode has ended). Thus the update for state V (s₂) will look like:

V ⁱ⁺¹(s₂) max r(s₂; stay) + V ⁱ(s₂); r(s₂; go)

(2)

The value function V is stored as an array of size jSj ( jSj = 2 in our case). For example, the array V ⁰ = [0; 1] denotes that V ⁰(s₁) = 0; V ⁰(s₂) = 1. Starting with a initial V ⁰ = [0; 0], peform the value iteration update to compute V 1; V 2; V 3 assuming a discount factor = 1. What is the optimal V ? [7 points]

• Value Iteration Convergence [25 points + 5 bonus points]

In part (c) of the previous question, we computed V i for a few updates of value iteration and also V , the optimal value function. What happened to the "error" in the value estimate at every update i.e. did jV i(s) V (s)j every increase for a particular state s? The answer is yes, the value estimate

2

^x2k₁,

for any s may fluctuate before eventually converging to V (s). If this is the case, do we have any hope that the value iteration algorithm will converge to V ? In this question, we will prove that value iteration indeed converges due to the monotonic decrease in a different error - the maximum

error over all states

s2S j

j or the

1 norm

V i

1.

max V i(s)

V

(s)

L

V

(a)

For V 1; V 2; V 3 obtained in 1(c), compute

V i

V

monotonically. [5 points]

1 and verify that this error decreased

1. Let T : R^jSj ! R^jSj be the function that takes the vector V i as input and applies the value iteration update to produce V i+1. We will show that for any two vectors V; V ⁰ 2 R^jSj, this operator decreases the L1 norm between them.

Prove that: kT(V ) T(V ⁰)k₁ kV V ⁰k₁ for any V; V ⁰ 2 R^jSj, [10 points]

1. Now consider the sequence of value functions fV ng obtained by iteratively performing the value iteration update as per Eq. 1. This sequence has the interesting property of being a cauchy sequence i.e. the elements of the sequence become arbitrarily close to each other as the

sequence progresses. Formally, we say a sequence is cauchy w.r.t. the L¹ norm if for every positive number , there is a positive integer N s.t. for all positive integers m; n greater than N, kx_m x_nk₁ < . Equipped with this fact, show that the error of V ⁿ⁺¹ w.r.t the optimal value function can be bounded as: 8 > 0; 9N > 0 s.t. 8n > N

_V n+1

_V

1

1

;










(3)


[10 points]

























































































, then apply the Cauchy


Hint: First try to prove that

_V n+1

V




V ⁿ

_V n+1





sequence condition on V n













₁




1
















1

Discussion: Given Equation 3, we now have a guarantee that value iteration converges to V . However, we made the assumption that fV ng is a Cauchy sequence. As a bonus question below, we will prove that the condition in (a) is sufficient to conclude that fV ng is indeed a cauchy sequence.

(d) [BONUS] Given a function T : Rⁿ ! Rⁿ such that kT(x₁) T(x₂)k₁ kx₁

2 [0; 1), prove first that T has a fixed point i.e. 9x : T(x ) = x and second that the fixed point of T is unique. [5 bonus points]

• Learning the Model. [20 bonus points]

This is a challenging question with all sub-parts as bonus credit.

While value iteration (VI) allows us to find the optimal policy, performing VI updates (Eq. 1) requires the transition function T(s; a) := p(s⁰ j s; a) ¹ and the reward function R(s; a). In many practical situations, these quantities are unknown. However, we can step through the environment and observe transitions to collect data of the form (s; a; s⁰; r). Using this data, if we can obtain an estimate of the transition and reward functions, we can hope to still find the optimal policy via value iteration. Let M denote the true (unknown to us) model of the world (a model consists of

¹we will denote T(s; a; s⁰) as the probability value of s⁰ give (s; a), whereas T(s; a) denotes the entire probability distribution p(s⁰ j s; a)

both a transition and reward function), and let ^{^} represent our approximate model of the world M

that we estimate from data.

In this question, given estimates of both the transition function T(s; a) = p(s⁰ j s; a) 2 ^{jSj 1} and the reward function R(s; a), we want to study the error of acting optimally in this estimated MDP as opposed to acting optimally in the “true” MDP. Naturally, we should expect this error to be smaller as we get better estimates which in turn is proportional to how many observations we make.

(a) [BONUS] For the first part of this problem, assume that we have used our favorite learning

^

^


method to obtain estimates of the transition function T and the reward function R. Given


^

^

T(s; a)jj_{1 P} , then for any policy


that max_s;a jR(s; a) R

(s; a)j _R and max_s;a jjT(s; a)

: S ! A; 8s 2 S, show that [10 bonus points]

jjV_M^ V_M jj<sub>1

R</sub>




+

_P

(4)


1




(1 )²

Assume that the reward function is in [0; 1].

Hint: 1) Expand V_M (s) using the Bellman equation 2) Holder’s Inequality: jju vjj₁ jjujj₁ jjvjj₁

(b) [BONUS] We just bounded the error of acting according to any policy : S ! A under the

^







. Now, we use this to bound the value error in


estimated model, M vs. the true model, M


acting optimally vs. acting optimally according to the estimated model ( _^ ).




















M





Using (4), show that [3 bonus points]

















V (s)

_^




2 sup

<sub>V

V</sub>

(5)


V ^M (s)


M

M

jj

M^{^}

M jj1











:S!A

(c) [BONUS] Given

R

r











































(6)








2n
















jS























=

1

log

4







Aj



















































_P

= jSj ^r

























(7)





_2n log

4

jS

A Sj























1

use (4) to simplify (5) i.e. V_M





(s) V_MM (s). What is the dependence on this error and number

of observations required for each state action pair, n? [2 bonus points]

1. [BONUS] Given a dataset D_s;a = f(s; a; s⁰₁; r₁); : : : (s; a; s⁰_n; r_n)g we estimate the unknown transition and reward function using empirical frequencies as:

^{^}(s; a; s⁰) =

1




n

1(s⁰ = s⁰)








^Xi

(8)


T

n

i









=1
























^

1




n








R(s; a) = _n

X_i

r

(9)


=1

4

For notational convenience, we will use ^{^} to denote a vector of size j j whose element is

T(s; a) S

the probability computed in (8) for each next state, s⁰ 2 S. Prove that _R and _P take values as in (6) and (7). [5 bonus points]

Hint: Hoeffding’s inequality2

• Policy Gradients Variance Reduction [10 points]

In class (will be covered on Oct 29th, but you don’t really need to know the derivation for this question), we derived the policy gradient as

r J( ) = r E [R( )]

(10)


= E [R( )r log ( )]

(11)


1

N











X_i

(12)





N

R( _i)r log ( _i)








=1

where is a trajectory3, R( ), the associated reward and N, the number of trials to estimate the expected reward, J( ), the expected reward and , the parameters of the policy. These gradients are often very noisy and lead to unstable training. In this question, we show a simple method to reduce the variance in the gradients.

(a) Show that adjusting the reward as R( ) := R( ) b does not change the gradient estimate in Eq. 10. [2 points]

1. Compute the variance of r J( ) and notice how subtracting b helped reduce it. What value of b will lead to the least variance? [8 points]

Hint: The variance of J( ) is written as V ar(x) = E[x²] (E[x])² where x is substituted as R( )r log ( )

• Paper Review: [4 points]

In this course’s final paper review section, we examine an interesting project that upends con-ventional research in reinforcement learning. While most RL problems are aimed at maximizing rewards, this work proposes ’upside down reinforcement learning’, where the objective is remodelled as a supervised learning problem. Limited experiments demonstrate promising and competitive results.

The paper can be accessed here.

As in the previous assignments, please limit your reviews to 350 words.

Briefly summarize the key findings, strengths and potential limitations of this work.

• Coding: Dynamic Programming and Deep Q-Learning [60 points + 5 bonus points]

Follow the instructions at this link to the HW4 coding webpage.

• https://en.wikipedia.org/wiki/Hoeffding%27s_inequality#General_case_of_bounded_random_variables

³A trajectory is defined as a sequence of states and actions i.e. (s₀; a₀; s₁; a₁; : : : s_{n 1}; a_{n 1}; s_n)

5