$29
Homework 05 Solution
Q0 (0pts correct answer, -1,000pts incorrect answer: (0,-1,000) pts): A correct answer to the following questions is worth 0pts. An incorrect answer is worth -1,000pts, which carries over to other homeworks and exams, and can result in an F grade in the course.
(1) Student interaction with other students / individuals:
(a) I have copied part of my homework from another student or another person (plagiarism).
(b) Yes, I discussed the homework with another person but came up with my own answers. Their name(s) is (are)
(c) No, I did not discuss the homework with anyone
(2) On using online resources:
(a) I have copied one of my answers directly from a website (plagiarism).
(b) I have used online resources to help me answer this question, but I came up with my own answers (you are allowed to use online resources as long as the answer is your own). Here is a list of the websites I have used in this homework:
(c) I have not used any online resources except the ones provided in the course website.
1
Homework 5
Q1: Theoretical Questions (2pts)
1. (1pt) Detail how we can replace the equivariant representation of the Transformer architecture with an
~ ~
equivariant representation using Janossy pooling and an LSTM architecture as f (see f in Equation (1) in our set representation lecture). Note that Janossy pooling originally gives invariant representations.
Speci cally, replace the representation of the i-th word of the m-th self-attention head, zi(m), of our Transformer lecture with the corresponding LSTM output using Janossy pooling.
Hint: Make sure your representation is equivariant. You will probably want to use the ht representation of the LSTM for your task (see Backpropagation-through-time lecture where we de ne the LSTM variables).
2. (1pt) (a) Explain one of the main shortcomings of RNNs and how Bi-directional RNNs help ameliorate this shortcoming. (b) Are Bi-directional LSTMs able to represent well dependencies between the rst and the last elements of a sequence? Why or why not?
Q2: Word Embeddings using 2nd order Markov Chains (2pts)
In class, we have seen that a word2vec model will de ne word embeddings using the SKIPGRAM model with window K (assumed to be an odd number) as a 1st order Markov chain. The negative log-likelihood (NLL) is (not accounting for the corner cases at the end and beginning of a sentence):
• k (K 1)=2
Xt
X
Lword2vec =
(log p(xt kjxt; U; V) + log p(xt+kjxt; U; V));
=k
k=1
and the probability function is
1
p(xjxt; U; V) = Z(xt; U; V) exp(hUx; Vxti);
with appropriately-de ned one-hot encoding of xi, i = 1; : : : ; n, and Z(xt; U; V) is a normalization function.
Task (2pts): Write down the NLL loss function and the probability function of a word embedding model that, again using a window of size K similar to the original SKIPGRAM approach, can embed words based on a 2nd order Markov chain.
Hint 1: We must change the original SKIPGRAM model since the dataset it creates is for a 1st order Markov chain.
Hint 2: There are multiple ways to de ne the model p. Note that you will need more parameters. Also pay attention that your 2nd order Markov chain that cannot be described by a 1st order Markov chain.
2
Q3: Domain Adaptation with Maximum Mean Discrepancy (MMD) (6pts)
In this homework, you are faced with the problem of unsupervised domain adaptation. You are provided a data set of labeled images (CIFAR-10), which we’ll call the source domain. Your task is to predict the labels for a target domain, in which the images have their color shifted. See an example in Figure 1.
0
10
20
30
40
50
60
0
50
100
150
200
250
300
Figure 1: Example of images. Top row: Source domain, CIFAR-100. Bottom row: Target domain, shifted hue.
To aid you in this task, you also have access to unlabeled images from the target domain to use during training. In a more formal description:
Training data: D(tr) = f(x(source)i; yi(source))gmi=1 [ fx(target)igni=1, where:
{ (x(source)i; yi(source)) are sampled iid from p(source)(x)p(yjx)
{ x(target)i are sampled iid from p(target)(x)
Test data: D(te) = f(x(target)i; yi(target))gni=1(te) , where:
{ (xi; yi) are sampled iid from p(target)(x)p(yjx).
For this homework, we will use the Maximum Mean Discrepancy (MMD) divergence to help us learn rep-resentations which are useful in both domains. Denote by f( x; W ); Wf ) our neural network classi er, where x; W ) is the part of the neural network which is tasked with learning the representations and f( ; Wf ) is the part of the neural network which is tasked on classifying an image given its representation.
We modify our loss function to incorporate the MMD as a regularization term:
L(D(tr); Wf ; W ) = LCE(D(tr); Wf ; W ) + MMD2(D(tr); W ); where LCE is the usual cross-entropy loss and is a hyperparameter.
We will be implementing MMD with the Gaussian kernel:
0k2
;
k(z; z0) = exp kz
2 2
z
2
where is a hyperparameter of the kernel. We will apply the MMD to the representations computed by ; W ) for both domains, with the goal of learning a representation that is invariant to both domains. To
compute the MMD term, we will use the unbiased estimator of Gretton et al. [2012] (Eq. 3, Lemma 6).
3
Action Items:
In your report, include the answer of the following questions:
1. (1 pt) Is this problem an example of covariate shift or label shift? Why? (describe through equations)
2. (1 pt) In your report, write the MMD equation below, replacing the question marks by the appropriate variables. Assume a batch of m examples from the source domain and a batch of n examples from the target domain. Your answer must use the variables variables x(source); x(target); m; n.
\2
(tr)
;W )=
MMD (D
1
? ?
=
X X
k(
i; W );
j; W ))
?(?
1)
i=1 j6=i
1
?
?
XX
+
k(
i; W );
j; W ))
?(?
1)
i=1 j6=i
2
?
?
X Xj
??
k(
i; W ); j; W )):
i=1 =1
3. (4 pts) Implement the MMD estimator in the le mmd.py and:
(a) (2 pts) Run the code without using MMD, through the command python hw5.py --reg_str 0. Include in your report:
i. Curve of training loss for each epoch.
ii. Curve of validation accuracy for each epoch, for both source and target domain.
iii. Final test accuracy of both source and target domain, for the best model.
(b) (2 pts) Run the code with MMD, through the command python hw5.py --reg_str 10 --kernel_sigma 20. Include in your report:
i. Curve of training loss for each epoch.
ii. Curve of validation accuracy for each epoch, for both source and target domain.
iii. Final test accuracy of both source and target domain, for the best model.
Hint: If you have GPUs available, pass the option --gpu 0 to use GPU 0. The code will run much faster on a GPU. The scholar cluster has GPUs, see the Piazza post on how to use it.
References
A. Gretton, K. M. Borgwardt, M. J. Rasch, B. Scholkopf, and A. Smola. A kernel two-sample test. Jour-nal of Machine Learning Research, 13(Mar):723{773, 2012. URL http://www.jmlr.org/papers/v13/ gretton12a.html.
4
Submission Instructions
Please read the instructions carefully. Failing to follow the instructions might lead to loss of points.
Naming convention: [your purdue login] hw-5
All your submission les, including a ReadMe, and codes, should be included in one folder. The folder should be named with the above naming convention. For example, if my purdue account is \jsmith123", then for Homework 5 I should name my folder as \jsmith123 hw-5".
Remove any unnecessary les in your folder, such as training datasets (Data folder). Make sure your folder is structured as the tree shown in Overview section.
Submit: TURNIN INSTRUCTIONS
Please submit your homework les on data.cs.purdue.edu using the turnin command, e.g.:
turnin -c cs690-dpl -p hw-5 jsmith123 hw-5.
Please make sure you didn’t use any library/source explicitly forbidden to use. If any such library/-source code is used, you will get 0 pt for the coding part of the assignment. If your code doesn’t run on scholar.rcac.purdue.edu, then even if it compiles in another computer, your code will still be considered not-running and the respective part of the assignment will receive 0 pt.
5
