[SPARK-28895][K8S] Support defining HADOOP_CONF_DIR and config map at the same time #25609

yaooqinn · 2019-08-28T10:22:06Z

What changes were proposed in this pull request?

Changes in this pull request will support users to define HADOOP_CONF_DIR and spark.kubernetes.hadoop.configMapName at the same time. When both of them are defined, Spark will take precedence over the config map to be mounted on the driver pod. This enables the spark client process to communicate any Hadoop cluster if it needs.

Why are the changes needed?

The BasicDriverFeatureStep for Spark on Kubernetes will upload the files/jars specified by --files/–jars to a Hadoop compatible file system configured by spark.kubernetes.file.upload.path. While using HADOOP_CONF_DIR, the spark-submit process can recognize the file system, but when using spark.kubernetes.hadoop.configMapName which only will be mount on the Pods not applied back to our client process.

 Kent@KentsMacBookPro  ~/Documents/spark-on-k8s/spark-3.0.0-SNAPSHOT-bin-2.7.3  bin/spark-submit --conf spark.kubernetes.file.upload.path=hdfs://hz-cluster10/user/kyuubi/udf --jars /Users/Kent/Documents/spark-on-k8s/spark-3.0.0-SNAPSHOT-bin-2.7.3/hadoop-lzo-0.4.20-SNAPSHOT.jar --conf spark.kerberos.keytab=/Users/Kent/Downloads/kyuubi.keytab --conf spark.kerberos.principal=kyuubi/dev@HADOOP.HZ.NETEASE.COM --conf  spark.kubernetes.kerberos.krb5.path=/etc/krb5.conf  --name hehe --deploy-mode cluster --class org.apache.spark.examples.HdfsTest   local:///opt/spark/examples/jars/spark-examples_2.12-3.0.0-SNAPSHOT.jar hdfs://hz-cluster10/user/kyuubi/hive_db/kyuubi.db/hive_tbl
Listening for transport dt_socket at address: 50014
# spark.master=k8s://https://10.120.238.100:7443
19/08/27 17:21:06 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
19/08/27 17:21:07 INFO SparkKubernetesClientFactory: Auto-configuring K8S client using current context from users K8S config file
Listening for transport dt_socket at address: 50014
Exception in thread "main" org.apache.spark.SparkException: Uploading file /Users/Kent/Documents/spark-on-k8s/spark-3.0.0-SNAPSHOT-bin-2.7.3/hadoop-lzo-0.4.20-SNAPSHOT.jar failed...
	at org.apache.spark.deploy.k8s.KubernetesUtils$.uploadFileUri(KubernetesUtils.scala:287)
	at org.apache.spark.deploy.k8s.KubernetesUtils$.$anonfun$uploadAndTransformFileUris$1(KubernetesUtils.scala:246)
	at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:237)
	at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
	at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
	at scala.collection.TraversableLike.map(TraversableLike.scala:237)
	at scala.collection.TraversableLike.map$(TraversableLike.scala:230)
	at scala.collection.AbstractTraversable.map(Traversable.scala:108)
	at org.apache.spark.deploy.k8s.KubernetesUtils$.uploadAndTransformFileUris(KubernetesUtils.scala:245)
	at org.apache.spark.deploy.k8s.features.BasicDriverFeatureStep.$anonfun$getAdditionalPodSystemProperties$1(BasicDriverFeatur# spark.master=k8s://https://10.120.238.100:7443
eStep.scala:165)
	at scala.collection.immutable.List.foreach(List.scala:392)
	at org.apache.spark.deploy.k8s.features.BasicDriverFeatureStep.getAdditionalPodSystemProperties(BasicDriverFeatureStep.scala:163)
	at org.apache.spark.deploy.k8s.submit.KubernetesDriverBuilder.$anonfun$buildFromFeatures$3(KubernetesDriverBuilder.scala:60)
	at scala.collection.LinearSeqOptimized.foldLeft(LinearSeqOptimized.scala:126)
	at scala.collection.LinearSeqOptimized.foldLeft$(LinearSeqOptimized.scala:122)
	at scala.collection.immutable.List.foldLeft(List.scala:89)
	at org.apache.spark.deploy.k8s.submit.KubernetesDriverBuilder.buildFromFeatures(KubernetesDriverBuilder.scala:58)
	at org.apache.spark.deploy.k8s.submit.Client.run(KubernetesClientApplication.scala:101)
	at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.$anonfun$run$10(KubernetesClientApplication.scala:236)
	at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.$anonfun$run$10$adapted(KubernetesClientApplication.scala:229)
	at org.apache.spark.util.Utils$.tryWithResource(Utils.scala:2567)
	at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.run(KubernetesClientApplication.scala:229)
	at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.start(KubernetesClientApplication.scala:198)
	at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:920)
	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:179)
	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:202)
	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:89)
	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:999)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1008)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.IllegalArgumentException: java.net.UnknownHostException: hz-cluster10
	at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:378)
	at org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:310)
	at org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:176)
	at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:678)
	at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:619)
	at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:149)
	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2669)
	at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94)
	at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2703)
	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2685)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:373)
	at org.apache.spark.util.Utils$.getHadoopFileSystem(Utils.scala:1881)
	at org.apache.spark.deploy.k8s.KubernetesUtils$.uploadFileUri(KubernetesUtils.scala:278)
	... 30 more
Caused by: java.net.UnknownHostException: hz-cluster10
	... 43 more

Other related spark configurations

spark.master=k8s://https://10.120.238.100:7443
# spark.master=k8s://https://10.120.238.253:7443
spark.kubernetes.container.image=harbor-inner.sparkonk8s.netease.com/tenant1-project1/spark:v3.0.0-20190813
# spark.kubernetes.driver.container.image=harbor-inner.sparkonk8s.netease.com/tenant1-project1/spark:v3.0.0-20190813
# spark.kubernetes.executor.container.image=harbor-inner.sparkonk8s.netease.com/tenant1-project1/spark:v3.0.0-20190813
spark.executor.instanses=5
spark.kubernetes.namespace=ns1
spark.kubernetes.container.image.pullSecrets=mysecret
spark.kubernetes.hadoop.configMapName=hz10-hadoop-dir
spark.kubernetes.kerberos.krb5.path=/etc/krb5.conf
spark.kerberos.principal=kyuubi/dev@HADOOP.HZ.NETEASE.COM
spark.kerberos.keytab=/Users/Kent/Downloads/kyuubi.keytab

Does this PR introduce any user-facing change?

I guess this pr will now use config map of Hadoop from k8s cluster to the local client process if there are files to upload to Hadoop.

How was this patch tested?

manually tested with spark + k8s cluster + standalone kerberized hdfs cluster
add an unit test

…if the client process has files to upload

SparkQA · 2019-08-28T10:28:05Z

Test build #109859 has finished for PR 25609 at commit ad10f94.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-08-28T10:36:14Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/14894/

SparkQA · 2019-08-28T10:43:57Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/14894/

SparkQA · 2019-08-28T11:28:47Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/14897/

SparkQA · 2019-08-28T11:37:00Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/14897/

SparkQA · 2019-08-28T12:17:52Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/14898/

SparkQA · 2019-08-28T12:35:31Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/14898/

SparkQA · 2019-08-28T13:07:25Z

Test build #109861 has finished for PR 25609 at commit 528dfc9.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-08-28T14:32:31Z

Test build #109862 has finished for PR 25609 at commit b31d487.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

skonto · 2019-09-03T12:47:26Z

...ce-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesUtils.scala

+        // should download the configmap to our client side.
+        // 1. add configurations from k8s configmap to hadoopConf
+        conf.get(KUBERNETES_HADOOP_CONF_CONFIG_MAP).foreach { cm =>
+          val hadoopConfFiles = client.configMaps().withName(cm).get().getData.asScala


@yaooqinn This happens at submission time at the launcher machine, it is weird to fetch the configmap from the cluster locally and not the way to go in my opinion. You could just point to the right hadoop config and spark submit will pick it up. spark.kubernetes.hadoop.configMapName was meant to be used at the driver pod and so that the hadoop files can be mounted on the fly within the cluster. Even if you launch cluster mode in the cluster you can do the same, mount the configmap and point to the files via the HADOOP_CONF_CONFIG var. @erikerlandson @dongjoon-hyun @ifilonenko fyi.
Btw configamaps are namespaced.

spark.kubernetes.hadoop.configMapName has no driver word in its name, so I guess it is ok to be used all over the spark application including the client process.

Using spark.kubernetes.hadoop.configMapName and HADOOP_CONF_DIR is an either-or thing, check here. It is more like that they both contain the same thing and are used by client to define the driver pod, so they should be equal. For now, the only difference between them is that the HADOOP_CONF_DIR is in the classpath of client process and the other not. With spark.kubernetes.hadoop.configMapName, if our application has no extra file (--jars/--files) to upload, it works. But once it does need one or more, it fails. I guess this behavior for spark.kubernetes.hadoop.configMapName is kind of unacceptable.

I see that they are either-or (extraneous is meant for the cluster deployment not this new feature), but I think if you specify them both (modify the code in there to allow both of them to be defined not having either-or and select according to user's preference or use the pod template feature to emulate the configmap mounting) it should work as spark submit is supposed to pick up the hadoop credentials (eg.

spark/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala

Line 700 in a950570

if ((clusterManager == MESOS || clusterManager == KUBERNETES)

). Initially config map was not meant for uploading files from the client machine (in general accessing hadoop from there) so the logic may need to be modified to play well with HADOOP_CONF_DIR, but I do find fetching the configmap from the cluster redundant. If you can download the configuration you could just add it at spark submit time at the client machine (it is not safer or anything if you fetch it afaik)?

@ifilonenko thoughts here? Do you think we can get the kerberos integration tests PR merged? That is the only viable way to make sure things are stable in the future.

If they are both specified, the main corner case seems like the scenario where they aren't consistent, but this could be checked for.

I agree with setting them both, which is my original idea too.

@skonto ill make sure to get that PR resolved so that we can use it for testing, yeah

skonto · 2019-09-03T15:18:42Z

...ce-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesUtils.scala

+        // 2. add configurations from arguments or spark properties file to hadoopConf
+        SparkHadoopUtil.appendS3AndSparkHadoopConfigurations(conf, hadoopConf)
+        // 3. set or rest user group information
+        UserGroupInformation.setConfiguration(hadoopConf)


Spark submit should take care of the security stuff.

Yes, I agree. But for now, the spark submit is quite yarn-specific. For security stuff contains in configMap of k8s, it's not working.

This reverts commit b31d487.

This reverts commit 528dfc9.

…nfigmap if the client process has files to upload" This reverts commit ad10f94.

SparkQA · 2019-09-05T06:22:45Z

Test build #110165 has finished for PR 25609 at commit 04a6aab.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-09-05T06:25:16Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/15171/

SparkQA · 2019-09-05T06:34:14Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/15173/

SparkQA · 2019-09-05T06:42:37Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/15171/

SparkQA · 2019-09-05T06:50:18Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/15173/

SparkQA · 2019-09-16T19:03:23Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/15644/

SparkQA · 2019-09-16T19:23:13Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/15644/

yaooqinn · 2019-09-17T02:25:45Z

gentle ping @skonto

skonto · 2019-10-11T13:16:24Z

docs/security.md

+is configured, the hadoop configurations will only be used by the Driver and its Executors. If your client process has
+extra dependencies to upload to `spark.kubernetes.file.upload.path`, you may need to configure `HADOOP_CONF_DIR` too.
+When these two variables are both set, Spark will prefer `spark.kubernetes.hadoop.configMapName` to be mounted on the
+Driver/Executor pods.


The concept looks good to me, @ifilonenko any corner cases?

I think it is still okay to warn the user about, the configuration is being picked from the configMap, inspite of the fact, HADOOP_CONF_DIR is defined.

...e/src/test/scala/org/apache/spark/deploy/k8s/features/HadoopConfDriverFeatureStepSuite.scala

skonto · 2019-10-11T13:32:22Z

@erikerlandson LGTM, we need an integration test for this @ifilonenko gentle ping.

SparkQA · 2019-10-12T02:19:31Z

Test build #111938 has finished for PR 25609 at commit 7f2c957.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-10-12T02:46:04Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/16953/

SparkQA · 2019-10-12T03:01:28Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/16953/

yaooqinn · 2019-10-22T09:41:50Z

anything blocks this？ @skonto

yaooqinn · 2019-12-10T02:29:11Z

retest this please

yaooqinn · 2019-12-10T02:33:21Z

cc @cloud-fan @vanzin, could any active committers help review this? thanks in advance.

SparkQA · 2019-12-10T02:53:47Z

Test build #115066 has finished for PR 25609 at commit 7f2c957.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-12-10T02:56:48Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/19884/

SparkQA · 2019-12-10T03:20:42Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/19884/

yaooqinn · 2019-12-10T03:22:27Z

retest this please

SparkQA · 2019-12-10T03:36:56Z

Test build #115069 has finished for PR 25609 at commit 7f2c957.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-12-10T03:43:49Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/19886/

SparkQA · 2019-12-10T03:54:19Z

Test build #115072 has finished for PR 25609 at commit 7f2c957.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-12-10T04:08:40Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/19886/

erikerlandson · 2019-12-10T23:02:05Z

Does this have an integration test now?
cc @ifilonenko @skonto

SparkQA · 2020-01-10T23:27:25Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/21297/

SparkQA · 2020-01-11T00:02:36Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/21297/

SparkQA · 2020-05-18T00:33:29Z

Test build #122767 has finished for PR 25609 at commit 7f2c957.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

github-actions · 2020-08-26T00:38:36Z

We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!

yaooqinn · 2020-08-26T00:40:59Z

retest this please

SparkQA · 2020-08-26T00:55:36Z

Test build #127901 has finished for PR 25609 at commit 7f2c957.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-08-26T01:25:26Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/32527/

SparkQA · 2020-08-26T01:42:40Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/32527/

[SPARK-28896][K8S] Download hadoop configurations from k8s configmap …

ad10f94

…if the client process has files to upload

few comments add

528dfc9

fix k8s it

b31d487

dongjoon-hyun added the KUBERNETES label Aug 28, 2019

skonto reviewed Sep 3, 2019

View reviewed changes

yaooqinn added 3 commits September 5, 2019 10:15

Revert "fix k8s it"

3e9c9a1

This reverts commit b31d487.

Revert "few comments add"

28dfba8

This reverts commit 528dfc9.

Revert "[SPARK-28896][K8S] Download hadoop configurations from k8s co…

553ed00

…nfigmap if the client process has files to upload" This reverts commit ad10f94.

yaooqinn changed the title ~~[SPARK-28896][K8S] Download hadoop configurations from k8s configmap if the client process has files to upload~~ [SPARK-28896][K8S] Support HADOOP_CONF_DIR and config map at the same time Sep 5, 2019

support both configure HADOOP_CONF_DIR and config map

e8ffc92

yaooqinn changed the title ~~[SPARK-28896][K8S] Support HADOOP_CONF_DIR and config map at the same time~~ [SPARK-28896][K8S] Support defining HADOOP_CONF_DIR and config map at the same time Sep 5, 2019

doc

04a6aab

skonto reviewed Oct 11, 2019

View reviewed changes

...e/src/test/scala/org/apache/spark/deploy/k8s/features/HadoopConfDriverFeatureStepSuite.scala Show resolved Hide resolved

rm dangling space

7f2c957

github-actions bot added the Stale label Aug 26, 2020

github-actions bot closed this Aug 27, 2020

yaooqinn changed the title ~~[SPARK-28896][K8S] Support defining HADOOP_CONF_DIR and config map at the same time~~ [SPARK-28895][K8S] Support defining HADOOP_CONF_DIR and config map at the same time Sep 29, 2020

[SPARK-28895][K8S] Support defining HADOOP_CONF_DIR and config map at the same time #25609

[SPARK-28895][K8S] Support defining HADOOP_CONF_DIR and config map at the same time #25609

Conversation

yaooqinn commented Aug 28, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

SparkQA commented Aug 28, 2019

Uh oh!

SparkQA commented Aug 28, 2019

Uh oh!

SparkQA commented Aug 28, 2019

Uh oh!

SparkQA commented Aug 28, 2019

Uh oh!

SparkQA commented Aug 28, 2019

Uh oh!

SparkQA commented Aug 28, 2019

Uh oh!

SparkQA commented Aug 28, 2019

Uh oh!

SparkQA commented Aug 28, 2019

Uh oh!

SparkQA commented Aug 28, 2019

Uh oh!

skonto Sep 3, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yaooqinn Sep 4, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

skonto Sep 4, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

skonto Sep 4, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

erikerlandson Sep 4, 2019

Choose a reason for hiding this comment

Uh oh!

yaooqinn Sep 5, 2019

Choose a reason for hiding this comment

Uh oh!

ifilonenko Sep 11, 2019

Choose a reason for hiding this comment

Uh oh!

skonto Sep 3, 2019

Choose a reason for hiding this comment

Uh oh!

yaooqinn Sep 4, 2019

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Sep 5, 2019

Uh oh!

SparkQA commented Sep 5, 2019

Uh oh!

SparkQA commented Sep 5, 2019

Uh oh!

SparkQA commented Sep 5, 2019

Uh oh!

SparkQA commented Sep 5, 2019

Uh oh!

SparkQA commented Sep 16, 2019

Uh oh!

SparkQA commented Sep 16, 2019

Uh oh!

yaooqinn commented Sep 17, 2019

Uh oh!

skonto Oct 11, 2019

Choose a reason for hiding this comment

Uh oh!

ScrapCodes Mar 4, 2020

Choose a reason for hiding this comment

yaooqinn commented Aug 28, 2019 •

edited

Loading

skonto Sep 3, 2019 •

edited

Loading

yaooqinn Sep 4, 2019 •

edited

Loading

skonto Sep 4, 2019 •

edited

Loading

skonto Sep 4, 2019 •

edited

Loading