Spark (AWS EMR): Update to emr-5.20 (with Spark 2.4) : Paralellism is not inherited

43
February 12, 2019, at 10:40 PM

We have updated aws emr to version 5.20 from 5.16(which uses spark 2.4 instead of 2.3.1). We noticed some jobs take longer. It seems to be related to a problem with the parallelism level on the RDDs. When running this code on aws emr:

JavaPairRDD<String, T> result = keyBy(baseRDD.repartition(1000), keyGetter) .mapValues(t -> function1(t));

The keyBy operation performs 1000 tasks, but the mapValues is using the default parallelism of the cluster, while in aws emr 5.16 the same code uses the inherited parallelism of the previous RDD (1000). The number of partitions of the RDD resulted from keyBy operation (getNumPartitions()) is indeed the default parallelism level.

We can't find whether this has changed on spark or aws-emr or is some sort of bug, but it is affecting performance a lot.

READ ALSO
How can I split string by a special character and ignore everything inside parentheses?

How can I split string by a special character and ignore everything inside parentheses?

I want to split the string by "/" and ignore "/" inside the outer parentheses

49
Comparing data in hql and return boolean value [on hold]

Comparing data in hql and return boolean value [on hold]

I want to compare data in hql and get boolean values to username variable, below some of my codes

50
Bad request only on safari browser

Bad request only on safari browser

I have a problem that I do not understand, on this site, in https and safari I have a bad request, on other browsers it's ok

34
After submitting form, if form_validation false, then redirect user to view and show form&#39;s filled input values using redirect()

After submitting form, if form_validation false, then redirect user to view and show form's filled input values using redirect()

In CodeIgniter, after submitting form, when we get POST data in controller function of form action, I am checking for form_validationBut if form_validation checking returns false, I want to redirect the user to view and show him the form's filled values

46