logo
down
shadow

Spark 1.5.2 and Hadoop 2.4 (Hive 2) version compatibility


Spark 1.5.2 and Hadoop 2.4 (Hive 2) version compatibility

Content Index :

Spark 1.5.2 and Hadoop 2.4 (Hive 2) version compatibility
Tag : hadoop , By : Pradeep Gowda
Date : November 29 2020, 09:01 AM

it fixes the issue I'd say the best practice would be to run spark and hadoop on the same cluster. In fact, spark can run as a yarn application (if you do spark-submit with --master yarn client). Why ? It boils down to data locality. Data Locality is a fundamental concept of hadoop and data systems in general. The general idea is that the data you want to process it's so big so rather than moving the data, you'd rather move the program to the node the data resides on. So, in the case of spark, if you run it on a different cluster, all the data will have to be moved from a cluster to another through the network. It's more efficient to have computation and data on the same node.
As for version, having two hadoop clusters with different versions can be a pain. I'd recommend you have 2 different installations of spark, one per cluster, compiled for the appropriate version of hadoop.

Comments
No Comments Right Now !

Boards Message :
You Must Login Or Sign Up to Add Your Comments .

Share : facebook icon twitter icon

How to fix 'org.apache.hadoop.hive.ql.metadata.HiveException' in Hive-Spark, when using simple query?


Tag : apache-spark , By : S Hall
Date : March 29 2020, 07:55 AM
wish help you to fix your issue After spending some time around this issue, I managed to find the solution. It has nothing to do with the query. There was another SparkClient process running, once I stopped it and executed the query it works fine.

Comparing Cassandra's CQL vs Spark/Shark queries vs Hive/Hadoop (DSE version)


Tag : cassandra , By : eataix
Date : March 29 2020, 07:55 AM

ERROR running HIVE at LINUX COMMAND LINE.. java version = 1.7, HADOOP 2.3.0, Apache HIVE 1.1.0


Tag : hadoop , By : user147496
Date : March 29 2020, 07:55 AM

Derby version mismatch between Spark and Hive : Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClie


Tag : apache-spark , By : user98986
Date : March 29 2020, 07:55 AM

Unrecognized Hadoop major version number: 3.0.0-beta1 at org.apache.hadoop.hive.shims.ShimLoader.getMajorVersion


Tag : hadoop , By : Nosayaba
Date : March 29 2020, 07:55 AM
it helps some times EDIT: As of May 21st, 2018 Hive 3.0.0 was released and supports Hadoop 3. Refer to the JIRA Changelog for more details.
shadow
Privacy Policy - Terms - Contact Us © scrbit.com