Spark RDD Optimization Techniques In Hadoop

Welcome to the lesson ‘Spark RDD Optimization Techniques’ of Big Data Hadoop Tutorial which is a part of ‘big data hadoop online training’ offered by OnlineItGuru.

In this lesson, we will look into the lineage of Resilient Distributed Datasets or RDDs and discuss how optimization and performance improvement can be achieved by using the Apache Spark technique of persisting the RDDs.

Let us look at the objectives of this lesson.

Objectives

After completing this lesson, you will be able to:

• Explain how RDD lineage is created
• Explain how to mark persistence on an RDD
• Explain the features of RDD persistence
• List the storage levels in RDD persistence
• Describe how distributed persistence and fault tolerance help avoid data loss

Check out the big data hadoop certification

In the next section of this Spark tutorial, we will learn the concept of RDD Lineage.

Resilient Distributed Database (RDD) Lineage

RDD lineage is built whenever transformations are applied on an RDD. Let us understand this concept with the help of an example.

Creating Child RDDs

In the example above, the first step is creating an RDD named mydata by reading the text file Simplilearn.txt.

Step 1: Creating an RDD named mydata

To create the RDD mydata, execute the sc.textfile command as shown in the first line of code on the section. Observe that the RDD is created, and it is linked to the base file. However, the RDD is empty at this time.

Step 2: Executing a transformation operation

The second step is to execute a transformation operation to convert the contents of simplilearn.txt to uppercase as shown in the second line of code.

A child MappedRDD2 is then created and linked to the base text file. This is also empty at this time.

Step 3: Executing a transformation operation to filter the sentences

The third step is to execute a transformation operation to filter the sentences from MappedRDD 2 that begin with uppercase “I” as shown in the second line of the code.

The RDD myrdd is now created and linked to the parent MappedRDD2, but it is also empty at this time. Each time a transformation operation is executed on the file, a child RDD is created. Thus, the RDD lineage is built.

The third step is to execute an Action. In the example, the Count action is executed as shown in the third line of code. The command will compute and return the value.

The Count action is executed on the last RDD, which is myrdd. Only when the action is executed, the RDDs are populated with the data.

The previously issued transformation commands are executed on the RDDs, and they are now populated with the data.

Step 4: Executing the Count Action

In the fourth step, the Count action is executed again. But it requires all the three transformations in the entire RDD lineage to be executed again because the data will not be stored.

Let us look at RDD persistence in the next section.

What is Spark RDD Persistence?

RDD persistence is an optimization technique for Apache Spark. It helps avoid re-computation of the whole lineage and saves the data by default in the memory. This improves performance.

Let us now observe how saved data is used for “Action” operations and how re-computation of the entire lineage is avoided when persistence is used on the RDD.

Marking an RDD for Persistence

To mark an RDD for persistence, use the Persist method. Once this is done, data of that RDD is saved by default in the memory. Observe the example shown in the section.

Step 1: Creating the RDD mydata

The first step is creating the RDD mydata by reading the text file simplilearn.txt.

Step 2: Executing the transformation

The second step is to execute the transformation to convert the contents of the text file to upper case as shown in the second line of the code. The child RDD myrdd1 is created.

Step 3: Mark the RDD for persistence

In this step, we mark the RDD for persistence. In the third line of the code, myrdd1 is marked for persistence. So, this data will be stored in memory.

Step 4: Filtering the sentences from myrdd

The fourth step is to filter the sentences from myrdd1 that begin with an upper case “I.” This creates the child RDD myrdd2. However, myrdd2 is not marked for persistence. Hence, the data of myrdd2 will not be stored.

Observe that all the child RDDs created when the transformations are executed are empty and will be populated on the execution of an Action.

Step 5: Optimization with RDD Persistence

In this step, we execute the Count action, and the entire lineage is computed. However, the data of myrdd1 will be saved because it was marked for persistence.

Step 6:Executing the Count Action

In the sixth step, the Count action is executed again. This time, however, the transformations of the entire lineage are not computed. The transformation applied for myrdd2 are computed, because the Count action has a dependency on the content of myrdd2.

Myrdd2 has a dependency on myrdd1, but since myrdd1 was marked for persistence, the Count action uses the saved data and outputs the result.

Observe that the transformation on mydata and the base are not recomputed. Thus, optimization is achieved, and performance is improved by the use of RDD persistence.

Features of RDD Persistence

The various features of RDD Persistence are explained below in detail.

Storage and reuse of the RDD partitions:

RDD persistence facilitates Storage and reuse of the RDD partitions. When an RDD is marked for persistence, every node stores any of the RDD partitions computed in memory. It then reuses them in other actions on the dataset. This facilitates better speed.

Automatic recomputation of lost RDD partitions:

Automatic re-computation of lost RDD partitions: If an RDD partition is lost, it is automatically re-computed using the original transformations. Thus, the cache is fault-tolerant.

Storage of persisted RDDs on different storage levels:

Every persisted RDD is stored on a different storage level that is determined by the Storage Level object passed to the Persist method.

Let us now look at the different storage levels in RDD Persistence.

Storage Levels in RDD Persistence

The various storage levels in RDD Persistence are explained below in detail.

MEMORY_ONLY

The Memory Only level allows storing RDD as deserialized Java objects. However, if any RDD does not fit in memory, a few partitions will not be cached. These will be re-computed on the go when required.

MEMORY_AND_DISK

The Memory and Disk level allows storage of RDD as deserialized Java objects. Also, if any RDD does not fit in memory, it stores it on the disk and reads from the disk when required.

MEMORY_ONLY_SER

The Memory Only Ser level stores RDD as serialized Java objects. This enables better space efficiency, especially in case of a fast serializer.

MEMORY_AND_DISK_SER

The Memory and Disk Ser level are similar to MEMORY_ONLY_SER, except that it spills partitions not fitting in memory to disk.

DISK_ONLY

The Disk Only level allows storage of RDD partitions only on disk.

MEMORY_ONLY_2, MEMORY_AND_DISK_2

The Memory Only 2, Memory And Disk 2, and other levels are similar to the Disk Only level, except that they replicate every partition on two cluster nodes.

OFF_HEAP

The Off-Heap level, experim

ental, allows storage of RDD in serialized format in Tachyon, the default off-heap option in Apache Spark. It reduces garbage collection overhead as compared to the Memory only Ser level and avoids losing the in-memory cache.

Next, we will see how to select the correct storage level.

Selecting the correct storage level

The storage level needs to be decided because of the trade-offs between memory usage and CPU efficiency. Consider the following when you need to decide on which storage level to select.

• If the RDDs fit with the default storage level, which is the Memory Only level, the default level should be left as is.
• If it is not possible to leave the default level as is, which may be the case if the storage space is insufficient, use the Memory Only Ser level and also use a fast serialization library.
• When fast fault recovery is needed, use the replicated storage levels, that is the MEMORY_ONLY_2, MEMORY_AND_DISK_2 levels.
• If environments have high amounts of memory or multiple applications, use the experimental Off-Heap storage level.

In the following section of this Spark Tutorial, we will try to understand what Distributed Persistence is.

Distributed Persistence

By default, Apache Spark RDDs are distributed across the cluster and persisted in memory in executor virtual machines.

As shown on the section, two RDDs are in different nodes and different executor JVMs. This enables fault-tolerance.

Now, suppose node 2 is down, as shown on the section. When the node is down, the RDD partition in that node is lost. Then the driver starts a new task to re-compute the partition on a different node.

Observe that the task is now executed on Node 3. This preserves the RDD Lineage, and the data is never lost. Persistence option on an RDD can be changed. To unmark persistence on the RDD, and to remove the RDD from memory and disk, use the rdd.unpersist method.

Fault Tolerance in Spark RDD

Apache Spark RDDs are fault tolerant as they track data lineage information.

• They rebuild lost data on failure using lineage.
• Each RDD remembers how it was created from other datasets (by transformations like a map, join, or group by) and recreates itself.
• You can mark an RDD to be persisted using the persist() or cache() methods.
• The first time it is computed in action, it will be kept in memory on the nodes.
• Apache Spark’s cache is fault-tolerant, which means if any partition of an RDD is lost, it will automatically be recomputed using the transformations that created it.
• The distributed persistence architecture is targeted at applications that have distributed active requirements.

Changing Persistence Options

To change the Apache Spark RDD persistence to a different storage level, the persistence on the RDD must first be removed. So the unpersist method must first be used, and then the RDD can again be marked for persistence with a different storage level.

Summary

Let’s now summarize what we learned in this lesson.

• When RDD lineage is preserved, every parent transformation is re-computed for each“Action operation.
• To overcome the issue of re-computation and to improve performance, RDD persistence can be used.
• RDD can be persisted by using the Persist method. This saves the data of the persisted RDD by default in the memory, which enables faster access and computation.
• Every persisted RDD is stored on a different storage level.
• Distributed persistence enables fault tolerance and avoids loss of data.
• Persistence options can be changed as needed.

Willing to take up a course in Haoop? Check out our ,big data and hadoop course

Conclusion

This concludes the lesson on Spark RDD Optimization Techniques.