Apache Spark & Big Data test (Advanced)

Sample questions

5 sample questions from the Apache Spark & Big Data test, with the answers. Samples are taken out of the live test, so you will not meet these when you sit it.

1. Which of the following accurately describes the preferred use cases for broadcast hash join and sort-merge join in Spark?

   1. Broadcast hash join is preferred when one of the tables is small enough to fit in memory, while sort-merge join is preferred when both tables are large and can be efficiently sorted and merged.
   2. Both broadcast hash join and sort-merge join are preferred when one of the tables is small enough to fit in memory.
   3. Sort-merge join is preferred when one of the tables is small enough to fit in memory, while broadcast hash join is preferred when both tables are large and can be efficiently sorted and merged.
   4. Broadcast hash join is preferred when both tables are large and can be efficiently sorted and merged, while sort-merge join is preferred when one of the tables is small enough to fit in memory.

   Answer: A. Broadcast hash join is preferred when one of the tables is small enough to fit in memory, while sort-merge join is preferred when both tables are large and can be efficiently sorted and merged.

   Broadcast hash join is used when one side of the join is small enough to be broadcasted to all nodes, allowing for efficient in-memory joins. Sort-merge join is used when both sides are large and require sorting and merging, which is more efficient for larger datasets. This distinction ensures optimal performance based on data size and distribution.

2. What happens when you use the `partitionBy` method in Spark to write a DataFrame to disk, specifying the 'region' column as the partitioning key?

   1. The data is not partitioned at all, and all rows are written into one file for easier management.
   2. The data is written into a single partition, regardless of the 'region' column values, to optimize write performance.
   3. The data is written into partitions based on the number of rows, ensuring even distribution across partitions.
   4. The data is written into directory partitions based on the values of the 'region' column, allowing for efficient querying and pruning during reads.

   Answer: D. The data is written into directory partitions based on the values of the 'region' column, allowing for efficient querying and pruning during reads.

   When using the `partitionBy` method in Spark, data is written into directory partitions based on the specified column(s). This allows for efficient querying and pruning during reads, as Spark can skip partitions that do not match the query's filter conditions. In this case, the 'region' column determines the partitioning, and the data is organized accordingly. Other options are incorrect because they either describe a different behavior or do not align with how dynamic partitioning works in Spark.

3. Which of the following statements accurately describes a shuffle in Apache Spark?

   1. A shuffle redistributes data across executors by key, and it is typically the most expensive operation in Spark.
   2. A shuffle is used to combine data from multiple partitions without redistributing data across executors.
   3. A shuffle is a process that occurs during the caching of data in Spark.
   4. A shuffle is a type of transformation that is lazy and does not trigger a job.

   Answer: A. A shuffle redistributes data across executors by key, and it is typically the most expensive operation in Spark.

   A shuffle is an expensive operation in Spark that redistributes data across executors by key. It involves writing shuffle files to local disk on the map side and fetching them over the network on the reduce side. Other options may be incorrect because they do not accurately describe the definition or cost of a shuffle.

4. In a Spark application, you are joining two DataFrames, 'big' and 'countries', using the following code: big.join(broadcast(countries), 'country_code'). What is the expected execution plan and behavior of this join?

   1. The SortMergeJoin is used, and both DataFrames are shuffled and sorted.
   2. The BroadcastNestedLoopJoin is used, and the 'countries' DataFrame is broadcasted.
   3. The ShuffleHashJoin is used, and both DataFrames are shuffled.
   4. The BroadcastHashJoin is used, and the 'countries' DataFrame is broadcasted to all executors.

   Answer: D. The BroadcastHashJoin is used, and the 'countries' DataFrame is broadcasted to all executors.

   When using the broadcast() function in a join, Spark automatically applies a BroadcastHashJoin strategy. This means the smaller DataFrame (in this case, 'countries') is broadcasted to all executors, allowing the join to occur without shuffling the larger DataFrame. This is optimal for performance when one side of the join is significantly smaller than the other.

5. Which of the following is a common mistake or edge case related to shuffle read merging in Apache Spark?

   1. Shuffle read merging is not possible when using a custom partitioner that does not align with the default hash-based partitioning.
   2. Shuffle read merging is automatically handled by Spark and does not require any configuration.
   3. Shuffle read merging is only affected by the size of the data, not the partitioning strategy.
   4. Shuffle read merging is always enabled by default regardless of the data distribution.

   Answer: A. Shuffle read merging is not possible when using a custom partitioner that does not align with the default hash-based partitioning.

   Shuffle read merging allows Spark to combine multiple shuffle reads into a single read, improving performance. However, this is only possible when the partitioning across the shuffle stages is consistent. Using a custom partitioner that doesn't align with the default hash-based partitioning breaks this consistency, making shuffle read merging impossible.