I was going through the Spark SQL for a join optimised using Adaptive Query Execution,
On the right side, spark get to know the size of table is small enough for broadcast and therefore decides for broadcast hash join.
As we know, broadcast hash join in a narrow operation, why do we still have exchange in the left table (large one)
Even in the final Physical plan, exchange is there
You get a local shuffle based on Mappers, not Reducers. I.e. a localized shuffle instead of regular such shuffle.
See https://www.waitingforcode.com/apache-spark-sql/what-new-apache-spark-3-local-shuffle-reader/read and https://dev.to/yaooqinn/how-to-use-spark-adaptive-query-execution-aqe-in-kyuubi-2ek2
For background you can also peruse Spark: disk I/O on stage boundaries explanation
