Alpine Data Spark Auto Tuning
Configuring a Spark cluster is not the easiest thing in the world, even if you are a data scientist
Look ma, no data scientist
Alpine Labs wants to support scenarios in which there is no data scientist involved
Machine Learning to the rescue
Alpine Labs uses ML under the hood.
Spark has its nuances, and these need to be taken into account when deploying jobs and tuning clusters
But can data scientists really know everything about Spark?
Spark cluster tuning challenges
Finding a good tuning solution for Spark clusters is not trivial.