Case study · Data engineering · BFH BTI1032
Garage data lake
A graded exercise from the Data Engineering course at BFH: a small data lake in Docker, analysed with Spark.
- Context: M.Sc. Data Science, BFH, course BTI1032
- Garage (S3) · Docker · fastparquet · PySpark · Jupyter
The setup
Garage is an S3-compatible object store. It runs in Docker next to a Spark container with Jupyter, both on one Docker network. I converted the CSV data to Parquet with fastparquet and stored it in Garage. Spark reads the Parquet files straight from Garage. The analysis is SQL-style DataFrame code wrapped in Python functions.
The pipeline
The result
5.3× faster
I timed the same work two ways: Spark RDDs on the raw CSV, and Spark DataFrames on Parquet from Garage. Each timing covers ingest and every analysis, averaged over 10 runs. RDD on CSV took 3.49 s. DataFrame on Parquet took 0.65 s. That's 5.3× faster, even with the data coming over the network.
Total time, average of 10 runs
Why it's faster
- No Python lambdas, so no row-by-row traffic between the JVM and Python.
- Spark's Catalyst optimiser plans the whole query.
- Parquet is columnar and typed, so there's no string parsing.
Classification
The last part predicts a customer's country from the products they bought, with a decision tree and a random forest. The tree reached 91% accuracy. The classes are heavily imbalanced, though: always guessing UK already scores 90.8%, so accuracy says little here. My aim was to compare training speed between the decision tree and the random forest, and the baseline check is what made that clear.
Speed: the random forest with 50 trees took 16.9 s and the single tree 11.8 s, so the forest was only 1.4× slower.
Code
The code is public, without keys or config secrets.