Skip to main content

IGeLU 2020 Digital Conference and Developers' Day 14 -17th September, 2020

All events will be held online. It is worth noting that all times in the program are quoted in British Summer Time (BST ; = UTC+1).

You can register here. As we are running multiple streams, we have had to break down the registration process to a day-by-day and session-by-session registration process.


PySpark : Combining Machine Learning & Big Data
Monday, September 14, 2020 at 1:30 PM–2:00 PM, Zoom - 1
Conference or Developers' Day
Developers' Day (30 min session)
Session Abstract
With the ever increasing flow of data, comes the industry focus on how to use those data for driving business & insights; but what about the size of the data these days, we have to deal with ?

How about using the potential of big data libraries with support in Python to deal with this huge amount of data for deriving business insights using ML techniques? But how can we amalgamate the two?

Usually people in the ML domain prefer using Python; so combining the potential of Big Data technologies like Spark etc to supplement ML is a matter of ease with pyspark ( A Python package to use the Spark’s capabilities ).

This talk would revolve around -

1) Why do we need to fuse Big Data with Machine Learning ?
2) How Spark’s architecture will help us boost our preparations for faster ML ?
3) How pyspark’s MLlib ( Machine Learning library ) helps you do ML so seamlessly ?
Applicable Ex Libris Product
None - General
Target Audience Skill Level
Register to the full Dev Day part at
Ayon Roy, Lulu International Exchange
Presenter's job title:
Data Science Intern
Mehmet Celik, KU Leuven