Machine Learning With Pyspark | What Is Pyspark?

When to Use PySpark vs Pandas?

The choice between PySpark and Pandas depends on the specific data analysis tasks and requirements. Here are some factors you can consider when deciding whether to use PySpark or Pandas:

1. Dataset Size: If you are working with small to medium-sized datasets that can fit in the memory of a single machine, Pandas is likely to be the better choice. However, if you are dealing with large-scale datasets that cannot fit in the memory of a single machine, PySpark is the better choice.
2. Computing Resources: PySpark is designed to leverage distributed computing resources to process large-scale datasets across a cluster of machines. If you have access to a distributed computing environment, such as a Hadoop cluster, PySpark can provide significant performance benefits.
3. The complexity of Data Processing Tasks: PySpark is more suitable for complex data processing tasks that involve multiple stages of data transformation and analysis. Pandas is more suitable for simple data analysis tasks that involve filtering, selecting, and aggregating data.
4. Learning curve: Understanding the Spark architecture and using PySpark can be a tedious task. On the other hand, if you know Python, you can start working with pandas within an hour. So, if you have a small dataset and you immediately want to perform analytical tasks on the data, go for pandas.

What is PySpark used for?

Most data scientists and analysts are familiar with Python and use it to implement machine learning workflows. PySpark allows them to work with a familiar language on large-scale distributed datasets.

Apache Spark can also be used with other data science programming languages like R. If this is something you are interested in learning, the Introduction to Spark with sparklyr in R course is a great place to start.

What is Pandas?

Pandas is a popular open-source data analysis library for the Python programming language. If you are a data analyst or data scientist who works with python, you must have used pandas in data analysis tasks.

• Pandas provides data structures and functions for working with structured data. It provides us with the Series and DataFrame data structures using which we can analyze one-dimensional and tabular data respectively.
• The Pandas library also provides a range of functions for data manipulation, including filtering, selecting, joining, grouping, and aggregating data. It also provides functionality for handling missing data, reshaping data, and handling time-series data.
• Pandas also provides great data visualization capabilities. The matplotlib library is integrated into pandas. Due to this, we can plot different metrics from dataframes using only a single function call.
• Data scientists and data analysts use pandas extensively due to their simplicity and ease of use. Pandas is built on top of the numpy library. Due to this, it also performs well in numerical data analysis.

End-to-end Machine Learning PySpark Tutorial

Now that you have PySpark up and running, we will show you how to execute an end-to-end customer segmentation project using the library.

Customer segmentation is a marketing technique companies use to identify and group users who display similar characteristics. For instance, if you visit Starbucks only during the summer to purchase cold beverages, you can be segmented as a “seasonal shopper” and enticed with special promotions curated for the summer season.

Data scientists usually build unsupervised machine learning algorithms such as K-Means clustering or hierarchical clustering to perform customer segmentation. These models are great at identifying similar patterns between user groups that often go unnoticed by the human eye.

In this tutorial, we will use K-Means clustering to perform customer segmentation on the e-commerce dataset we downloaded earlier.

By the end of this tutorial, you will be familiar with the following concepts:

• Reading csv files with PySpark
• Exploratory Data Analysis with PySpark
• Grouping and sorting data
• Performing arithmetic operations
• Aggregating datasets
• Data Pre-Processing with PySpark
• Working with datetime values
• Type conversion
• Joining two dataframes
• The rank() function
• PySpark Machine Learning
• Creating a feature vector
• Standardizing data
• Building a K-Means clustering model
• Interpreting the model

Step 1: Creating a SparkSession

A SparkSession is an entry point into all functionality in Spark, and is required if you want to build a dataframe in PySpark. Run the following lines of code to initialize a SparkSession:


spark = SparkSession.builder.appName("Datacamp Pyspark Tutorial").config("spark.memory.offHeap.enabled","true").config("spark.memory.offHeap.size","10g").getOrCreate()


Using the codes above, we built a spark session and set a name for the application. Then, the data was cached in off-heap memory to avoid storing it directly on disk, and the amount of memory was manually specified.

Step 2: Creating the DataFrame

We can now read the dataset we just downloaded:


df = spark.read.csv('datacamp_ecommerce.csv',header=True,escape="\"")


Note that we defined an escape character to avoid commas in the .csv file when parsing.

Let’s take a look at the head of the dataframe using the show() function:


df.show(5,0)


The dataframe consists of 8 variables:

1. InvoiceNo: The unique identifier of each customer invoice.
2. StockCode: The unique identifier of each item in stock.
3. Description: The item purchased by the customer.
4. Quantity: The number of each item purchased by a customer in a single invoice.
5. InvoiceDate: The purchase date.
6. UnitPrice: Price of one unit of each item.
7. CustomerID: Unique identifier assigned to each user.
8. Country: The country from where the purchase was made

Step 3: Exploratory Data Analysis

Now that we have seen the variables present in this dataset, let’s perform some exploratory data analysis to further understand these data points:

1. Let’s start by counting the number of rows in the dataframe:


df.count() # Answer: 2,500


1. How many unique customers are present in the dataframe?


df.select('CustomerID').distinct().count() # Answer: 95


1. What country do most purchases come from?

To find the country from which most purchases are made, we need to use the groupBy() clause in PySpark:


from pyspark.sql.functions import * from pyspark.sql.types import * df.groupBy('Country').agg(countDistinct('CustomerID').alias('country_count')).show()


The following table will be rendered after running the codes above:

Almost all the purchases on the platform were made from the United Kingdom, and only a handful were made from countries like Germany, Australia, and France.

Notice that the data in the table above isn’t presented in the order of purchases. To sort this table, we can include the orderBy() clause:


df.groupBy('Country').agg(countDistinct('CustomerID').alias('country_count')).orderBy(desc('country_count')).show()


The output displayed is now sorted in descending order:

1. When was the most recent purchase made by a customer on the e-commerce platform?

To find when the latest purchase was made on the platform, we need to convert the “InvoiceDate” column into a timestamp format and use the max() function in Pyspark:


spark.sql("set spark.sql.legacy.timeParserPolicy=LEGACY") df = df.withColumn('date',to_timestamp("InvoiceDate", 'yy/MM/dd HH:mm')) df.select(max("date")).show()


You should see the following table appear after running the code above:

1. When was the earliest purchase made by a customer on the e-commerce platform?

Similar to what we did above, the min() function can be used to find the earliest purchase date and time:


df.select(min("date")).show()


Notice that the most recent and earliest purchases were made on the same day just a few hours apart. This means that the dataset we downloaded contains information of only purchases made on a single day.

Step 4: Data Pre-processing

Now that we have analyzed the dataset and have a better understanding of each data point, we need to prepare the data to feed into the machine learning algorithm.

Let’s take a look at the head of the dataframe once again to understand how the pre-processing will be done:


df.show(5,0)


From the dataset above, we need to create multiple customer segments based on each user’s purchase behavior.

The variables in this dataset are in a format that cannot be easily ingested into the customer segmentation model. These features individually do not tell us much about customer purchase behavior.

Due to this, we will use the existing variables to derive three new informative features – recency, frequency, and monetary value (RFM).

RFM is commonly used in marketing to evaluate a client’s value based on their:

1. Recency: How recently has each customer made a purchase?
2. Frequency: How often have they bought something?
3. Monetary Value: How much money do they spend on average when making purchases?

We will now preprocess the dataframe to create the above variables.

Recency

First, let’s calculate the value of recency – the latest date and time a purchase was made on the platform. This can be achieved in two steps:

i) Assign a recency score to each customer

We will subtract every date in the dataframe from the earliest date. This will tell us how recently a customer was seen in the dataframe. A value of 0 indicates the lowest recency, as it will be assigned to the person who was seen making a purchase on the earliest date.


df = df.withColumn("from_date", lit("12/1/10 08:26")) df = df.withColumn('from_date',to_timestamp("from_date", 'yy/MM/dd HH:mm')) df2=df.withColumn('from_date',to_timestamp(col('from_date'))).withColumn('recency',col("date").cast("long") - col('from_date').cast("long"))


ii) Select the most recent purchase

One customer can make multiple purchases at different times. We need to select only the last time they were seen buying a product, as this is indicative of when the most recent purchase was made:


df2 = df2.join(df2.groupBy('CustomerID').agg(max('recency').alias('recency')),on='recency',how='leftsemi')


Let’s look at the head of the new dataframe. It now has a variable called “recency” appended to it:


df2.show(5,0)


An easier way to view all the variables present in a PySpark dataframe is to use its printSchema() function. This is the equivalent of the info() function in Pandas:


df2.printSchema()


The output rendered should look like this:

Frequency

Let’s now calculate the value of frequency – how often a customer bought something on the platform. To do this, we just need to group by each customer ID and count the number of items they purchased:


df_freq = df2.groupBy('CustomerID').agg(count('InvoiceDate').alias('frequency'))


Look at the head of this new dataframe we just created:


df_freq.show(5,0)


There is a frequency value appended to each customer in the dataframe. This new dataframe only has two columns, and we need to join it with the previous one:


df3 = df2.join(df_freq,on='CustomerID',how='inner')


Let’s print the schema of this dataframe:


df3.printSchema()


Monetary Value

Finally, let’s calculate monetary value – the total amount spent by each customer in the dataframe. There are two steps to achieving this:

i) Find the total amount spent in each purchase:

Each customerID comes with variables called “Quantity” and “UnitPrice” for a single purchase:

To get the total amount spent by each customer in one purchase, we need to multiply “Quantity” with “UnitPrice”:


m_val = df3.withColumn('TotalAmount',col("Quantity") * col("UnitPrice"))


ii) Find the total amount spent by each customer:

To find the total amount spent by each customer overall, we just need to group by the CustomerID column and sum the total amount spent:


m_val = m_val.groupBy('CustomerID').agg(sum('TotalAmount').alias('monetary_value'))


Merge this dataframe with the all the other variables:


finaldf = m_val.join(df3,on='CustomerID',how='inner')


Now that we have created all the necessary variables to build the model, run the following lines of code to select only the required columns and drop duplicate rows from the dataframe:


finaldf = finaldf.select(['recency','frequency','monetary_value','CustomerID']).distinct()


Look at the head of the final dataframe to ensure that the pre-processing has been done accurately:

Standardization

Before building the customer segmentation model, let’s standardize the dataframe to ensure that all the variables are around the same scale:


from pyspark.ml.feature import VectorAssembler from pyspark.ml.feature import StandardScaler assemble=VectorAssembler(inputCols=[ 'recency','frequency','monetary_value' ], outputCol='features') assembled_data=assemble.transform(finaldf) scale=StandardScaler(inputCol='features',outputCol='standardized') data_scale=scale.fit(assembled_data) data_scale_output=data_scale.transform(assembled_data)


Run the following lines of code to see what the standardized feature vector looks like:


data_scale_output.select('standardized').show(2,truncate=False)


These are the scaled features that will be fed into the clustering algorithm.

If you’d like to learn more about data preparation with PySpark, take this feature engineering course on Datacamp.

Step 5: Building the Machine Learning Model

Now that we have completed all the data analysis and preparation, let’s build the K-Means clustering model.

The algorithm will be created using PySpark’s machine learning API.

i) Finding the number of clusters to use

When building a K-Means clustering model, we first need to determine the number of clusters or groups we want the algorithm to return. If we decide on three clusters, for instance, then we will have three customer segments.

The most popular technique used to decide on how many clusters to use in K-Means is called the “elbow-method.”

This is done simply running the K-Means algorithm for a wide range of clusters and visualizing the model results for each cluster. The plot will have an inflection point that looks like an elbow, and we just pick the number of clusters at this point.

Read this Datacamp K-Means clustering tutorial to learn more about how the algorithm works.

Let’s run the following lines of code to build a K-Means clustering algorithm from 2 to 10 clusters:


from pyspark.ml.clustering import KMeans from pyspark.ml.evaluation import ClusteringEvaluator import numpy as np cost = np.zeros(10) evaluator = ClusteringEvaluator(predictionCol='prediction', featuresCol='standardized',metricName='silhouette', distanceMeasure='squaredEuclidean') for i in range(2,10): KMeans_algo=KMeans(featuresCol='standardized', k=i) KMeans_fit=KMeans_algo.fit(data_scale_output) output=KMeans_fit.transform(data_scale_output) cost[i] = KMeans_fit.summary.trainingCost


With the codes above, we have successfully built and evaluated a K-Means clustering model with 2 to 10 clusters. The results have been placed in an array, and can now be visualized in a line chart:


import pandas as pd import pylab as pl df_cost = pd.DataFrame(cost[2:]) df_cost.columns = ["cost"] new_col = range(2,10) df_cost.insert(0, 'cluster', new_col) pl.plot(df_cost.cluster, df_cost.cost) pl.xlabel('Number of Clusters') pl.ylabel('Score') pl.title('Elbow Curve') pl.show()


The codes above will render the following chart:

ii) Building the K-Means Clustering Model

From the plot above, we can see that there is an inflection point that looks like an elbow at four. Due to this, we will proceed to build the K-Means algorithm with four clusters:


KMeans_algo=KMeans(featuresCol='standardized', k=4) KMeans_fit=KMeans_algo.fit(data_scale_output)


iii) Making Predictions

Let’s use the model we created to assign clusters to each customer in the dataset:


preds=KMeans_fit.transform(data_scale_output) preds.show(5,0)


Notice that there is a “prediction” column in this dataframe that tells us which cluster each CustomerID belongs to:

Step 6: Cluster Analysis

The final step in this entire tutorial is to analyze the customer segments we just built.

Run the following lines of code to visualize the recency, frequency, and monetary value of each customerID in the dataframe:


import matplotlib.pyplot as plt import seaborn as sns df_viz = preds.select('recency','frequency','monetary_value','prediction') df_viz = df_viz.toPandas() avg_df = df_viz.groupby(['prediction'], as_index=False).mean() list1 = ['recency','frequency','monetary_value'] for i in list1: sns.barplot(x='prediction',y=str(i),data=avg_df) plt.show()


The codes above will render the following plots:

Here is an overview of characteristics displayed by customers in each cluster:

• Cluster 0: Customers in this segment display low recency, frequency, and monetary value. They rarely shop on the platform and are low potential customers who are likely to stop doing business with the ecommerce company.

• Cluster 1: Users in this cluster display high recency but haven’t been seen spending much on the platform. They also don’t visit the site often. This indicates that they might be newer customers who have just started doing business with the company.

• Cluster 2: Customers in this segment display medium recency and frequency and spend a lot of money on the platform. This indicates that they tend to buy high-value items or make bulk purchases.

• Cluster 3: The final segment comprises users who display high recency and make frequent purchases on the platform. However, they don’t spend much on the platform, which might mean that they tend to select cheaper items in each purchase.

To go beyond the predictive modelingmodelling concepts covered in this course, you can take the Machine Learning with PySpark course on Datacamp.

Let’s Create some Predictions from the Model

PySpark Linear Regression Predict

We can see that the R-squared value obtained is 0.85 or 85% which is quite good, considering that we did not perform exploratory data analysis and hyperparameter Tuning.

In this image, you can see the features and the predictions made by the model on the test data.

Build an Awesome Job Winning Project Portfolio with Solved End-to-End Big Data Projects

PySpark Machine Learning Example to Implement Linear Regression

The Linear Regression model predicts a continuous value from given features. The regression model targets prediction value based on independent variables also used to find the relationship between variables and forecasting.

1. Importing the Libraries
   

1. Creating Spark Session and reading the CSV file, which you can find Insurance.CSV.

1. Data Exploration for better understanding via data samples.

Our data has 7 columns where the ‘charges’ column is our dependent variable and ‘age’, ‘gender’, ‘BMI’, ‘children’, ‘smoker’, and ‘region’ are the independent variables.

1. Performing Exploratory Data Analysis
   

Exploratory Data Analysis is a process of performing initial experiments on data to discover patterns, detect outliers, and test assumptions with the help of statistics and graphical representations. Here we are using the describe data function to get the statistics of our data.

1. Using StringIndexer for dealing with Categorical data
   

StringIndexer: It is used to convert a string column into numerical form. It allocates unique values to each of the categories present in the respective column.

Output Variables

We can see from the output that columns like children, region, and smoker have been assigned to numeric values from the string.

1. Feature Engineering
   

Given the multiple columns, we need to merge them into a single column using VectorAssembler. It is a feature transformer that merges multiple columns into a vector column. One can select the number of columns used as input features and pass only those columns through the VectorAssembler. We will pass all seven input columns to create a single feature vector column in our case.

VectorAssembler: It is a transformer that helps to select the input columns that we need to create a single feature vector to train our Machine Learning models.

Output: Here the column features represent the VectorAssembler values.

Now that we have our features ready and the input variables are known, let’s get them together and dive into the machine-learning technique.

1. Splitting the Dataset
   

Splitting the Data into Train and Test sets to train our model and check its efficiency.

Checking the statistics of our train and test sets.

Get More Practice, More Big Data and Analytics Projects, and More guidance.Fast-Track Your Career Transition with ProjectPro

1. Build and Train Linear Regression Model
   

In this part, we build and train the Linear Regression model using features of the input and output columns. We can fetch the coefficients and intercept values of the model as well. We can also evaluate the performance of the model on training data as well using r2. This model gives a very good accuracy (77%) on training datasets which we will see in some time.

The Math Behind Measuring R-Squared Value

Assume that without Linear Regression, the Sum of Squared error is coming to 158 and after applying the Linear Regression model, we get that to 1.2; let’s check a way to calculate the R-squared value:

TSS (Total Sum of Squared Errors ) = SSE (Sum of squared errors) + SSR (Residual Sum of squared errors)

The total sum of squares is the sum of the squared difference between the actual and the mean values and is always fixed. This is equal to 158.

The SSE is the squared difference from the actual to predicted values of the target variable, which, as mentioned, is 1.2 after using Linear Regression.

SSR is the Sum of Squares explained by Regression and can be calculated by (TSS – SSE).

We use the evaluate function to make predictions for the test data and can use r2 to check the model’s accuracy on test data.

SSR = 158 – 1.2 =156.8

‘rsquare’ (Coefficient of determination) = SSR/TSS = 156.8/158 = 0.99

This percentage indicates that the Linear Regression model can predict with 99 % accuracy in predicting the salary amount given the person’s age. The other 1% can be attributed to errors the model cannot explain. Our Linear Regression line fits the model well, but it can also be a case of overfitting.

Overfitting occurs when the model predicts high accuracy on training data, but its performance drops on the unseen/test data, which causes weak prediction models. The technique to address the overfitting issues is known as regularization, and there are different types of regularization techniques.

In terms of Linear Regression, one can use Ridge, Lasso, or Elasticnet Regularization techniques to handle overfitting. Ridge Regression is also known as L2 regularization. It focuses on restricting the coefficient values of input features close to zero, whereas Lasso regression (L1) makes some of the coefficients zero to improve the model’s generalization. Elasticnet is a combination of both techniques.

Regression is still a parametric-driven approach and assumes few underlying patterns about distributions of input data points. The Linear Regression model does not perform well if the input data does not affiliate with those assumptions. Hence it is essential to go over these assumptions before using the Linear Regression model –

• There must be a linear relationship between the input and output variables.

• The independent variables (input features) should not be correlated to each other (also known as multicollinearity).

• There must be no correlation between the error values.

• There must be a linear relationship between the error and the output variable.

• The error values must be normally distributed.

What is PySpark?

PySpark is a Python library that provides an interface for Apache Spark. Spark is an open-source framework for big data processing. Spark is built to process large amounts of data quickly by distributing computing tasks across a cluster of machines.

• PySpark allows us to use Apache Spark and its ecosystem of libraries, such as Spark SQL for working with structured data.
• We can also use Spark MLlib for machine learning and GraphX for graph processing using Pyspark in Python.
• PySpark supports many data sources, including Hadoop Distributed File System (HDFS), Apache Cassandra, and Amazon S3.
• Along with the data processing capabilities, we can also use pyspark with popular Python libraries such as NumPy and Pandas.

Learn to Use Apache Spark for Machine Learning

Spark is a powerful, general purpose tool for working with Big Data. Spark transparently handles the distribution of compute tasks across a cluster. This means that operations are fast, but it also allows you to focus on the analysis rather than worry about technical details. In this course you’ll learn how to get data into Spark and then delve into the three fundamental Spark Machine Learning algorithms: Linear Regression, Logistic Regression/Classifiers, and creating pipelines.

What is Machine Learning?

Machine Learning is one of the many applications of Artificial Intelligence (AI) where the primary aim is to enable computers to learn automatically without any human assistance. With the help of Machine Learning, computers are able to tackle the tasks that were, until now, only handled and carried out by people. It is basically a process of teaching a system how to make accurate predictions when fed with the right data. It has the ability to learn and improve from past experience without being specifically programmed for a task. Machine Learning mainly focuses on developing computer programs and algorithms that make predictions and learn from the provided data.

Get certified from the top Big Data Course in Singapore now!

First, learn the basics of DataFrames in PySpark to get started with Machine Learning in PySpark.

What is PySpark MLlib?

Basic Introduction to PySpark MLlib

Spark MLlib is the short form of the Spark Machine Learning library. Machine Learning in PySpark is easy to use and scalable. It works on distributed systems. You can use Spark Machine Learning for data analysis. There are various techniques you can make use of with Machine Learning algorithms such as regression, classification, etc., all because of the PySpark MLlib.

Parameters in PySpark MLlib

Some of the main parameters of PySpark MLlib are listed below:

• Ratings: This parameter is used to create an RDD of ratings, rows, or tuples.
• Rank: It shows the number of features computed and ranks them.
• Lambda: Lambda is a regularization parameter.
• Blocks: Blocks are used to parallel the number of computations. The default value for this is −1.

Watch this PySpark Course for Beginners video from Intellipaat:

Why PySpark for Machine Learning?

Over 3,000 companies use Apache Spark, including top players like Oracle, Hortonworks, Cisco, Verizon, Visa, Microsoft, Databricks, and Amazon. Spark made waves in the recent year as the Big Data product with the shortest learning curve, popular with SMBs and Enterprise teams. Big companies heavily use machine learning models for making predictive analyses and keeping themselves ahead in the market; models like logistic regression, XGboost, SVMs, Random forest classifier model, decision trees, logistic regression model, and gradient boosted tree classifier are being heavily deployed from there end.

1. You gain the ability to work with Big Data.

2. Very good native SQL support.

3. Great documentation is available.

4. Supports machine learning for supervised learning algorithms.

5. Provides more scalable analysis.

PySpark for Machine Learning is fast and uses multiple machines for large-scale data processing. It runs on distributed computing, such as YARN, Mesos, and Standalone clusters. PySpark has two main Abstractions:

1. RDD – A distributed collection of objects.

2. Dataframe – Distributed dataset for tabular data.

RDD – Resilient Distributed Datasets are a distributed collection of immutable JVM objects that allows you to perform calculations very quickly, and they are the backbone of Apache spark. As the name suggests, the dataset is distributed; it is split into chunks based on some key and distributed to nodes. Doing this allows calculations to be performed very quickly. RDDs are schema-less data structures.

Dataframe – This is an immutable distributed collection of data organized into named columns, the same as a table in a relational database. People who work with pandas data frame can completely relate to this abstraction.

About this book

Master the new features in PySpark 3.1 to develop data-driven, intelligent applications. This updated edition covers topics ranging from building scalable machine learning models, to natural language processing, to recommender systems.

Machine Learning with PySpark, Second Edition begins with the fundamentals of Apache Spark, including the latest updates to the framework. Next, you will learn the full spectrum of traditional machine learning algorithm implementations, along with natural language processing and recommender systems. You’ll gain familiarity with the critical process of selecting machine learning algorithms, data ingestion, and data processing to solve business problems. You’ll see a demonstration of how to build supervised machine learning models such as linear regression, logistic regression, decision trees, and random forests. You’ll also learn how to automate the steps using Spark pipelines, followed by unsupervised models such as K-means and hierarchical clustering. A section on Natural Language Processing (NLP) covers text processing, text mining, and embeddings for classification. This new edition also introduces Koalas in Spark and how to automate data workflow using Airflow and PySpark’s latest ML library.

After completing this book, you will understand how to use PySpark’s machine learning library to build and train various machine learning models, along with related components such as data ingestion, processing and visualization to develop data-driven intelligent applications

What you will learn:

• Build a spectrum of supervised and unsupervised machine learning algorithms
• Use PySpark’s machine learning library to implement machine learning and recommender systems
• Leverage the new features in PySpark’s machine learning library
• Understand data processing using Koalas in Spark
• Handle issues around feature engineering, class balance, bias and variance, and cross validation to build optimally fit models

Who This Book Is For

Data science and machine learning professionals.

Advantages of Pandas Over PySpark

Pandas and PySpark are both popular tools for data analysis and processing. However, they have different strengths and weaknesses. Here are some advantages of Pandas over PySpark.

1. Ease of Use: Pandas is generally easier to use and has a lower learning curve compared to PySpark. The pandas API is simple and the syntax is similar to SQL and Excel. This makes it easy for analysts and data scientists to get started with data analysis and manipulation using pandas.
2. Interactivity: Pandas provides an interactive environment for data exploration and analysis through Jupyter notebooks. This allows us to visualize data and experiment with code more easily. PySpark, on the other hand, can have a higher barrier to entry. It requires setting up a distributed computing cluster before running code.
3. Well-suited for small to medium-sized data: Pandas is well-suited for handling small to medium-sized datasets that can fit in memory. It provides fast and efficient data manipulation and processing on a single machine, without requiring distributed computing resources.
4. Flexibility: The pandas module is highly flexible and can work with a wide variety of data sources. We can use CSV, Excel, SQL databases, parquet files, and more. It also provides a wide range of data manipulation functions that can handle complex data transformation tasks.
5. Integration with Other Libraries: Pandas integrates well with other data science libraries in the Python ecosystem, such as NumPy, Matplotlib, and Scikit-learn. This makes it easy to build end-to-end data analysis pipelines and machine learning workflows using a variety of tools.
6. Community Support: Pandas has a large and active community of users and contributors. It also has extensive documentation that explains each function with examples. This makes it easy to find help and resources when working with Pandas.

Spark MLLib to Leverage PySpark for Machine Learning

MLLib is Spark’s machine learning (ML) library that helps make functional machine-learning models scalable and manageable.

Spark MLLib consists of in-built tools such as.

• Machine Learning Algorithms: Common learning algorithms include logistic regression models, regression, decision trees, random forest algorithms, and many other ensemble methods.

• Featurization: Feature extraction, feature scaling, feature selection, and dimensionality reduction.

• Pipelines: Tools for constructing, evaluating, and building machine learning pipelines.

• Persistence: Saving and loading algorithms, models, and pipelines.

While building a machine learning model, data scientists must perform many tasks, including data cleaning, feature engineering, making inferences from data, building machine learning pipelines, saving models, and finally deploying it. Spark MLlib library provides multiple features, making it a go-to choice for data scientists across top tech companies.

Performing Linear Regression on a Real-world Dataset

Let’s understand Machine Learning better by implementing a full-fledged code to perform linear regression on the dataset of the top 5 Fortune 500 companies in the year 2017.

Go through these Spark Interview Questions and Answers to excel in your Apache Spark interview!

Loading Data

As mentioned above, you are going to use a DataFrame that is created directly from a CSV file. Following are the commands to load data into a DataFrame and to view the loaded data.

from pyspark import SparkConf, SparkContext
from pyspark.sql import SQLContext
Sc = SparkContext()
sqlContext = SQLContext(sc)

company_df = sqlContext.read.format(‘com.databricks.spark.csv’).options(header=’true’, inferschema=’true’).load(‘C:/Users/intellipaat/Downloads/spark-2.3.2-bin-hadoop2.7/Fortune5002017.csv’)
company_df.take(1)

You can choose the number of rows you want to view while displaying the data of the DataFrame. Here, only the first row is displayed.

[Row (Rank=1, Title= ‘Walmart’, Website= ‘http:/www.walmart.com’, Employees-2300000, Sector= ‘retailing’)]

Data Exploration

To check the data type of every column of a DataFrame and to print the schema of the DataFrame in a tree format, you can use the following commands, respectively:

company_df.cache()
company_df.printSchema()

DataFrame[Rank: int, Title: string, Website: string, Employees: Int, Sector: string]
root
|– Rank:  integer (nullable = true)
|– Title:  string (nullable = true)
|– Website:  string (nullable = true)
|– Employees:  integer (nullable = true)
|– Sector:  string (nullable = true)

Become an Apache Spark Specialist by going for this Big Data Online Course in London!

Performing Descriptive Analysis

company_df.describe().toPandas().transpose()

Finding the Correlation Between Independent Variables

To find out if any of the variables, i.e., fields have correlations or dependencies, you can plot a scatter matrix. Plotting a scatter matrix is one of the best ways in Machine Learning to identify linear correlations if any.

You can plot a scatter matrix on your DataFrame using the following code:

import pandas as pdnumeric_features = [t[0] for t in company_df.dtypes if t[1] == ‘int’ or t[1] == ‘double’]
sampled_data = company_df.select(numeric_features).sample(False, 0.8).toPandas()
axs = pd.scatter_matrix(sampled_data, figsize=(10, 10))
n = len(sampled_data.columns)
for i in range(n):
v = axs[i, 0]
v.yaxis.label.set_rotation(0)
v.yaxis.label.set_ha(‘right’)
v.set_yticks(())
h = axs[n-1, i]
h.xaxis.label.set_rotation(90)
h.set_xticks(())

Here, you can come to the conclusion that in the dataset, the “Rank” and “Employees” columns have a correlation. Let’s dig a little deeper into finding the correlation specifically between these two columns.

In case you have doubts or queries related to Spark and Hadoop, kindly refer to our Big Data Hadoop and Spark Community!

Correlation Between Independent Variables

import six
for i in comapny_df.columns:
if not( isinstance(company_df.select(i).take(1)[0][0], six.string_types)):
print( “Correlation to Employees for “, i, company_df.stat.corr(‘Employees’,i))

Correlation to Employees for Rank   −0.778372714650932
Correlation to Employees  1.0

The value of correlation ranges from −1 to 1, the closer it is to ‘1’ the more positive correlation can be found between the fields. If the value is closer to −1, it means that there is a strong negative correlation between the fields. Now, you can analyze your output and see if there is a correlation or not, and if there is, then if it is a strong positive or negative correlation.

Preparing Data

from pyspark.ml.feature import VectorAssembler
vectorAssembler = VectorAssembler(inputCols = [‘Rank’, ‘Employees’], outputCol = ‘features’)
tcompany_df = vectorAssembler.transform(company_df)
tcompany_df = tcompany_df.select([‘features’, ‘Employees’])
tcompany_df.show(3)

• Output:
  • Out [7]:
• Input:

splits = tcompany_df.randomSplit([0.7, 0.3])
train_df = splits[0]
test_df = splits[1]

Linear Regression

from pyspark.ml.regression import LinearRegression
lr = LinearRegression(featuresCol = ‘features’, labelCol=’Emplyees’, maxIter=10, regParam=0.3, elasticNetParam=0.8)
lr_model = lr.fit(train_df)
print(“Coefficients: ” + str(lr_model.coefficients))
print(“Intercept: ” + str(lr_model.intercept))

Coefficients: [-32251.88812374517, 0.9255193858709874]
Intercept: 140317.88600801243

After performing linear regression on the dataset, you can finally come to the conclusion that ‘Employees’ is the most important field or factor, in the given dataset, which can be used to predict the ranking of the companies in the coming future. ‘Ranks’ has a linear correlation with ‘Employees,’ indicating that the number of employees in a particular year, in the companies in our dataset, has a direct impact on the Rank of those companies.

Enhance your skills in Apache Spark by grabbing this Big Data Training!

Performing Linear Regression on a Real-world Dataset

Let’s understand Machine Learning better by implementing a full-fledged code to perform linear regression on the dataset of the top 5 Fortune 500 companies in the year 2017.

Go through these Spark Interview Questions and Answers to excel in your Apache Spark interview!

Loading Data

As mentioned above, you are going to use a DataFrame that is created directly from a CSV file. Following are the commands to load data into a DataFrame and to view the loaded data.

from pyspark import SparkConf, SparkContext
from pyspark.sql import SQLContext
Sc = SparkContext()
sqlContext = SQLContext(sc)

company_df = sqlContext.read.format(‘com.databricks.spark.csv’).options(header=’true’, inferschema=’true’).load(‘C:/Users/intellipaat/Downloads/spark-2.3.2-bin-hadoop2.7/Fortune5002017.csv’)
company_df.take(1)

You can choose the number of rows you want to view while displaying the data of the DataFrame. Here, only the first row is displayed.

[Row (Rank=1, Title= ‘Walmart’, Website= ‘http:/www.walmart.com’, Employees-2300000, Sector= ‘retailing’)]

Data Exploration

To check the data type of every column of a DataFrame and to print the schema of the DataFrame in a tree format, you can use the following commands, respectively:

company_df.cache()
company_df.printSchema()

DataFrame[Rank: int, Title: string, Website: string, Employees: Int, Sector: string]
root
|– Rank:  integer (nullable = true)
|– Title:  string (nullable = true)
|– Website:  string (nullable = true)
|– Employees:  integer (nullable = true)
|– Sector:  string (nullable = true)

Become an Apache Spark Specialist by going for this Big Data Online Course in London!

Performing Descriptive Analysis

company_df.describe().toPandas().transpose()

Finding the Correlation Between Independent Variables

To find out if any of the variables, i.e., fields have correlations or dependencies, you can plot a scatter matrix. Plotting a scatter matrix is one of the best ways in Machine Learning to identify linear correlations if any.

You can plot a scatter matrix on your DataFrame using the following code:

import pandas as pdnumeric_features = [t[0] for t in company_df.dtypes if t[1] == ‘int’ or t[1] == ‘double’]
sampled_data = company_df.select(numeric_features).sample(False, 0.8).toPandas()
axs = pd.scatter_matrix(sampled_data, figsize=(10, 10))
n = len(sampled_data.columns)
for i in range(n):
v = axs[i, 0]
v.yaxis.label.set_rotation(0)
v.yaxis.label.set_ha(‘right’)
v.set_yticks(())
h = axs[n-1, i]
h.xaxis.label.set_rotation(90)
h.set_xticks(())

Here, you can come to the conclusion that in the dataset, the “Rank” and “Employees” columns have a correlation. Let’s dig a little deeper into finding the correlation specifically between these two columns.

In case you have doubts or queries related to Spark and Hadoop, kindly refer to our Big Data Hadoop and Spark Community!

Correlation Between Independent Variables

import six
for i in comapny_df.columns:
if not( isinstance(company_df.select(i).take(1)[0][0], six.string_types)):
print( “Correlation to Employees for “, i, company_df.stat.corr(‘Employees’,i))

Correlation to Employees for Rank   −0.778372714650932
Correlation to Employees  1.0

The value of correlation ranges from −1 to 1, the closer it is to ‘1’ the more positive correlation can be found between the fields. If the value is closer to −1, it means that there is a strong negative correlation between the fields. Now, you can analyze your output and see if there is a correlation or not, and if there is, then if it is a strong positive or negative correlation.

Preparing Data

from pyspark.ml.feature import VectorAssembler
vectorAssembler = VectorAssembler(inputCols = [‘Rank’, ‘Employees’], outputCol = ‘features’)
tcompany_df = vectorAssembler.transform(company_df)
tcompany_df = tcompany_df.select([‘features’, ‘Employees’])
tcompany_df.show(3)

• Output:
  • Out [7]:
• Input:

splits = tcompany_df.randomSplit([0.7, 0.3])
train_df = splits[0]
test_df = splits[1]

Linear Regression

from pyspark.ml.regression import LinearRegression
lr = LinearRegression(featuresCol = ‘features’, labelCol=’Emplyees’, maxIter=10, regParam=0.3, elasticNetParam=0.8)
lr_model = lr.fit(train_df)
print(“Coefficients: ” + str(lr_model.coefficients))
print(“Intercept: ” + str(lr_model.intercept))

Coefficients: [-32251.88812374517, 0.9255193858709874]
Intercept: 140317.88600801243

After performing linear regression on the dataset, you can finally come to the conclusion that ‘Employees’ is the most important field or factor, in the given dataset, which can be used to predict the ranking of the companies in the coming future. ‘Ranks’ has a linear correlation with ‘Employees,’ indicating that the number of employees in a particular year, in the companies in our dataset, has a direct impact on the Rank of those companies.

Enhance your skills in Apache Spark by grabbing this Big Data Training!

Introduction to Spark MLlib

Apache Spark comes with a library named MLlib to perform Machine Learning tasks using the Spark framework. Since there is a Python API for Apache Spark, i.e., PySpark, you can also use this Spark ML library in PySpark. MLlib contains many algorithms and Machine Learning utilities.

In this tutorial, you will learn how to use Machine Learning in PySpark. The dataset of Fortune 500 is used in this tutorial to implement this. This dataset consists of the information related to the top 5 companies ranked by Fortune 500 in the year 2017. This tutorial will use the first five fields. You can download the dataset by clicking here.

The dataset looks like below:

In this Spark ML tutorial, you will implement Machine Learning to predict which one of the fields is the most important factor to predict the ranking of the above-mentioned companies in the coming years. Also, you will use DataFrames to implement Machine Learning.

Before diving right into this Spark MLlib tutorial, have a quick rundown of all the topics included in this tutorial:

Conclusion

In this article, we discussed pyspark vs pandas to compare their performance, speed, memory consumption, and use cases. To learn more about programming, you can read this article on spark vs Hadoop. You might also like this article on the best python debugging tools.

I hope you enjoyed reading this article. Stay tuned for more informative articles.

Happy Learning!

Disclosure of Material Connection: Some of the links in the post above are “affiliate links.” This means if you click on the link and purchase the item, I will receive an affiliate commission. Regardless, I only recommend products or services I use personally and believe will add value to my readers.

Data Scientists love working with PySpark as it helps streamline the overall process of deploying production-grade machine learning models from the prototyping stage. Data scientists across organizations claim that it helps them reduce the support required from the development team to scale machine learning models from prototyping to production. If you’re a data scientist or a machine learning engineer keen on getting your hands dirty with PySpark, you’re on the right page. This PySpark Machine Learning Tutorial is a beginner’s guide to building and deploying machine learning pipelines at scale using Apache Spark with Python.

Data Scientist spends 80% of their time wrangling and cleaning data, but as soon as we start to work with Big Data, using Python Pandas might be ineffective when working with large datasets for machine learning. PySpark comes to the rescue here as it provides various features such as querying in SQL, working with DataFrame, Streaming, Machine Learning, and abstracts that help to work with Big Data. PySpark is an easy-to-use interface for writing Apache Spark Code.

PySpark Linear Regression Example with Source Code

Now, the next step will be to pass the training set to our model and evaluate the same on the test data.

We can see that the R-squared value obtained is 0.77 or 77% which is quite decent, considering that we did not perform exploratory data analysis and hyperparameter Tuning.

R-Squared (R² or the coefficient of determination) is a statistical term used in a regression model that helps to determine the proportion of variance in the target variable that the independent variable can explain. In other words, r-squared shows how well the data fits in the regression model.

How to Build Machine Learning Pipelines with PySpark?

PySpark provides several APIs to work with DataFrames that ease the job of Data Scientists when building machine learning pipelines.

PySpark Machine Learning Pipeline Fundamentals

• DataFrame: This Machine Learning API is the same as the one used in Python Pandas for data representation.

• Transformers: A Transformer is an algorithm that can transform one DataFrame into another DataFrame.

• Estimator: When building machine learning models, we choose the best models that train on our DataFrame and produce results using estimators.

• Pipeline: A Pipeline combines numerous Transformers and Estimators to carry out a Machine Learning workflow.

Keywords searched by users: machine learning with pyspark

Categories: Khám phá 11 Machine Learning With Pyspark

See more here: kientrucannam.vn

See more: https://kientrucannam.vn/vn/