Isha Jain's blog

Support Vector Machines for Regression

Support Vector Machines for Regression

Isha Jain's photo
Isha Jain
·

5 min read

Table of contents

About

Support Vector Machines are a part of Supervise Learning Algorithms. These can be used as a part of classification or regression. These can be used to solve linear and non-linear problems. Here, in this part, we are going to see SVM Using Regression.(As for this article, I have focused on the initial stages of laying the foundation for further optimization.)

How SVM functions !!

Support Vector Machines (SVMs) are machine learning algorithm that seeks to find an optimal hyperplane in an n-dimensional space to effectively separate data into distinct classes. The dimension of this hyperplane is directly influenced by the number of features in the dataset. When dealing with two features, the hyperplane takes the form of a 2D plane, while in cases with three features, it remains a plane but exists in a 3D space. However, as the number of features increases beyond three, finding a hyperplane that perfectly separates the data becomes increasingly challenging due to the complex and high-dimensional nature of the space.

Types of Kernel for Regression

  • Linear Kernel: The linear kernel creates a straight hyperplane in the feature space, suitable for modelling linear relationships between variables.

  • Polynomial Kernel: The polynomial kernel captures polynomial relationships in the data, enabling the modelling of curved patterns with a specified degree of the polynomial.

  • RBF (Radial Basis Function) Kernel: The RBF kernel, often the default choice, is highly versatile and capable of capturing various non-linear patterns due to its Gaussian distribution-based transformation.

  • Sigmoid Kernel: The sigmoid kernel is used when the relationship between variables follows a sigmoid (logistic) curve, useful in specific situations where linear or polynomial kernels are not suitable.

Code

The steps for the SVM model are as follows :

  • Importing the required libraries and reading the dataset.
import numpy as np 
import pandas as pd 
from sklearn.model_selection import train_test_split
from sklearn.svm import SVR
from sklearn.metrics import accuracy_score
from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_sco
  • Reading the dataset
data = pd.read_csv("/kaggle/input/breast-cancer-wisconsin-data/data.csv")
  • Performing initial check on the dataset.
df.head() # gives the first five rows oof the dataset
df.shape # gives the no. of rows and columns in the dataset
df.dropna() # drops the null values
df.dtypes # gives the datatypes of all column names
df. describe # gives a breif statistical description of dataset
  • Checking the values in the diagnosis column. This column will be used as target column.
data['diagnosis'].values
  • The values of the diagnosis column are malignant or benign. Mapping the malignant value to 0 and the benign value to 1.
data['diagnosis'] = data['diagnosis'].map({'M':0,'B': 1})
  • Checking for the duplicate and null values in the dataset.
df.duplicated().sum() # gives the total count of duplicated values
df.isnull().sum() # gives the total count of null values
  • x and y as input features and target labels, respectively, for training a machine learning model.
x = data[['radius_mean' , 'texture_mean' , 'perimeter_mean' , 'area_mean']]
y = data['diagnosis']
  • Splitting the dataset
y = y.values.reshape(-1, 1) 
x_train,x_test,y_train,y_test= train_test_split(x,y,train_size=0.4,random_state=42)

Common Terms Used

reg = variable

C = regularisation parameter

gamma = kernel coefficient

pred = variable used for prediction

fit = used for fitting the model

SVR = support vector regression

epsilon = it is a hyperparameter determines margins of error or tolerance

  • RBF Kernel

reg = SVR(kernel='rbf',C=100 , gamma=0.1, epsilon=0.1)
reg.fit(x_train,y_train)
pred = reg.predict(x_test)
pred

mae = mean_absolute_error(y_test, pred)
print(f"Mean Absolute Error (MAE): {mae:.2f}")

mse = mean_squared_error(y_test, pred)
print(f"Mean Squared Error (MSE): {mse:.2f}")

r2 = r2_score(y_test, pred)
print(f"R-squared (R^2): {r2:.2f}")

Mean Absolute Error (MAE): 0.39
Mean Squared Error (MSE): 0.18
R-squared (R^2): 0.21

  • Linear Kernel

reg = SVR(kernel="linear", C=100, gamma="auto")
reg.fit(x_train,y_train)
pred = reg.predict(x_test)
pred

mae = mean_absolute_error(y_test, pred)
print(f"Mean Absolute Error (MAE): {mae:.2f}")

mse = mean_squared_error(y_test, pred)
print(f"Mean Squared Error (MSE): {mse:.2f}")

r2 = r2_score(y_test, pred)
print(f"R-squared (R^2): {r2:.2f}")

Mean Absolute Error (MAE): 5.23
Mean Squared Error (MSE): 96.79
R-squared (R^2): -419.26

  • Sigmoid Kernel

reg = SVR(kernel='sigmoid', C=1.0, gamma='scale', epsilon=0.1)
reg.fit(x_train,y_train)
pred = reg.predict(x_test)
pred

mae = mean_absolute_error(y_test, pred)
print(f"Mean Absolute Error (MAE): {mae:.2f}")

mse = mean_squared_error(y_test, pred)
print(f"Mean Squared Error (MSE): {mse:.2f}")

r2 = r2_score(y_test, pred)
print(f"R-squared (R^2): {r2:.2f}")

Mean Absolute Error (MAE): 4.70
Mean Squared Error (MSE): 55.75
R-squared (R^2): -241.07

  • Poly Kernel

reg = SVR(kernel='poly', C=1.0, gamma='scale', epsilon=0.1)
reg.fit(x_train,y_train)
pred = reg.predict(x_test)
pred

mae = mean_absolute_error(y_test, pred)
print(f"Mean Absolute Error (MAE): {mae:.2f}")

mse = mean_squared_error(y_test, pred)
print(f"Mean Squared Error (MSE): {mse:.2f}")

r2 = r2_score(y_test, pred)
print(f"R-squared (R^2): {r2:.2f}")

Mean Absolute Error (MAE): 0.23
Mean Squared Error (MSE): 0.12
R-squared (R^2): 0.48

Results

Among the three kernel functions tested (RBF, Sigmoid, and Polynomial) for Support Vector Regression (SVR), the RBF (Radial Basis Function) kernel is the better choice. It yields the lowest Mean Absolute Error (MAE) and Mean Squared Error (MSE) and exhibits a higher R-squared (R^2) value, indicating better predictive accuracy and model performance on the given dataset.

Link to GitHub repository - https://github.com/ishaj72/ML-Practise-Model/blob/main/model6-2.ipynb

Readers can access my GitHub repository through the provided link to view the code along with its corresponding output. This allows for a more comprehensive understanding of the code's functionality and results.

Conclusion

In conclusion, the article explored the use of SVM in determining whether the cancer is malignant or benign using the breast cancer Wisconsin dataset. We started with reading the dataset and further performing a basic EDA on the dataset. Then we mapped the diagnosis column to a numerical value.

Further, we split the dataset for training and testing purposes. We delved into the core of SVMs with the creation of an SVM classifier using the scikit-learn library. The model was trained on the training data and utilized for predictions on the testing data. We then used different types of kernels and checked their mean absolute error, mean squared error and r-squared. Since in regression, we don't calculate accuracy as done in classification.

Link to SVM for classification - https://whizzx.hashnode.dev/support-vector-machines-for-classification

SVM#RegressionMachine Learning