References

Variables

Variable is any characteristic, number, or quantity that can be measured or counted. In machine learning, we need to understand all types of variables so that we can handle them accordingly in Machine learning pro-processing steps.

Input variables

Input variables are the independent variables in the model. They are also called features or attributes.

Input variables are denoted . The number of input variables in a data matrix is denoted nn.

Target variables

Target variables are the dependent variables in the model.

Target variables are denoted $t$. The number of target variables in a data matrix is denoted mm.

In approximation problems, targets are continuous variables (power consumption, product quality, etc.).

In classification problems, targets are categorical variables (fault, churn, etc.). In this type of application, targets are also called categories or labels.

Links

Types of variablesMedium

Variable Types in Machine LearningMedium

Independent and Dependent Variables in Machine LearningMedium

Variables in Machine Learning — EITCeitc.org

Loss Function

A loss function evaluates the performance of a model on a single data point by comparing the model’s prediction with the actual (ground truth) value, thereby computing a penalty for that individual prediction. Let’s take an example of a machine learning model predicting the temperature for a single day. If the actual temperature is 20°C and the model predicts 22°C, the loss function calculates the error for this specific prediction.

Mathematical Representation:

• Mean Squared Error (MSE) for a single data point where L represent the loss function:

  • Formula: $L(y,\hat{y}) =(y–\hat{y})^2$

  • Here, $y$ is the actual value and $\hat{y}$ is the predicted value.

  • For our temperature example: $y = 20, \hat{y} = 20,$ so $L = (20-22)^2 = 4$.

Cost Function

A cost function evaluates the overall performance of the model across the entire dataset (or a batch from it). It aggregates the loss penalties of all individual data points, and may include additional constraints or penalties (like L1 or L2 regularizations). Let’s understand with the same example of the machine learning model predicting the temperature for a single day. The cost function assesses the model’s performance over the entire month’s data.

Mathematical Representation:

• Average MSE for the entire dataset:

  • Formula: $J = \frac{1}{n} \sum_{i=1}^{N} (y_i - \hat{y_i}) ^2$

  • is the number of days in the dataset.

  • $J$ represents the cost function calculated as the average MSE over the dataset

  • $y_i$ is the actual value of the i-th data point, and $\hat{y_i}.$

  • Calculates the mean of the squared differences between the actual and predicted values for all data points in the dataset.

• Incorporating Regularization:

  • L1 (Lasso) adds the sum of the absolute values of the coefficients.

  • L2 (Ridge) adds the sum of the squares of the coefficients.