What Is Data Science and How Does It Work?
Every online purchase generates transaction records containing customer identifiers, product IDs, timestamps, quantities, payment values, and transaction statuses. Similarly, digital platforms collect clickstream events, API request logs, user sessions, and application telemetry. These datasets contain information that can be processed to identify behavioural patterns, detect anomalies, and generate predictions.
Data science is an interdisciplinary field that combines statistical inference, linear algebra, probability theory, programming, and machine learning to extract information from structured, semi-structured, and unstructured datasets.
A typical data science system transforms raw observations into analytical outputs through data ingestion, preprocessing, feature engineering, statistical modelling, and model evaluation.
Python, SQL, NumPy, Pandas, Scikit-learn, and statistical visualisation libraries are commonly used to implement these processes.
What Are the Main Components of Data Science?
Data science combines several technical disciplines, each responsible for a specific part of the analytical process.
| Component | Technical Role | Common Tools or Methods |
|---|---|---|
| Data Engineering | Data ingestion, transformation, and storage | SQL, Apache Spark, ETL pipelines |
| Statistics | Estimation, hypothesis testing, and uncertainty analysis | SciPy, statistical tests |
| Mathematics | Numerical computation and optimisation | Linear algebra, calculus |
| Programming | Data processing and automation | Python, SQL |
| Machine Learning | Predictive modelling and pattern recognition | Scikit-learn, XGBoost |
| Data Visualisation | Exploratory analysis and communication | Matplotlib, Seaborn |
| Model Deployment | Serving and monitoring predictions | FastAPI, Docker, MLflow |
Although these components are connected, data science projects do not always require every technique. Descriptive statistical analysis may be sufficient for some business problems, whereas complex forecasting or classification tasks may require machine learning.
What Types of Data Are Used in Data Science?
Data science systems generally process three categories of information.
1. Structured Data
Structured data follows a predefined schema and is commonly stored in relational database tables.
Consider the following transaction dataset:
| transaction_id | customer_id | amount | payment_status |
|---|---|---|---|
| 1001 | C101 | 2500 | Success |
| 1002 | C102 | 1800 | Failed |
| 1003 | C101 | 4200 | Success |
Each column represents a defined attribute, while each row represents an observation.
SQL can retrieve records satisfying specific conditions.
SELECT
customer_id,
SUM(amount) AS total_spending
FROM transactions
WHERE payment_status = 'Success'
GROUP BY customer_id;This query calculates total successful transaction value for each customer.
2. Semi-Structured Data
Semi-structured data contains identifiable fields but does not necessarily follow a fixed relational schema.
JSON documents are a common example.
{
"customer_id": "C101",
"event": "add_to_cart",
"product_id": "P205",
"session": {
"duration_seconds": 340,
"device": "mobile"
}
}Such records are frequently generated by REST APIs, event-tracking systems, and web applications.
Python can parse JSON records and extract individual attributes for analysis.
3. Unstructured Data
Unstructured data does not follow a predefined tabular schema.
Examples include customer reviews, images, audio recordings, support conversations, and video files.
Techniques such as natural language processing, computer vision, and representation learning help transform this information into numerical features.
For example, customer reviews can be converted into vector representations using TF-IDF or text embedding models before classification.
How Is Raw Data Converted Into Analytical Information?
A basic data science transformation pipeline can be represented as:
Raw Data → Data Ingestion → Data Validation → Data Cleaning → Feature Engineering → Statistical Analysis → Predictive Modelling → Evaluation
Consider a dataset containing customer transactions.
A data scientist can use Pandas to inspect the records.
import pandas as pd
df = pd.read_csv("transactions.csv")
print(df.head())
print(df.info())
print(df.isnull().sum())The operations perform three different tasks:
head()displays the initial observations.info()reports column types and non-null counts.isnull().sum()identifies missing-value counts.
A derived feature can then be created.
df["transaction_value"] = (
df["quantity"] * df["unit_price"]
)This transforms two raw variables into a numerical feature representing transaction value.
Feature engineering is important because predictive models depend on how relevant information is represented.
What Is the Mathematical Foundation of Data Science?
Data science relies on mathematical methods to represent observations, estimate relationships, and minimise prediction errors.
A supervised machine learning dataset is commonly represented as:
[
X \in \mathbb{R}^{n \times p}
]
Here:
- (X) represents the feature matrix.
- (n) represents the number of observations.
- (p) represents the number of input features.
The corresponding target vector is:
[
y \in \mathbb{R}^{n}
]
A predictive model estimates the relationship between input features and the target.
[
\hat{y} = f(X;\theta)
]
Here, (f) is the predictive function and (\theta) represents the model parameters.
For linear regression, the relationship can be expressed as:
[
\hat{y} = X\beta + b
]
The model estimates coefficients that minimise a suitable loss function, such as Mean Squared Error.
[
MSE = \frac{1}{n}\sum_{i=1}^{n}(y_i-\hat{y}_i)^2
]
This mathematical framework supports numerous predictive applications, including sales forecasting and numerical risk estimation.
How Is Data Science Different From Traditional Programming?
Traditional rule-based programming uses explicitly defined instructions to generate outputs.
For example, a payment-processing application may flag transactions exceeding a manually defined spending limit.
if transaction_amount > 50000:
flag_transaction = TrueA machine learning system approaches the same problem differently.
A model can learn statistical relationships between transaction amounts, device information, account activity, and historical fraud labels.
Rather than applying one fixed rule, it estimates a fraud probability.
fraud_probability = model.predict_proba(
transaction_features
)[0, 1]The probability can then be compared with a decision threshold.
flag_transaction = fraud_probability >= 0.80The threshold should be selected using validation data and the relative costs of false positives and false negatives.
Machine learning therefore introduces data-dependent decision functions, but it does not eliminate the need for explicit programming, validation, or business rules.
Key Takeaways
Data science combines computational methods and statistical reasoning to analyse datasets and develop evidence-based predictions.
Structured data is commonly processed using SQL and Pandas, whereas semi-structured and unstructured information may require additional parsing or representation techniques.
Machine learning extends statistical analysis by learning predictive relationships from historical observations. However, the quality of the resulting models depends on data representation, mathematical assumptions, and proper evaluation.
Data Science Tutorial for Beginners
Vaishali Ardhana








