In this lesson, you'll process the data you collected in the previous lesson and implement algorithms to analyse it. This builds on your hypothesis and prepares you for building the full analytics artefact.

We'll guide you through reviewing your data, cleaning it, exploring patterns, selecting and coding algorithms, evaluating results, and documenting your work. By the end, you'll have initial insights from your data analysis.

Begin by examining the dataset from your previous lesson. Check for completeness, structure, and relevance to your hypothesis.

Key Checks:

• Is the data complete? Look for missing entries.
• Does it have a clear structure (e.g., columns for variables like age, score)?
• Is it relevant? Does it help test your hypothesis?
• Identify issues like inconsistencies (e.g., different date formats).

import csv

# Example: Load data from CSV
data = []
with open('your_data.csv', 'r') as file:
    reader = csv.reader(file)
    for row in reader:
        data.append(row)

print(data[:5])  # Print first 5 rows

Handle any issues found in the review. Clean the data to make it ready for analysis. This step is crucial for structuring and transforming raw data to prepare it for analysis (outcome 3.5).

Steps:

1. Remove or fill missing values (e.g., replace with averages or remove rows with too many gaps).
2. Normalise formats (e.g., convert all dates to YYYY-MM-DD or standardise text cases).
3. Transform variables (e.g., convert strings to numbers, or scale values for consistency).
4. Check for duplicates and remove them if necessary.

Use statistics and visualisations to understand your data's patterns, trends, and relationships.

Approach:

• Calculate basics like mean, min, max.
• Create simple plots (use print for text-based or matplotlib if available).
• Look for correlations.

# Assume cleaned_data has numeric values in index 1
values = [row[1] for row in cleaned_data]
mean = sum(values) / len(values)
print('Mean:', mean)

# Simple text visualization
print('Histogram:')
for i in range(0, max(values)+1, 5):
    count = sum(1 for v in values if i <= v < i+5)
    print(f'{i}-{i+4}: ' + '*' * count)