Lab 1: Data exploration¶

Your first graded Python lab. You explore a table of diamonds for an imaginary jewellery retailer: check its structure and quality, build a pivot table of average price, draw a correlation heatmap, a violin plot and a hexbin plot, answer short questions about what you see, and submit one PDF on GAP.

Adapted from the DATA 300 GAP course, In-Class Module 1, Lab 1: Data Exploration. Written in our own words; follow GAP for the official wording.

1. What and why¶

The data has one row per diamond: its weight (carat), three quality grades (cut, color, clarity), two proportions (depth and table, both percentages), its price in US dollars, and its length, width and depth in millimetres (x, y, z).

2. Route A: Try JupyterLab (as on GAP)¶

Part 1: Open and name a notebook¶

Open Try JupyterLab, start a Python (Pyodide) notebook and rename it with the file name GAP gives (your last name, the course code and "AppliedLab1").

Part 2: Load the data and inspect its structure¶

Paste GAP's first code block into a cell and run it. In outline, it:

1. installs seaborn into the browser session with %pip;
2. imports pandas, seaborn, matplotlib and numpy, and open_url from pyodide.http;
3. loads diamonds.csv from the seaborn-data repository on GitHub;
4. prints info(), the missing-value count per column and the describe() summary.

Part 3: Pivot table¶

Paste and run GAP's second block. It builds a pivot table of the mean price with cut quality down the side and colour grade across the top. On GAP's scales, colour runs from D (colourless, best) to J (faint yellow), and cut from Fair (lowest) to Ideal (best).

Part 4: Three charts for a large dataset¶

Paste and run GAP's third block. It draws:

1. a correlation heatmap of the numeric columns, annotated, on a fixed −1 to +1 scale;
2. a violin plot of price for each cut;
3. a hexbin plot of carat against price, with a colour bar for the number of diamonds per hexagon.

Part 5: Answer the questions¶

Add a Markdown cell at the bottom and answer GAP's seven questions in full sentences. They come in three groups: data quality and structure (two questions), the pivot table (an unexpected comparison and the hidden variable behind it), and the charts (the strongest correlation with price, the shape of the price distribution, and where the hexbin is densest). Read every answer from your own output and name the table or chart it comes from.

Part 6: Export and submit¶

Run everything from the top, then File → Print → Save as PDF with Background graphics ticked. See Submission below.

3. Route B: offline in Anaconda JupyterLab¶

Use this route if Try JupyterLab will not load the data, or if you prefer your own laptop. It follows the same steps as GAP's code but reads the bundled copy of the data, so nothing is installed or downloaded. Create a notebook inside the DATA300_GAP folder (next to data). You can paste the whole block into one cell, or split it at the # --- lines into four cells, which matches GAP's parts. The questions are exactly the same as in Route A.

# Route B for Lab 1: run offline in Anaconda JupyterLab.
# The diamonds file comes from the course folder, so nothing is installed or downloaded.
import numpy as np
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt

pd.set_option("display.max_columns", None)   # show every column when printing

# --- Part 2: load the data and inspect its structure
diamonds = pd.read_csv("data/diamonds.csv")

print("--- Structure ---")
diamonds.info()

print("\n--- Missing values in each column ---")
print(diamonds.isnull().sum())

print("\n--- Summary of the numeric columns ---")
print(diamonds.describe().round(2))

# --- Part 3: pivot table of mean price (US dollars) by cut and colour
price_pivot = pd.pivot_table(diamonds, values="price", index="cut",
                             columns="color", aggfunc="mean")
print("\n--- Mean price (US dollars) by cut (rows) and colour grade (columns) ---")
print(price_pivot.round(2))

# --- Part 4: three charts
sns.set_theme(style="whitegrid")

# Chart 1: correlation heatmap of the numeric columns
corr_matrix = diamonds.select_dtypes(include="number").corr()
plt.figure(figsize=(10, 8))
sns.heatmap(corr_matrix, annot=True, fmt=".2f", cmap="coolwarm",
            linewidths=0.5, vmin=-1, vmax=1)
plt.title("Correlations between the numeric diamond attributes")
plt.show()

# Chart 2: violin plot of price for each cut (one colour, works on any seaborn version)
plt.figure(figsize=(10, 6))
sns.violinplot(data=diamonds, x="cut", y="price", color="#9cc3ef")
plt.title("Distribution of price for each cut quality")
plt.xlabel("Cut quality")
plt.ylabel("Price (US dollars)")
plt.show()

# Chart 3: hexbin of carat (x) against price (y)
plt.figure(figsize=(9, 6))
plt.hexbin(diamonds["carat"], diamonds["price"], gridsize=40, cmap="Blues", mincnt=1)
plt.colorbar(label="Number of diamonds in the hexagon")
plt.title("Where the diamonds concentrate: carat and price")
plt.xlabel("Carat (weight)")
plt.ylabel("Price (US dollars)")
plt.show()

In JupyterLab you may replace the print(...) around describe() and the pivot table with display(...) for nicer tables; the numbers are the same. Export with Kernel → Restart Kernel and Run All Cells, then File → Print → Save as PDF.

4. Checks you can make¶

These tell you whether your output is right without giving any answers away.

• The table has 10 columns: three text columns (cut, color, clarity) and seven numeric ones. The row count you report comes from info() (the RangeIndex line) or from diamonds.shape; it is more than 50,000.
• describe() lists the seven numeric columns only, and its count row agrees with what info() and the missing-value check say.
• The pivot table has 5 rows (one per cut) and 7 columns (colour grades D to J), and every value is a price in US dollars.
• The heatmap is a 7 × 7 square with 1.00 down the diagonal; the colour bar runs from −1 to +1.
• The violin plot has five violins, one per cut, and a y-axis in US dollars.
• The hexbin plot has carat along the bottom, price up the side and a colour bar counting diamonds.
• Each answer names the output it comes from, gives numbers with units (US dollars, carats), and, for the "why" question, names a variable and says how you would check it.

5. Known pitfalls and fixes¶

6. Beyond the lab¶

Optional, not graded. Try these in the same notebook after you have answered GAP's questions:

1. Count the problem rows yourself: diamonds.duplicated().sum() and (diamonds[["x", "y", "z"]] == 0).any(axis=1).sum(). Which data quality dimensions from the morning lecture do they break?
2. Build a second pivot table with aggfunc="count" in place of "mean". Which cells rest on few diamonds, and how should that change how much you trust their averages?
3. Draw a histogram of price with np.log10(diamonds["price"]) on the x-axis. How does the shape change compared with the violin plot?
4. Take the variable you named as hidden in your answer, split it into a few bands of your choice with pd.cut, and build a pivot table of mean price by band and cut. Does the comparison between cuts look different within each band?
5. Calculate a price per carat (diamonds["price"] / diamonds["carat"]) and compare its median across cuts. Write one sentence for the retailer's buyer.

7. Submission¶

1. Run Kernel → Restart Kernel and Run All Cells (Route B) or run every cell from the top (Route A), and check that every table and chart is visible.
2. Export with File → Print → Save as PDF, Background graphics ticked. If you wrote answers in a separate document instead, combine the PDFs offline as described in Combining PDFs without internet.
3. Open the PDF and check every part is present and readable.
4. Upload it to Applied Lab 1 on GAP and submit before Tuesday 10 November, 23:59 China Standard Time (how to submit).