!pip install ucimlrepo
!pip install scipy

Requirement already satisfied: ucimlrepo in /home/studio-lab-user/.conda/envs/default/lib/python3.9/site-packages (0.0.7)
Requirement already satisfied: pandas>=1.0.0 in /home/studio-lab-user/.conda/envs/default/lib/python3.9/site-packages (from ucimlrepo) (2.2.3)
Requirement already satisfied: certifi>=2020.12.5 in /home/studio-lab-user/.conda/envs/default/lib/python3.9/site-packages (from ucimlrepo) (2024.2.2)
Requirement already satisfied: numpy>=1.22.4 in /home/studio-lab-user/.conda/envs/default/lib/python3.9/site-packages (from pandas>=1.0.0->ucimlrepo) (2.0.2)
Requirement already satisfied: python-dateutil>=2.8.2 in /home/studio-lab-user/.conda/envs/default/lib/python3.9/site-packages (from pandas>=1.0.0->ucimlrepo) (2.9.0)
Requirement already satisfied: pytz>=2020.1 in /home/studio-lab-user/.conda/envs/default/lib/python3.9/site-packages (from pandas>=1.0.0->ucimlrepo) (2024.1)
Requirement already satisfied: tzdata>=2022.7 in /home/studio-lab-user/.conda/envs/default/lib/python3.9/site-packages (from pandas>=1.0.0->ucimlrepo) (2024.2)
Requirement already satisfied: six>=1.5 in /home/studio-lab-user/.conda/envs/default/lib/python3.9/site-packages (from python-dateutil>=2.8.2->pandas>=1.0.0->ucimlrepo) (1.16.0)

[notice] A new release of pip is available: 24.3.1 -> 25.0.1
[notice] To update, run: pip install --upgrade pip
Collecting scipy
  Downloading scipy-1.13.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (60 kB)
Requirement already satisfied: numpy<2.3,>=1.22.4 in /home/studio-lab-user/.conda/envs/default/lib/python3.9/site-packages (from scipy) (2.0.2)
Downloading scipy-1.13.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (38.6 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 38.6/38.6 MB 119.3 MB/s eta 0:00:00a 0:00:01
Installing collected packages: scipy
Successfully installed scipy-1.13.1

[notice] A new release of pip is available: 24.3.1 -> 25.0.1
[notice] To update, run: pip install --upgrade pip

<link rel="stylesheet" type="text/css" href="custom.css" />

from ucimlrepo import fetch_ucirepo 
  
# fetch dataset 
adult = fetch_ucirepo(id=2) 
  
# data (as pandas dataframes) 
X = adult.data.features 
y = adult.data.targets 
  
# metadata 
print(adult.metadata) 
  
# variable information 
print(adult.variables)

{'uci_id': 2, 'name': 'Adult', 'repository_url': 'https://archive.ics.uci.edu/dataset/2/adult', 'data_url': 'https://archive.ics.uci.edu/static/public/2/data.csv', 'abstract': 'Predict whether annual income of an individual exceeds $50K/yr based on census data. Also known as "Census Income" dataset. ', 'area': 'Social Science', 'tasks': ['Classification'], 'characteristics': ['Multivariate'], 'num_instances': 48842, 'num_features': 14, 'feature_types': ['Categorical', 'Integer'], 'demographics': ['Age', 'Income', 'Education Level', 'Other', 'Race', 'Sex'], 'target_col': ['income'], 'index_col': None, 'has_missing_values': 'yes', 'missing_values_symbol': 'NaN', 'year_of_dataset_creation': 1996, 'last_updated': 'Tue Sep 24 2024', 'dataset_doi': '10.24432/C5XW20', 'creators': ['Barry Becker', 'Ronny Kohavi'], 'intro_paper': None, 'additional_info': {'summary': "Extraction was done by Barry Becker from the 1994 Census database.  A set of reasonably clean records was extracted using the following conditions: ((AAGE>16) && (AGI>100) && (AFNLWGT>1)&& (HRSWK>0))\n\nPrediction task is to determine whether a person's income is over $50,000 a year.\n", 'purpose': None, 'funded_by': None, 'instances_represent': None, 'recommended_data_splits': None, 'sensitive_data': None, 'preprocessing_description': None, 'variable_info': 'Listing of attributes:\r\n\r\n>50K, <=50K.\r\n\r\nage: continuous.\r\nworkclass: Private, Self-emp-not-inc, Self-emp-inc, Federal-gov, Local-gov, State-gov, Without-pay, Never-worked.\r\nfnlwgt: continuous.\r\neducation: Bachelors, Some-college, 11th, HS-grad, Prof-school, Assoc-acdm, Assoc-voc, 9th, 7th-8th, 12th, Masters, 1st-4th, 10th, Doctorate, 5th-6th, Preschool.\r\neducation-num: continuous.\r\nmarital-status: Married-civ-spouse, Divorced, Never-married, Separated, Widowed, Married-spouse-absent, Married-AF-spouse.\r\noccupation: Tech-support, Craft-repair, Other-service, Sales, Exec-managerial, Prof-specialty, Handlers-cleaners, Machine-op-inspct, Adm-clerical, Farming-fishing, Transport-moving, Priv-house-serv, Protective-serv, Armed-Forces.\r\nrelationship: Wife, Own-child, Husband, Not-in-family, Other-relative, Unmarried.\r\nrace: White, Asian-Pac-Islander, Amer-Indian-Eskimo, Other, Black.\r\nsex: Female, Male.\r\ncapital-gain: continuous.\r\ncapital-loss: continuous.\r\nhours-per-week: continuous.\r\nnative-country: United-States, Cambodia, England, Puerto-Rico, Canada, Germany, Outlying-US(Guam-USVI-etc), India, Japan, Greece, South, China, Cuba, Iran, Honduras, Philippines, Italy, Poland, Jamaica, Vietnam, Mexico, Portugal, Ireland, France, Dominican-Republic, Laos, Ecuador, Taiwan, Haiti, Columbia, Hungary, Guatemala, Nicaragua, Scotland, Thailand, Yugoslavia, El-Salvador, Trinadad&Tobago, Peru, Hong, Holand-Netherlands.', 'citation': None}}
              name     role         type      demographic  \
0              age  Feature      Integer              Age   
1        workclass  Feature  Categorical           Income   
2           fnlwgt  Feature      Integer             None   
3        education  Feature  Categorical  Education Level   
4    education-num  Feature      Integer  Education Level   
5   marital-status  Feature  Categorical            Other   
6       occupation  Feature  Categorical            Other   
7     relationship  Feature  Categorical            Other   
8             race  Feature  Categorical             Race   
9              sex  Feature       Binary              Sex   
10    capital-gain  Feature      Integer             None   
11    capital-loss  Feature      Integer             None   
12  hours-per-week  Feature      Integer             None   
13  native-country  Feature  Categorical            Other   
14          income   Target       Binary           Income   

                                          description units missing_values  
0                                                 N/A  None             no  
1   Private, Self-emp-not-inc, Self-emp-inc, Feder...  None            yes  
2                                                None  None             no  
3    Bachelors, Some-college, 11th, HS-grad, Prof-...  None             no  
4                                                None  None             no  
5   Married-civ-spouse, Divorced, Never-married, S...  None             no  
6   Tech-support, Craft-repair, Other-service, Sal...  None            yes  
7   Wife, Own-child, Husband, Not-in-family, Other...  None             no  
8   White, Asian-Pac-Islander, Amer-Indian-Eskimo,...  None             no  
9                                       Female, Male.  None             no  
10                                               None  None             no  
11                                               None  None             no  
12                                               None  None             no  
13  United-States, Cambodia, England, Puerto-Rico,...  None            yes  
14                                       >50K, <=50K.  None             no

import polars as pl
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import scipy.stats as stats
import pandas as pd
from ucimlrepo import fetch_ucirepo

# Fetch dataset
adult = fetch_ucirepo(id=2)
X = pl.from_pandas(adult.data.features)
y = pl.from_pandas(adult.data.targets)

# Combine features and target into a single Polars DataFrame
df = X.with_columns(y)

# The idea of this project is to analyze bias in artificial intelligence

# Convert necessary columns to Pandas for visualization
df_pandas = df.to_pandas()

# Function to compute proportion of >50K income in a given group
def compute_income_proportion(column):
    return df_pandas.groupby(column)['income'].value_counts(normalize=True).unstack()[">50K"]

# Analyze Income Distribution by Gender
gender_income = compute_income_proportion("sex")
print("Income Distribution by Gender:")
print(gender_income)

# Analyze Income Distribution by Race
race_income = compute_income_proportion("race")
print("Income Distribution by Race:")
print(race_income)

# Analyze Income Distribution by Education Level
education_income = compute_income_proportion("education")
print("Income Distribution by Education Level:")
print(education_income)

# Analyze Income Distribution by Occupation
occupation_income = compute_income_proportion("occupation")
print("Income Distribution by Occupation:")
print(occupation_income)

# Visualization: Bar plots for each demographic analysis
fig, axes = plt.subplots(2, 2, figsize=(15, 10))

# Gender
sns.barplot(x=gender_income.index, y=gender_income.values, ax=axes[0, 0], palette='coolwarm')
axes[0, 0].set_title("Income Distribution by Gender")
axes[0, 0].set_ylabel("Proportion earning >50K")

# Race
sns.barplot(x=race_income.index, y=race_income.values, ax=axes[0, 1], palette='coolwarm')
axes[0, 1].set_title("Income Distribution by Race")
axes[0, 1].set_ylabel("Proportion earning >50K")
axes[0, 1].set_xticklabels(axes[0, 1].get_xticklabels(), rotation=45)

# Education
sns.barplot(x=education_income.index, y=education_income.values, ax=axes[1, 0], palette='coolwarm')
axes[1, 0].set_title("Income Distribution by Education Level")
axes[1, 0].set_ylabel("Proportion earning >50K")
axes[1, 0].set_xticklabels(axes[1, 0].get_xticklabels(), rotation=45)

# Occupation
sns.barplot(x=occupation_income.index, y=occupation_income.values, ax=axes[1, 1], palette='coolwarm')
axes[1, 1].set_title("Income Distribution by Occupation")
axes[1, 1].set_ylabel("Proportion earning >50K")
axes[1, 1].set_xticklabels(axes[1, 1].get_xticklabels(), rotation=45)

plt.tight_layout()
plt.show()

# Statistical Testing: Chi-Square test for independence between categorical variables and income
def chi_square_test(column):
    contingency_table = pd.crosstab(df_pandas[column], df_pandas['income'])
    chi2, p, dof, expected = stats.chi2_contingency(contingency_table)
    return chi2, p

# Perform Chi-Square Tests
demographics = ['sex', 'race', 'education', 'occupation']
chi_square_results = {col: chi_square_test(col) for col in demographics}

# Print Results
for col, (chi2, p) in chi_square_results.items():
    print(f"Chi-Square Test for {col}:")
    print(f"Chi2 Statistic: {chi2:.4f}, P-Value: {p:.4f}")
    print("---")

Income Distribution by Gender:
sex
Female    0.072814
Male      0.204043
Name: >50K, dtype: float64
Income Distribution by Race:
race
Amer-Indian-Eskimo    0.076596
Asian-Pac-Islander    0.181698
Black                 0.082604
Other                 0.061576
White                 0.170418
Name: >50K, dtype: float64
Income Distribution by Education Level:
education
10th            0.044636
11th            0.033113
12th            0.050228
1st-4th         0.024291
5th-6th         0.031434
7th-8th         0.041885
9th             0.035714
Assoc-acdm      0.165522
Assoc-voc       0.175158
Bachelors       0.276760
Doctorate       0.515152
HS-grad         0.106120
Masters         0.360933
Preschool            NaN
Prof-school     0.507194
Some-college    0.127505
Name: >50K, dtype: float64
Income Distribution by Occupation:
occupation
?                    0.103635
Adm-clerical         0.090358
Armed-Forces         0.066667
Craft-repair         0.151996
Exec-managerial      0.323365
Farming-fishing      0.077181
Handlers-cleaners    0.041506
Machine-op-inspct    0.082727
Other-service        0.027829
Priv-house-serv      0.004132
Prof-specialty       0.301199
Protective-serv      0.214649
Sales                0.178597
Tech-support         0.195712
Transport-moving     0.135881
Name: >50K, dtype: float64

/tmp/ipykernel_130/282010350.py:50: FutureWarning: 

Passing `palette` without assigning `hue` is deprecated and will be removed in v0.14.0. Assign the `x` variable to `hue` and set `legend=False` for the same effect.

  sns.barplot(x=gender_income.index, y=gender_income.values, ax=axes[0, 0], palette='coolwarm')
/tmp/ipykernel_130/282010350.py:55: FutureWarning: 

Passing `palette` without assigning `hue` is deprecated and will be removed in v0.14.0. Assign the `x` variable to `hue` and set `legend=False` for the same effect.

  sns.barplot(x=race_income.index, y=race_income.values, ax=axes[0, 1], palette='coolwarm')
/tmp/ipykernel_130/282010350.py:58: UserWarning: set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.
  axes[0, 1].set_xticklabels(axes[0, 1].get_xticklabels(), rotation=45)
/tmp/ipykernel_130/282010350.py:61: FutureWarning: 

Passing `palette` without assigning `hue` is deprecated and will be removed in v0.14.0. Assign the `x` variable to `hue` and set `legend=False` for the same effect.

  sns.barplot(x=education_income.index, y=education_income.values, ax=axes[1, 0], palette='coolwarm')
/tmp/ipykernel_130/282010350.py:64: UserWarning: set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.
  axes[1, 0].set_xticklabels(axes[1, 0].get_xticklabels(), rotation=45)
/tmp/ipykernel_130/282010350.py:67: FutureWarning: 

Passing `palette` without assigning `hue` is deprecated and will be removed in v0.14.0. Assign the `x` variable to `hue` and set `legend=False` for the same effect.

  sns.barplot(x=occupation_income.index, y=occupation_income.values, ax=axes[1, 1], palette='coolwarm')
/tmp/ipykernel_130/282010350.py:70: UserWarning: set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.
  axes[1, 1].set_xticklabels(axes[1, 1].get_xticklabels(), rotation=45)

Chi-Square Test for sex:
Chi2 Statistic: 2250.0324, P-Value: 0.0000
---
Chi-Square Test for race:
Chi2 Statistic: 494.4066, P-Value: 0.0000
---
Chi-Square Test for education:
Chi2 Statistic: 6565.7480, P-Value: 0.0000
---
Chi-Square Test for occupation:
Chi2 Statistic: 6804.7619, P-Value: 0.0000
---

Bias Notebook (Dark Theme)

A Markdown-like Cell