Jan Kirenz - Using Keras for Structured Data

This tutorial is mainly based on the Keras tutorial “Structured data classification from scratch” by François Chollet and “Classify structured data using Keras preprocessing layers” by TensorFlow.

This tutorial demonstrates how to do structured binary classification with Keras (Version 3), starting from a raw CSV file.

Setup

First of all, we use Anaconda to create a new environment called keras. Open your terminal (macOS) or your Anaconda Command Prompt (Windows) and enter:

conda create -n keras python=3.11 pip

Activate the environment:

conda activate keras

Let’s install some packages.

pip install ipykernel jupyter pandas numpy tensorflow keras pydot

If you are using Visual Studio Code, you first need to restart Vode before you can slect the new environment in your kernel picker.

You also need to install graphviz (see instructions at https://graphviz.gitlab.io/download/) to plot the model architecture.

import numpy as np
import pandas as pd

import tensorflow as tf
from tensorflow.keras import layers

tf.__version__

Data

We use the features below to predict whether a patient has a heart disease (Target).

featureumn	Description	Feature Type
Age	Age in years	Numerical
Sex	(1 = male; 0 = female)	Categorical
CP	Chest pain type (0, 1, 2, 3, 4)	Categorical
Trestbpd	Resting blood pressure (in mm Hg on admission)	Numerical
Chol	Serum cholesterol in mg/dl	Numerical
FBS	fasting blood sugar in 120 mg/dl (1 = true; 0 = false)	Categorical
RestECG	Resting electrocardiogram results (0, 1, 2)	Categorical
Thalach	Maximum heart rate achieved	Numerical
Exang	Exercise induced angina (1 = yes; 0 = no)	Categorical
Oldpeak	ST depression induced by exercise relative to rest	Numerical
Slope	Slope of the peak exercise ST segment	Numerical
CA	Number of major vessels (0-3) featureored by fluoroscopy	Both numerical & categorical
Thal	normal; fixed defect; reversible defect	Categorical (string)
Target	Diagnosis of heart disease (1 = true; 0 = false)	Target

Data import

Let’s download the data and load it into a Pandas dataframe:

file_url = "http://storage.googleapis.com/download.tensorflow.org/data/heart.csv"
df = pd.read_csv(file_url)

df.head()

df.info()

Define label

Define outcome variable as y_label

y_label = 'target'

Data format

First, we make some changes to our data
Due to computational performance reasons we change:
- int64 to int32
- float64 to float32

# Make a dictionary with int64 featureumns as keys and np.int32 as values
int_32 = dict.fromkeys(df.select_dtypes(np.int64).columns, np.int32)
# Change all columns from dictionary
df = df.astype(int_32)

# Make a dictionary with float64 columns as keys and np.float32 as values
float_32 = dict.fromkeys(df.select_dtypes(np.float64).columns, np.float32)
df = df.astype(float_32)

int_32

Next, we take care of our categorical data:

# Convert to string
df['thal'] = df['thal'].astype("string")

# Convert to categorical

# make a list of all categorical variables
cat_convert = ['sex', 'cp', 'fbs', 'restecg', 'exang', 'ca']

# convert variables
for i in cat_convert:
    df[i] = df[i].astype("category")

Finally, we make lists of feature variables for later data preprocessing steps
Since we don’t want to include our label in our data preprocessing steps, we make sure to exclude it

# Make list of all numerical data (except label)
list_num = df.drop(columns=[y_label]).select_dtypes(include=[np.number]).columns.tolist()

# Make list of all categorical data which is stored as integers (except label)
list_cat_int = df.drop(columns=[y_label]).select_dtypes(include=['category']).columns.tolist()

# Make list of all categorical data which is stored as string (except label)
list_cat_string = df.drop(columns=[y_label]).select_dtypes(include=['string']).columns.tolist()

df.info()

Data splitting

Let’s split the data into a training and validation set

# Make validation data
df_val = df.sample(frac=0.2, random_state=1337)

# Create training data
df_train = df.drop(df_val.index)

# Save training data
df_train.to_csv("df_train.csv", index=False)

print(
    "Using %d samples for training and %d for validation"
    % (len(df_train), len(df_val))
)

Transform to Tensors

Let’s generate tf.data.Dataset objects for our training and validation dataframes
The following utility function converts each training and validation set into a tf.data.Dataset, then shuffles and batches the data.

# Define a function to create our tensors

def dataframe_to_dataset(dataframe, shuffle=True, batch_size=32):
    df = dataframe.copy()
    labels = df.pop(y_label)
    ds = tf.data.Dataset.from_tensor_slices((dict(df), labels))
    if shuffle:
        ds = ds.shuffle(buffer_size=len(dataframe))
    ds = ds.batch(batch_size)
    df = ds.prefetch(batch_size)
    return ds

Next, we use our function to create batches of tensors with size 32

batch_size = 32

ds_train = dataframe_to_dataset(df_train, shuffle=True, batch_size=batch_size)
ds_val = dataframe_to_dataset(df_val, shuffle=True, batch_size=batch_size)

Feature preprocessing

Next, we define utility functions to do the feature preprocessing operations.
In this tutorial, you will use the following preprocessing layers to demonstrate how to perform preprocessing, structured data encoding, and feature engineering:
- tf.keras.layers.Normalization: Performs feature-wise normalization of input features.
- tf.keras.layers.CategoryEncoding: Turns integer categorical features into one-hot, multi-hot, or tf-idf dense representations.
- tf.keras.layers.StringLookup: Turns string categorical values into integer indices.
- tf.keras.layers.IntegerLookup: Turns integer categorical values into integer indices.

Numerical preprocessing function

Define a new utility function that returns a layer which applies feature-wise normalization to numerical features using that Keras preprocessing layer:

# Define numerical preprocessing function
def get_normalization_layer(name, dataset):
    
    # Create a Normalization layer for our feature
    normalizer = layers.Normalization(axis=None)

    # Prepare a dataset that only yields our feature
    feature_ds = dataset.map(lambda x, y: x[name])

    # Learn the statistics of the data
    normalizer.adapt(feature_ds)

    # Normalize the input feature
    return normalizer

Categorical preprocessing functions

Define another new utility function that returns a layer which maps values from a vocabulary to integer indices and multi-hot encodes the features using the preprocessing layers:
- layers.StringLookup
- layers.IntegerLookup
- layersCategoryEncoding
If the vocabulary is capped in size (max_tokens), the most frequent tokens will be used to create the vocabulary and all others will be treated as out-of-vocabulary (OOV).

def get_category_encoding_layer(name, dataset, dtype, max_tokens=None):
  
  # Create a layer that turns strings into integer indices.
  if dtype == 'string':
    index = layers.StringLookup(max_tokens=max_tokens)
  # Otherwise, create a layer that turns integer values into integer indices.
  else:
    index = layers.IntegerLookup(max_tokens=max_tokens)

  # Prepare a `tf.data.Dataset` that only yields the feature.
  feature_ds = dataset.map(lambda x, y: x[name])

  # Learn the set of possible values and assign them a fixed integer index.
  index.adapt(feature_ds)

  # Encode the integer indices.
  encoder = layers.CategoryEncoding(num_tokens=index.vocabulary_size())

  # Apply multi-hot encoding to the indices. The lambda function captures the
  # layer, so you can use them, or include them in the Keras Functional model later.
  return lambda feature: encoder(index(feature))

Data preprocessing

Next, we will:

Apply the preprocessing utility functions defined earlier on our numerical and categorical features and store it into a list called encoded_features
We also add all of our features to a list called all_inputs.
Let’s start by preparing two empty lists:

all_inputs = []
encoded_features = []

Numerical preprocessing

For every feature:
- create a tf.keras.Input called numeric_feature
- Normalize the numerical feature
- Add the feature name to the list all_inputs
- Add the encoded numeric feature ro our list encoded_features:

# Numerical features
for feature in list_num:
  numeric_feature = tf.keras.Input(shape=(1,), name=feature)
  normalization_layer = get_normalization_layer(feature, ds_train)
  encoded_numeric_feature = normalization_layer(numeric_feature)
  all_inputs.append(numeric_feature)
  encoded_features.append(encoded_numeric_feature)

Categorical preprocessing

Turn the integer categorical values from the dataset into integer indices, perform multi-hot encoding and add the resulting feature inputs to encoded_feature

for feature in list_cat_int:
  categorical_feature = tf.keras.Input(shape=(1,), name=feature, dtype='int32')
  encoding_layer = get_category_encoding_layer(name=feature,
                                               dataset=ds_train,
                                               dtype='int32',
                                               max_tokens=5)
  encoded_categorical_feature = encoding_layer(categorical_feature)
  all_inputs.append(categorical_feature)
  encoded_features.append(encoded_categorical_feature)

for feature in list_cat_string:
  categorical_feature = tf.keras.Input(shape=(1,), name=feature, dtype='string')
  encoding_layer = get_category_encoding_layer(name=feature,
                                               dataset=ds_train,
                                               dtype='string',
                                               max_tokens=5)
  encoded_categorical_feature = encoding_layer(categorical_feature)
  all_inputs.append(categorical_feature)
  encoded_features.append(encoded_categorical_feature)

Model

Now we can build the model using the Keras Functional API:
1. Merge the list of feature inputs (encoded_features) into one vector via concatenation with layers.concatenate.
2. We use 32 number of units in the first layer
3. We use layers.Dropout() to prevent overvitting
4. Our output layer has 1 output (since the classification task is binary)
5. tf.keras.Model groups layers into an object with training and inference features.

# Input
all_features = layers.concatenate(encoded_features)

# First layer
x = layers.Dense(32, activation="relu")(all_features)

# Dropout to prevent overvitting
x = layers.Dropout(0.5)(x)

# Output layer
output = layers.Dense(1, activation="sigmoid")(x)

# Group all layers 
model = tf.keras.Model(all_inputs, output)

Configure the model with Keras Model.compile:

model.compile(optimizer="adam", 
              loss ="binary_crossentropy", 
              metrics=["accuracy"])

Let’s visualize our connectivity graph:

# `rankdir='LR'` is to make the graph horizontal.
tf.keras.utils.plot_model(model, show_shapes=True, rankdir="LR")

Training

Next, train and test the model:

%%time
model.fit(ds_train, epochs=10, validation_data=ds_val)

We quickly get to arounf 80% validation accuracy.

loss, accuracy = model.evaluate(ds_val)

print("Accuracy", round(accuracy, 2))

Perform inference

The model you have developed can now classify a row from a CSV file directly after you’ve included the preprocessing layers inside the model itself.
Next, we demonstrate the process.
First, save the heart diseases classification model

model.save('my_hd_classifier')

Load the model (we call it reloaded_model):

reloaded_model = tf.keras.models.load_model('my_hd_classifier')

To get a prediction for a new sample, you can simply call the Keras Model.predict method.
There are just two things you need to do:
- Wrap scalars into a list so as to have a batch dimension (Models only process batches of data, not single samples).
- Call tf.convert_to_tensor on each feature.

sample = {
    "age": 60,
    "sex": 1,
    "cp": 1,
    "trestbps": 145,
    "chol": 233,
    "fbs": 1,
    "restecg": 2,
    "thalach": 150,
    "exang": 0,
    "oldpeak": 2.3,
    "slope": 3,
    "ca": 0,
    "thal": "fixed",
}

input_dict = {name: tf.convert_to_tensor([value]) for name, value in sample.items()}

predictions = reloaded_model.predict(input_dict)

print(
    "This particular patient had a %.1f percent probability "
    "of having a heart disease, as evaluated by our model." % (100 * predictions[0][0],)
)

Next steps

To learn more about classifying structured data, try working with other datasets. Below are some suggestions for datasets:

TensorFlow Datasets: MovieLens: A set of movie ratings from a movie recommendation service.
TensorFlow Datasets: Wine Quality: Two datasets related to red and white variants of the Portuguese “Vinho Verde” wine. You can also find the Red Wine Quality dataset on Kaggle.
Kaggle: arXiv Dataset: A corpus of 1.7 million scholarly articles from arXiv, covering physics, computer science, math, statistics, electrical engineering, quantitative biology, and economics.