Hyperparameter Selection 2ΒΆ

This example demonstrates how to do model selection in a pipeline where segments are learned directly by a neural network

Grid Search Scores

Out:

/home/circleci/miniconda/envs/testenv/lib/python3.8/site-packages/seglearn/split.py:100: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray
  Xt_new = np.array(Xt_new)
/home/circleci/miniconda/envs/testenv/lib/python3.8/site-packages/sklearn/utils/validation.py:67: FutureWarning: Pass memory=None as keyword args. From version 0.25 passing these as positional arguments will result in an error
  warnings.warn("Pass {} as keyword args. From version 0.25 "
Fitting 3 folds for each of 3 candidates, totalling 9 fits
[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.
[CV] crnn__width=seg__width, seg__overlap=0.0, seg__width=100 ........
/home/circleci/miniconda/envs/testenv/lib/python3.8/site-packages/seglearn/transform.py:237: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray
  Xt = np.array([sliding_tensor(Xt[i], self.width, self._step, self.order)
[CV]  crnn__width=seg__width, seg__overlap=0.0, seg__width=100, total=   3.6s
[Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:    3.6s remaining:    0.0s
[CV] crnn__width=seg__width, seg__overlap=0.0, seg__width=100 ........
[CV]  crnn__width=seg__width, seg__overlap=0.0, seg__width=100, total=   3.8s
[CV] crnn__width=seg__width, seg__overlap=0.0, seg__width=100 ........
[CV]  crnn__width=seg__width, seg__overlap=0.0, seg__width=100, total=   3.7s
[CV] crnn__width=seg__width, seg__overlap=0.0, seg__width=200 ........
[CV]  crnn__width=seg__width, seg__overlap=0.0, seg__width=200, total=   4.0s
[CV] crnn__width=seg__width, seg__overlap=0.0, seg__width=200 ........
[CV]  crnn__width=seg__width, seg__overlap=0.0, seg__width=200, total=   3.8s
[CV] crnn__width=seg__width, seg__overlap=0.0, seg__width=200 ........
[CV]  crnn__width=seg__width, seg__overlap=0.0, seg__width=200, total=   4.0s
[CV] crnn__width=seg__width, seg__overlap=0.0, seg__width=400 ........
[CV]  crnn__width=seg__width, seg__overlap=0.0, seg__width=400, total=   4.2s
[CV] crnn__width=seg__width, seg__overlap=0.0, seg__width=400 ........
[CV]  crnn__width=seg__width, seg__overlap=0.0, seg__width=400, total=   4.6s
[CV] crnn__width=seg__width, seg__overlap=0.0, seg__width=400 ........
[CV]  crnn__width=seg__width, seg__overlap=0.0, seg__width=400, total=   4.4s
[Parallel(n_jobs=1)]: Done   9 out of   9 | elapsed:   36.1s finished

# Author: David Burns
# License: BSD

import matplotlib.pyplot as plt
from tensorflow.python.keras.layers import Dense, LSTM, Conv1D
from tensorflow.python.keras.models import Sequential
from tensorflow.python.keras.wrappers.scikit_learn import KerasClassifier

from sklearn.model_selection import GridSearchCV

from seglearn.datasets import load_watch
from seglearn.pipe import Pype
from seglearn.split import TemporalKFold
from seglearn.transform import Segment


def crnn_model(width=100, n_vars=6, n_classes=7, conv_kernel_size=5,
               conv_filters=2, lstm_units=2):
    # create a crnn model with keras with one cnn layers, and one rnn layer
    input_shape = (width, n_vars)
    model = Sequential()
    model.add(Conv1D(filters=conv_filters, kernel_size=conv_kernel_size,
                     padding='valid', activation='relu', input_shape=input_shape))
    model.add(LSTM(units=lstm_units, dropout=0.1, recurrent_dropout=0.1))
    model.add(Dense(n_classes, activation="softmax"))

    model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

    return model


# load the data
data = load_watch()
X = data['X']
y = data['y']

# temporal splitting of data
splitter = TemporalKFold(n_splits=3)
Xs, ys, cv = splitter.split(X, y)

# create a segment learning pipeline
pipe = Pype([('seg', Segment(order='C')),
             ('crnn', KerasClassifier(build_fn=crnn_model, epochs=1, batch_size=256, verbose=0))])

# create a parameter dictionary using the sklearn API
#
# you can also set a parameter to be always equal to another parameter, by setting its value to
# parameter name to track (this is an extension from sklearn)
#
# note that if you want to set a parameter to a single value, it will still need to be as a list

par_grid = {'seg__width': [100, 200, 400],
            'seg__overlap': [0.],
            'crnn__width': ['seg__width']}

clf = GridSearchCV(pipe, par_grid, cv=cv, verbose=2)
clf.fit(Xs, ys)
scores = clf.cv_results_['mean_test_score']
stds = clf.cv_results_['std_test_score']

plt.plot(par_grid['seg__width'], scores, '-o')
plt.title("Grid Search Scores")
plt.xlabel("Width [s]")
plt.ylabel("CV Average Score")
plt.fill_between(par_grid['seg__width'], scores - stds, scores + stds, alpha=0.2, color='navy')
plt.show()

Total running time of the script: ( 0 minutes 40.049 seconds)

Gallery generated by Sphinx-Gallery