Tutorial 5. Advanced Model Composition

Goals:

1. Learn how to build a prototypes of a composite model, called learning networks
2. Learn how to "export" a learning network as a new stand-alone model type

To run the code in this tutorial in a live Julia session, first follow the instructions given here.

Pipelines are great for composing models in an unbranching sequence. Another built-in type of model composition is a model stack; see here for details. For other more complicated model compositions you'll want to use MLJ's generic model composition syntax.

There are two main steps:

• Prototype the composite model by building a learning network, which can be tested on some (dummy) data as you build it.

• Export the learning network as a new stand-alone model type.

Like pipeline models, instances of the exported model type behave like any other model (and are not bound to any data, until you wrap them in a machine).

Building a pipeline using the generic composition syntax

using MLJ
LogisticClassifier = @load LogisticClassifier pkg=MLJLinearModels

MLJLinearModels.LogisticClassifier

To warm up, we'll build a learning network to replace this basic pipeline model:

pipe = Standardizer |> LogisticClassifier(lambda=0.001)

Here's some dummy data we'll be using to test our learning network:

X, y = make_blobs(5, 3)
pretty(X)

┌────────────┬────────────┬────────────┐
│ x1         │ x2         │ x3         │
│ Float64    │ Float64    │ Float64    │
│ Continuous │ Continuous │ Continuous │
├────────────┼────────────┼────────────┤
│ -5.74465   │ 2.46153    │ -2.99641   │
│ -5.84665   │ 2.092      │ -0.271891  │
│ -13.83     │ 0.276595   │ 4.05293    │
│ -12.237    │ 9.43369    │ 5.14761    │
│ -7.77455   │ 0.941789   │ -0.369465  │
└────────────┴────────────┴────────────┘

Step 0 - Proceed as if you were combining the models "by hand", using all the data available for training, transformation and prediction:

standardizer = Standardizer();
linear = LogisticClassifier(lambda=0.001);

mach1 = machine(standardizer, X);
fit!(mach1);
Xstand = transform(mach1, X);

mach2 = machine(linear, Xstand, y);
fit!(mach2);
yhat = predict(mach2, Xstand)

5-element CategoricalDistributions.UnivariateFiniteVector{ScientificTypesBase.Multiclass{3}, Int64, UInt32, Float64}:
 UnivariateFinite{ScientificTypesBase.Multiclass{3}}(1=>2.75e-5, 2=>0.000152, 3=>1.0)
 UnivariateFinite{ScientificTypesBase.Multiclass{3}}(1=>0.000372, 2=>0.00087, 3=>0.999)
 UnivariateFinite{ScientificTypesBase.Multiclass{3}}(1=>0.993, 2=>0.00215, 3=>0.00446)
 UnivariateFinite{ScientificTypesBase.Multiclass{3}}(1=>0.00352, 2=>0.995, 3=>0.00124)
 UnivariateFinite{ScientificTypesBase.Multiclass{3}}(1=>0.00453, 2=>0.000961, 3=>0.995)

Step 1 - Edit your code as follows:

• wrap the data in Source nodes

• delete the fit! calls

X = source(X)  # or X = source() if not testing
y = source(y)  # or y = source()

standardizer = Standardizer();
linear = LogisticClassifier(lambda=0.001);

mach1 = machine(standardizer, X);
Xstand = transform(mach1, X);

mach2 = machine(linear, Xstand, y);
yhat = predict(mach2, Xstand)

Node @121 → LogisticClassifier(…)
  args:
    1:	Node @394 → Standardizer(…)
  formula:
    predict(
      machine(LogisticClassifier(lambda = 0.001, …), …), 
      transform(
        machine(Standardizer(features = Symbol[], …), …), 
        Source @574,
      ),
    )

Now X, y, Xstand and yhat are nodes ("variables" or "dynamic data") instead of data. All training, predicting and transforming is now executed lazily, whenever we fit! one of these nodes. We call a node to retrieve the data it represents in the original manual workflow.

fit!(Xstand)
Xstand() |> pretty

[ Info: Training machine(Standardizer(features = Symbol[], …), …).
┌────────────┬────────────┬────────────┐
│ x1         │ x2         │ x3         │
│ Float64    │ Float64    │ Float64    │
│ Continuous │ Continuous │ Continuous │
├────────────┼────────────┼────────────┤
│ 0.894666   │ -0.157523  │ -1.21267   │
│ 0.867361   │ -0.257956  │ -0.408588  │
│ -1.26987   │ -0.751356  │ 0.867785   │
│ -0.8434    │ 1.7374     │ 1.19086    │
│ 0.351241   │ -0.570566  │ -0.437385  │
└────────────┴────────────┴────────────┘

fit!(yhat); # training is smart and so `Standardizer` is not retrained

[ Info: Not retraining machine(Standardizer(features = Symbol[], …), …). Use `force=true` to force.
[ Info: Training machine(LogisticClassifier(lambda = 0.001, …), …).
┌ Info: Solver: MLJLinearModels.LBFGS{Optim.Options{Float64, Nothing}, @NamedTuple{}}
│   optim_options: Optim.Options{Float64, Nothing}
└   lbfgs_options: @NamedTuple{} NamedTuple()

yhat()

5-element CategoricalDistributions.UnivariateFiniteVector{ScientificTypesBase.Multiclass{3}, Int64, UInt32, Float64}:
 UnivariateFinite{ScientificTypesBase.Multiclass{3}}(1=>2.75e-5, 2=>0.000152, 3=>1.0)
 UnivariateFinite{ScientificTypesBase.Multiclass{3}}(1=>0.000372, 2=>0.00087, 3=>0.999)
 UnivariateFinite{ScientificTypesBase.Multiclass{3}}(1=>0.993, 2=>0.00215, 3=>0.00446)
 UnivariateFinite{ScientificTypesBase.Multiclass{3}}(1=>0.00352, 2=>0.995, 3=>0.00124)
 UnivariateFinite{ScientificTypesBase.Multiclass{3}}(1=>0.00453, 2=>0.000961, 3=>0.995)

The node yhat is the "descendant" (in an associated DAG we have defined) of a unique source node:

origins(yhat)

1-element Vector{MLJBase.Source}:
 Source @574 ⏎ `ScientificTypesBase.Table{AbstractVector{ScientificTypesBase.Continuous}}`

The data at the source node is replaced by Xnew to obtain a new prediction when we call yhat like this:

Xnew, _ = make_blobs(2, 3);
yhat(Xnew)

2-element CategoricalDistributions.UnivariateFiniteVector{ScientificTypesBase.Multiclass{3}, Int64, UInt32, Float64}:
 UnivariateFinite{ScientificTypesBase.Multiclass{3}}(1=>0.000861, 2=>2.17e-12, 3=>0.999)
 UnivariateFinite{ScientificTypesBase.Multiclass{3}}(1=>1.11e-5, 2=>2.65e-9, 3=>1.0)

Step 2 - Export the learning network as a new stand-alone model type

We start by defining a new model type for our composite. We subtype ProbabilisticNetworkComposite because our composite is to be a probabilistic predictor. If it were a deterministic predictor, we would use DeterministicNetworkComposite instead. There is also a UnsupervisedNetworkComposite for transformers.

mutable struct YourPipe <: ProbabilisticNetworkComposite
    standardizer
    classifier
end

Next, we make our learning network above generic by substituting each model instance with the corresponding symbol representing a property (field) of the new model struct:

mach1 = machine(:standardizer, X);
Xstand = transform(mach1, X);

mach2 = machine(:classifier, Xstand, y);
yhat = predict(mach2, Xstand)

Node @768 → :classifier
  args:
    1:	Node @820 → :standardizer
  formula:
    predict(
      machine(:classifier, …), 
      transform(
        machine(:standardizer, …), 
        Source @574,
      ),
    )

Incidentally, this network can be used as before except we must provide an instance of YourPipe in our fit! calls, to indicate which models replace the symbols:

your_pipe = YourPipe(standardizer, linear)
fit!(yhat, composite=your_pipe);

[ Info: Training machine(:standardizer, …).
[ Info: Training machine(:classifier, …).
┌ Info: Solver: MLJLinearModels.LBFGS{Optim.Options{Float64, Nothing}, @NamedTuple{}}
│   optim_options: Optim.Options{Float64, Nothing}
└   lbfgs_options: @NamedTuple{} NamedTuple()

In this case :standardizer is being substituted by standardizer and :classifier by linear in training.

The final step is to wrap our learning network code in a method called prefit dispatched on YourPipe. This method returns a "learning network interface" which is a named tuple telling the method which node of the network returns predictions for the composite model.

import MLJ.MLJBase
function MLJBase.prefit(composite::YourPipe, verbosity, X, y)
    # the learning network from above:
    X = source(X)
    y = source(y)
    mach1 = machine(:standardizer, X);
    Xstand = transform(mach1, X);
    mach2 = machine(:classifier, Xstand, y);
    yhat = predict(mach2, Xstand)

    verbosity > 0 && @info "I'm a noisy fellow!"

    # return "learning network interface":
    return (; predict=yhat)
end

Instantiating and training on some new data:

pipe = YourPipe(standardizer, linear)
X, y = @load_iris;   # built-in data set
mach = machine(pipe, X, y)
fit!(mach);

[ Info: Training machine(YourPipe(standardizer = Standardizer(features = Symbol[], …), …), …).
[ Info: I'm a noisy fellow!
[ Info: Training machine(:standardizer, …).
[ Info: Training machine(:classifier, …).
┌ Info: Solver: MLJLinearModels.LBFGS{Optim.Options{Float64, Nothing}, @NamedTuple{}}
│   optim_options: Optim.Options{Float64, Nothing}
└   lbfgs_options: @NamedTuple{} NamedTuple()

The learned parameters and report (where non-empty) for each component model are accessible:

fitted_params(mach).classifier.coefs

4-element Vector{Pair{Symbol, SubArray{Float64, 1, Matrix{Float64}, Tuple{Int64, Base.Slice{Base.OneTo{Int64}}}, true}}}:
 :sepal_length => [-2.2884323728871254, 1.4607388496829115, 0.8276935232042247]
  :sepal_width => [2.5849165395349716, -0.6646830918858928, -1.9202334476490799]
 :petal_length => [-3.5313085896004583, -1.406180852406463, 4.937489442006912]
  :petal_width => [-3.4480076498857017, -1.6977190775004856, 5.145726727386192]

report(mach).standardizer

(features_fit = [:sepal_length, :sepal_width, :petal_length, :petal_width],)

Component models can be swapped out for new ones:

pipe.classifier = ConstantClassifier()
fit!(mach)
fitted_params(mach).classifier.target_distribution

(CategoricalArrays.CategoricalValue{String, UInt32}[CategoricalValue(CategoricalArrays.CategoricalPool{String, UInt32}(["setosa", "versicolor", "virginica"]), 1), CategoricalValue(CategoricalArrays.CategoricalPool{String, UInt32}(["setosa", "versicolor", "virginica"]), 2), CategoricalValue(CategoricalArrays.CategoricalPool{String, UInt32}(["setosa", "versicolor", "virginica"]), 3)], [0.3333333333333333 0.3333333333333333 0.3333333333333333])

A composite model to average two regressor predictors

Next, we define a composite model that:

• standardizes the input data
• learns and applies a Box-Cox transformation to the target variable
• averages the predictions of two supervised learning models - a ridge regressor and a random forest regressor - using a simple average
• applies the inverse Box-Cox transformation to this blended prediction

We'll start with a learning network, with source nodes bound to some dummy test data:

RandomForestRegressor = @load RandomForestRegressor pkg=DecisionTree
RidgeRegressor = @load RidgeRegressor pkg=MLJLinearModels

MLJLinearModels.RidgeRegressor

Input layer with dummy data:

X, y = make_regression()
y = abs.(y)
X = source(X)
y = source(y)

Source @332 ⏎ `AbstractVector{ScientificTypesBase.Continuous}`

First layer and target transformation:

standardizer = Standardizer()
mach1 = machine(standardizer, X)
W = MLJ.transform(mach1, X)

box_model = UnivariateBoxCoxTransformer()
mach2 = machine(box_model, y)
z = MLJ.transform(mach2, y)

Node @907 → UnivariateBoxCoxTransformer(…)
  args:
    1:	Source @332
  formula:
    transform(
      machine(UnivariateBoxCoxTransformer(n = 171, …), …), 
      Source @332,
    )

Second layer:

regressor1 = RidgeRegressor(lambda=0.1)
mach3 = machine(regressor1, W, z)

regressor2 = RandomForestRegressor(n_trees=50)
mach4 = machine(regressor2, W, z)

zhat = 0.5*predict(mach3, W) + 0.5*predict(mach4, W)

Node @983
  args:
    1:	Node @241
    2:	Node @953
  formula:
    +(
     var"#*##0#*##1"(
       predict(
         machine(RidgeRegressor(lambda = 0.1, …), …), 
         transform(
           machine(Standardizer(features = Symbol[], …), …), 
           Source @902,
         ),
       ),
     ),
     var"#*##0#*##1"(
       predict(
         machine(RandomForestRegressor(max_depth = -1, …), …), 
         transform(
           machine(Standardizer(features = Symbol[], …), …), 
           Source @902,
         ),
       ),
     ),
    )

Output:

yhat = inverse_transform(mach2, zhat)

Node @508 → UnivariateBoxCoxTransformer(…)
  args:
    1:	Node @983
  formula:
    inverse_transform(
      machine(UnivariateBoxCoxTransformer(n = 171, …), …), 
      +(
       var"#*##0#*##1"(
         predict(
           machine(RidgeRegressor(lambda = 0.1, …), …), 
           transform(
             machine(Standardizer(features = Symbol[], …), …), 
             Source @902,
           ),
         ),
       ),
       var"#*##0#*##1"(
         predict(
           machine(RandomForestRegressor(max_depth = -1, …), …), 
           transform(
             machine(Standardizer(features = Symbol[], …), …), 
             Source @902,
           ),
         ),
       ),
      ),
    )

Let's test this learning network (always a good idea!):

fit!(yhat)
yhat(rows=1:3)

3-element Vector{Float64}:
 0.41876708160520987
 0.8130133557286682
 1.7366068735520672

Now for the new model type:

mutable struct CompositeModel <: DeterministicNetworkComposite
    standardizer
    box_cox
    regressor1
    regressor2
end

And the prefit function wrapping our learning network code, with model substitutions

function MLJBase.prefit(composite::CompositeModel, verbosity, X, y)
    X = source(X)
    y = source(y)

    # First layer and target transformation:
    mach1 = machine(:standardizer, X)
    W = MLJ.transform(mach1, X)
    mach2 = machine(:box_cox, y)
    z = MLJ.transform(mach2, y)

    # Second layer:
    mach3 = machine(:regressor1, W, z)
    mach4 = machine(:regressor2, W, z)
    zhat = 0.5*predict(mach3, W) + 0.5*predict(mach4, W)

    # Output:
    yhat = inverse_transform(mach2, zhat)

    return (; predict=yhat)
end

We instantiate the new model type and try it out on some new data:

composite = CompositeModel(standardizer, box_model, regressor1, regressor2)
X, y = @load_boston
evaluate(composite, X, y; resampling=CV(nfolds=6, shuffle=true), measures=[rms, mae])

PerformanceEvaluation object with these fields:
  model, tag, measure, operation,
  measurement, uncertainty_radius_95, per_fold, per_observation,
  fitted_params_per_fold, report_per_fold,
  train_test_rows, resampling, repeats
Tag: CompositeModel-892
Extract:
┌───┬────────────────────────┬───────────┬─────────────┐
│   │ measure                │ operation │ measurement │
├───┼────────────────────────┼───────────┼─────────────┤
│ A │ RootMeanSquaredError() │ predict   │ 3.96        │
│ B │ LPLoss(                │ predict   │ 2.54        │
│   │   p = 1)               │           │             │
└───┴────────────────────────┴───────────┴─────────────┘
┌───┬──────────────────────────────────────┬─────────┐
│   │ per_fold                             │ 1.96*SE │
├───┼──────────────────────────────────────┼─────────┤
│ A │ [4.85, 4.11, 3.08, 4.19, 3.77, 3.48] │ 0.539   │
│ B │ [2.73, 2.49, 2.33, 2.82, 2.37, 2.49] │ 0.171   │
└───┴──────────────────────────────────────┴─────────┘

Tutorial 5 Resources

• From the MLJ manual:
  • Learning Networks
• From Data Science Tutorials:
  • Model ensembles via learning networks
  • Model stacking via learning networks

This page was generated using Literate.jl.