Symbolic Machine Learning¶

Machine learning (ML) is sensitive to neural architectures and hyperparameters, making their representation critical to the design of an ML system. ML practitioners rely on these representations to conduct experiments and deploy models in production.

In a traditional ML system, ML components are built to serve the system's purpose, with a separate configuration system used to adjust their hyperparameters. This requires developers and users to maintain a mapping between the components and their configurations, ensuring that they remain consistent. PyGX offers an alternative approach by using symbolic classes to develop ML components. Symbolic objects are a natural and mutable representation of these components, eliminating the need for a separate configuration system. As a result, symbolic ML systems are simpler and more user-friendly. In this Colab, we provide an example of a symbolic ML pipeline that can support the entire experimentation life-cycle with ease and power.

In [ ]:

!pip install pygx

!pip install pygx

An Overview of a Symbolic ML Experiment¶

In PyGX, an ML pipeline (or experiment) can be represented as a symbolic object - an object that can be manipulated safely after their creation based on their construction signatures. The root object is the whole experiment, whose sub-nodes are symbolic objects of ML components that specify the details of the experiment.

All ML components (including the Experiment class) are symbolic, meaning that they can be safely manipulated after creation. With PyGX, users can create a symbolic class by extending pg.Object or symbolizing a regular class via pg.symbolize. The code below shows the definition of Experiment class, and lower-level ML components symbolized from functions or PyTorch classes.

In [2]:

import torch
from torch import nn
from torch import optim
from torch.utils.data import DataLoader, TensorDataset
from torchvision import datasets, transforms
import pygx as pg

# Symbolize regular PyTorch classes to make them serializable and
# manipulable. `pg.symbolize` keeps the original class behavior and adds
# the symbolic representation on top.
Sequential = pg.symbolize(nn.Sequential)
Flatten = pg.symbolize(nn.Flatten)
Linear = pg.symbolize(nn.Linear)
Dropout = pg.symbolize(nn.Dropout)
ReLU = pg.symbolize(nn.ReLU)
CrossEntropyLoss = pg.symbolize(nn.CrossEntropyLoss)


class Optimizer(pg.Object, topo=True):
  """A symbolic optimizer *spec*.

  A `torch.optim.Optimizer` needs the model parameters at construction,
  which would tie it to one model. Describing the optimizer symbolically
  and building it at run time keeps the two independently manipulable —
  so `training.optimizer.learning_rate` stays a knob you can rebind,
  patch, and tune.
  """

  learning_rate: float = 0.001

  def make(self, params) -> optim.Optimizer:
    raise NotImplementedError()


class Adam(Optimizer):

  def make(self, params) -> optim.Optimizer:
    return optim.Adam(params, lr=self.learning_rate)


class Experiment(pg.Object, topo=True):

  model: nn.Module = pg.field(
      value_spec=pg.typing.Object(nn.Module),
      doc='Model used for both training and evaluation.')

  # A nested `pg.typing.Dict` spec keeps the per-key docstrings and
  # constraints, which `Experiment.__schema__` prints below.
  training: dict = pg.field(value_spec=pg.typing.Dict([
      ('input', pg.typing.Callable(),
       'A callable that returns a tuple of (features, labels) for training.'),
      ('batch_size', pg.typing.Int(min_value=1),
       'Training batch size.'),
      ('num_epochs', pg.typing.Int(min_value=1),
       'Number of epochs for training.'),
      ('loss', pg.typing.Object(nn.Module),
       'Loss used for optimization.'),
      ('optimizer', pg.typing.Object(Optimizer),
       'Optimizer spec for minimizing the loss.'),
  ]))

  evaluation: dict = pg.field(value_spec=pg.typing.Dict([
      ('input', pg.typing.Callable(),
       'A callable that returns a tuple of (features, labels) for evaluation.'),
      ('batch_size', pg.typing.Int(min_value=1),
       'Evaluation batch size.'),
  ]))

  def run(self):
    model, training, evaluation = self.model, self.training, self.evaluation
    optimizer = training.optimizer.make(model.parameters())

    features, labels = training.input()
    loader = DataLoader(
        TensorDataset(features, labels),
        batch_size=training.batch_size, shuffle=True)

    model.train()
    for _ in range(training.num_epochs):
      for batch_features, batch_labels in loader:
        optimizer.zero_grad()
        loss = training.loss(model(batch_features), batch_labels)
        loss.backward()
        optimizer.step()

    model.eval()
    features, labels = evaluation.input()
    with torch.no_grad():
      predicted = model(features).argmax(dim=1)
    return (predicted == labels).sum().item() / len(labels)


@pg.symbolize
def mnist(training):
  """Returns MNIST features and labels as tensors."""
  transform = transforms.Compose([
      transforms.ToTensor(),
      transforms.Normalize((0.1307,), (0.3081,)),
  ])
  dataset = datasets.MNIST(
      './data', train=training, download=True, transform=transform)
  # A small subset keeps this notebook quick to run.
  size = 2048 if training else 512
  loader = DataLoader(dataset, batch_size=size, shuffle=False)
  features, labels = next(iter(loader))
  return features, labels

import torch from torch import nn from torch import optim from torch.utils.data import DataLoader, TensorDataset from torchvision import datasets, transforms import pygx as pg # Symbolize regular PyTorch classes to make them serializable and # manipulable. `pg.symbolize` keeps the original class behavior and adds # the symbolic representation on top. Sequential = pg.symbolize(nn.Sequential) Flatten = pg.symbolize(nn.Flatten) Linear = pg.symbolize(nn.Linear) Dropout = pg.symbolize(nn.Dropout) ReLU = pg.symbolize(nn.ReLU) CrossEntropyLoss = pg.symbolize(nn.CrossEntropyLoss) class Optimizer(pg.Object, topo=True): """A symbolic optimizer *spec*. A `torch.optim.Optimizer` needs the model parameters at construction, which would tie it to one model. Describing the optimizer symbolically and building it at run time keeps the two independently manipulable — so `training.optimizer.learning_rate` stays a knob you can rebind, patch, and tune. """ learning_rate: float = 0.001 def make(self, params) -> optim.Optimizer: raise NotImplementedError() class Adam(Optimizer): def make(self, params) -> optim.Optimizer: return optim.Adam(params, lr=self.learning_rate) class Experiment(pg.Object, topo=True): model: nn.Module = pg.field( value_spec=pg.typing.Object(nn.Module), doc='Model used for both training and evaluation.') # A nested `pg.typing.Dict` spec keeps the per-key docstrings and # constraints, which `Experiment.__schema__` prints below. training: dict = pg.field(value_spec=pg.typing.Dict([ ('input', pg.typing.Callable(), 'A callable that returns a tuple of (features, labels) for training.'), ('batch_size', pg.typing.Int(min_value=1), 'Training batch size.'), ('num_epochs', pg.typing.Int(min_value=1), 'Number of epochs for training.'), ('loss', pg.typing.Object(nn.Module), 'Loss used for optimization.'), ('optimizer', pg.typing.Object(Optimizer), 'Optimizer spec for minimizing the loss.'), ])) evaluation: dict = pg.field(value_spec=pg.typing.Dict([ ('input', pg.typing.Callable(), 'A callable that returns a tuple of (features, labels) for evaluation.'), ('batch_size', pg.typing.Int(min_value=1), 'Evaluation batch size.'), ])) def run(self): model, training, evaluation = self.model, self.training, self.evaluation optimizer = training.optimizer.make(model.parameters()) features, labels = training.input() loader = DataLoader( TensorDataset(features, labels), batch_size=training.batch_size, shuffle=True) model.train() for _ in range(training.num_epochs): for batch_features, batch_labels in loader: optimizer.zero_grad() loss = training.loss(model(batch_features), batch_labels) loss.backward() optimizer.step() model.eval() features, labels = evaluation.input() with torch.no_grad(): predicted = model(features).argmax(dim=1) return (predicted == labels).sum().item() / len(labels) @pg.symbolize def mnist(training): """Returns MNIST features and labels as tensors.""" transform = transforms.Compose([ transforms.ToTensor(), transforms.Normalize((0.1307,), (0.3081,)), ]) dataset = datasets.MNIST( './data', train=training, download=True, transform=transform) # A small subset keeps this notebook quick to run. size = 2048 if training else 512 loader = DataLoader(dataset, batch_size=size, shuffle=False) features, labels = next(iter(loader)) return features, labels

With symbolic classes, an experiment can be expressed as an Experiment. And we can run the experiment by calling its run method, and save it for reproduction later.

In [3]:

# Create an instance of Experiment via composition.
exp1 = Experiment(
  model=Sequential(
     Flatten(),
     Linear(784, 1024),
     ReLU(),
     Dropout(0.2),
     Linear(1024, 10),
  ),
  training=pg.Dict(
      input=mnist(training=True),
      batch_size=32,
      num_epochs=2,
      loss=CrossEntropyLoss(),
      optimizer=Adam(),
  ),
  evaluation=pg.Dict(
      input=mnist(training=False),
      batch_size=32,
  ))

# Run and save experiment.
exp1.run()
exp1.save('exp1.json')

# Create an instance of Experiment via composition. exp1 = Experiment( model=Sequential( Flatten(), Linear(784, 1024), ReLU(), Dropout(0.2), Linear(1024, 10), ), training=pg.Dict( input=mnist(training=True), batch_size=32, num_epochs=2, loss=CrossEntropyLoss(), optimizer=Adam(), ), evaluation=pg.Dict( input=mnist(training=False), batch_size=32, )) # Run and save experiment. exp1.run() exp1.save('exp1.json')

  0%|          | 0.00/9.91M [00:00<?, ?B/s]

  0%|          | 32.8k/9.91M [00:00<00:35, 277kB/s]

  1%|          | 98.3k/9.91M [00:00<00:24, 401kB/s]

  2%|▏         | 197k/9.91M [00:00<00:17, 546kB/s] 

  4%|▍         | 426k/9.91M [00:00<00:08, 1.07MB/s]

  7%|▋         | 655k/9.91M [00:00<00:07, 1.29MB/s]

  9%|▊         | 852k/9.91M [00:00<00:07, 1.22MB/s]

 17%|█▋        | 1.70M/9.91M [00:01<00:03, 2.54MB/s]

 28%|██▊       | 2.82M/9.91M [00:01<00:01, 4.32MB/s]

 44%|████▎     | 4.33M/9.91M [00:01<00:00, 5.91MB/s]

 55%|█████▍    | 5.44M/9.91M [00:01<00:00, 6.00MB/s]

 61%|██████    | 6.06M/9.91M [00:01<00:00, 5.13MB/s]

 66%|██████▋   | 6.59M/9.91M [00:01<00:00, 4.49MB/s]

 75%|███████▌  | 7.47M/9.91M [00:02<00:00, 4.57MB/s]

 94%|█████████▍| 9.31M/9.91M [00:02<00:00, 6.06MB/s]

100%|██████████| 9.91M/9.91M [00:02<00:00, 4.46MB/s]

  0%|          | 0.00/28.9k [00:00<?, ?B/s]

100%|██████████| 28.9k/28.9k [00:00<00:00, 382kB/s]

  0%|          | 0.00/1.65M [00:00<?, ?B/s]

  6%|▌         | 98.3k/1.65M [00:00<00:02, 723kB/s]

 26%|██▌       | 426k/1.65M [00:00<00:00, 1.69MB/s]

 83%|████████▎ | 1.38M/1.65M [00:00<00:00, 3.91MB/s]

100%|██████████| 1.65M/1.65M [00:00<00:00, 3.82MB/s]

  0%|          | 0.00/4.54k [00:00<?, ?B/s]

100%|██████████| 4.54k/4.54k [00:00<00:00, 1.79MB/s]

Life-cycle of a Symbolic ML Experiment¶

Most advancements in machine learning are based on iterating existing experiments. The life-cycle for experimenting a new idea can be described in 4 phases:

Reproduction¶

Reproducing an existing experiment is simply clone the existing experiment object, or load if from a saved JSON file.

In [4]:

experiment = exp1.sym_clone(deep=True)
assert pg.eq(experiment, exp1)

experiment = pg.load('exp1.json')
# Equal even though `exp1` has already been trained: the learned weights are
# internal state, not part of the binding, so they are neither serialized nor
# compared. What round-trips is the experiment's *definition*.
assert pg.eq(experiment, exp1)

experiment.run()

experiment = exp1.sym_clone(deep=True) assert pg.eq(experiment, exp1) experiment = pg.load('exp1.json') # Equal even though `exp1` has already been trained: the learned weights are # internal state, not part of the binding, so they are neither serialized nor # compared. What round-trips is the experiment's *definition*. assert pg.eq(experiment, exp1) experiment.run()

Out[4]:

0.8828125

Modification¶

To iterate an idea, an existing ML experiment needs to be modified into new ones. With PyGX, a symbolic object can be manipluated into another object without modifying existing code. This gives the maximum flexibility to accomondate unanticpated changes when new ideas pop up.

In [5]:

@pg.symbolize
class NewModel(nn.Module):

  def __init__(self, activation):
    super().__init__()
    self._flatten = Flatten()
    self._dense1 = Linear(784, 1024)
    # `activation` is already held by `NewModel` as its init arg, so give
    # this holder its own copy rather than a second hold.
    self._activation = pg.maybe_clone(activation)
    self._dropout = Dropout(0.2)
    self._dense2 = Linear(1024, 10)

  def forward(self, inputs):
    x = self._flatten(inputs)
    x = self._dense1(x)
    x = self._activation(x)
    x = self._dropout(x)
    return self._dense2(x)


class RMSProp(Optimizer):

  def make(self, params) -> optim.Optimizer:
    return optim.RMSprop(params, lr=self.learning_rate)


experiment.sym_rebind({
  'model': NewModel(activation=ReLU()),
  'training.batch_size': 256,
  'training.optimizer': RMSProp(learning_rate=0.01)
})
print(experiment)
experiment.run()

@pg.symbolize class NewModel(nn.Module): def __init__(self, activation): super().__init__() self._flatten = Flatten() self._dense1 = Linear(784, 1024) # `activation` is already held by `NewModel` as its init arg, so give # this holder its own copy rather than a second hold. self._activation = pg.maybe_clone(activation) self._dropout = Dropout(0.2) self._dense2 = Linear(1024, 10) def forward(self, inputs): x = self._flatten(inputs) x = self._dense1(x) x = self._activation(x) x = self._dropout(x) return self._dense2(x) class RMSProp(Optimizer): def make(self, params) -> optim.Optimizer: return optim.RMSprop(params, lr=self.learning_rate) experiment.sym_rebind({ 'model': NewModel(activation=ReLU()), 'training.batch_size': 256, 'training.optimizer': RMSProp(learning_rate=0.01) }) print(experiment) experiment.run()

Experiment(
  model = NewModel(
    activation = ReLU(
      inplace = False
    )
  ),
  training = {
    input = mnist(
      training = True
    ),
    batch_size = 256,
    num_epochs = 2,
    loss = CrossEntropyLoss(
      weight = None,
      size_average = None,
      ignore_index = -100,
      reduce = None,
      reduction = 'mean',
      label_smoothing = 0.0
    ),
    optimizer = RMSProp(
      learning_rate = 0.01
    )
  },
  evaluation = {
    input = mnist(
      training = False
    ),
    batch_size = 32
  }
)

Out[5]:

0.826171875

Tuning¶

Once an idea works, ML practitioners can squeeze out the performance by tuning its hyperparameters. With PyGX, ML practitioners can tune any part of the experiment with ease. Here we do a hyperparameter sweep on both learning rate for the model.

In [6]:

ssd = experiment.sym_clone(deep=True).sym_rebind({
    'training.optimizer.learning_rate': 
pg.oneof([1e-2, 1e-1]),
})

for exp, feedback in pg.sample(ssd, pg.geno.Sweeping()):
  print(f'Trial {feedback.id}: {feedback.dna}')
  accuracy = exp.run()
  feedback(accuracy)

ssd = experiment.sym_clone(deep=True).sym_rebind({ 'training.optimizer.learning_rate': pg.oneof([1e-2, 1e-1]), }) for exp, feedback in pg.sample(ssd, pg.geno.Sweeping()): print(f'Trial {feedback.id}: {feedback.dna}') accuracy = exp.run() feedback(accuracy)

Trial 1: DNA(0)

Trial 2: DNA(1)

Release¶

Release is simply to save the experiment for future reproduction, with making all new components available to others.

In [7]:

experiment.save('exp2.json')

experiment.save('exp2.json')

The Power of Symbolic Machine Learning¶

The power of symbolic ML is revealed during the whole process of experimentation, from boosting the productivity of developement and iterations, to implementing ideas with new ways of programming, and applying AutoML.

Experimenting with More Productivity¶

Rich formatting in human-readable form¶

In [8]:

print(experiment)

print(experiment)

Experiment(
  model = NewModel(
    activation = ReLU(
      inplace = False
    )
  ),
  training = {
    input = mnist(
      training = True
    ),
    batch_size = 256,
    num_epochs = 2,
    loss = CrossEntropyLoss(
      weight = None,
      size_average = None,
      ignore_index = -100,
      reduce = None,
      reduction = 'mean',
      label_smoothing = 0.0
    ),
    optimizer = RMSProp(
      learning_rate = 0.01
    )
  },
  evaluation = {
    input = mnist(
      training = False
    ),
    batch_size = 32
  }
)

In [9]:

# A single-line print.
print(repr(experiment))

# A single-line print. print(repr(experiment))

Experiment(model=NewModel(activation=ReLU(inplace=False)), training={input=mnist(training=True), batch_size=256, num_epochs=2, loss=CrossEntropyLoss(weight=None, size_average=None, ignore_index=-100, reduce=None, reduction='mean', label_smoothing=0.0), optimizer=RMSProp(learning_rate=0.01)}, evaluation={input=mnist(training=False), batch_size=32})

In [10]:

# Print with docstr, and hide the default values.
print(pg.format(experiment, verbose=True, exclude_defaults=True))

# Print with docstr, and hide the default values. print(pg.format(experiment, verbose=True, exclude_defaults=True))

Experiment(
  # Model used for both training and evaluation.
  model = NewModel(
    activation = ReLU()
  ),
  training = {
    # A callable that returns a tuple of (features, labels) for training.
    input = mnist(
      training = True
    ),
    # Training batch size.
    batch_size = 256,
    # Number of epochs for training.
    num_epochs = 2,
    # Loss used for optimization.
    loss = CrossEntropyLoss(),
    # Optimizer spec for minimizing the loss.
    optimizer = RMSProp(
      learning_rate = 0.01
    )
  },
  evaluation = {
    # A callable that returns a tuple of (features, labels) for evaluation.
    input = mnist(
      training = False
    ),
    # Evaluation batch size.
    batch_size = 32
  }
)

Retrieving the schema of hyper-parameters¶

In [11]:

print(Experiment.__schema__)

print(Experiment.__schema__)

Schema(
  name='__main__.Experiment',
  fields=[
    Field(
      key=model,
      value=Object(
        Module
      ),
      description='Model used for both training and evaluation.',
      origin=<class '__main__.Experiment'>
    ),
    Field(
      key=training,
      value=Dict(
        fields=[
          Field(
            key=input,
            value=Callable(),
            description='A callable that returns a tuple of (features, labels) for training.'
          ),
          Field(
            key=batch_size,
            value=Int(
              min=1
            ),
            description='Training batch size.'
          ),
          Field(
            key=num_epochs,
            value=Int(
              min=1
            ),
            description='Number of epochs for training.'
          ),
          Field(
            key=loss,
            value=Object(
              Module
            ),
            description='Loss used for optimization.'
          ),
          Field(
            key=optimizer,
            value=Object(
              Optimizer
            ),
            description='Optimizer spec for minimizing the loss.'
          )
        ]
      ),
      origin=<class '__main__.Experiment'>
    ),
    Field(
      key=evaluation,
      value=Dict(
        fields=[
          Field(
            key=input,
            value=Callable(),
            description='A callable that returns a tuple of (features, labels) for evaluation.'
          ),
          Field(
            key=batch_size,
            value=Int(
              min=1
            ),
            description='Evaluation batch size.'
          )
        ]
      ),
      origin=<class '__main__.Experiment'>
    )
  ],
  metadata={
    'init_arg_list': ['model', 'training', 'evaluation']
  }
)

Catching bad experiment specifications¶

In [12]:

try:
  experiment.sym_rebind({
      # Misput batch size.
      'training.batch_size': 0
  })
except ValueError as e:
  print(e)

try: experiment.sym_rebind({ # Misput batch size. 'training.batch_size': 0 }) except ValueError as e: print(e)

Value 0 is out of range (min=1, max=None). (path=training.batch_size)

Showing the differences between two experiments¶

In [13]:

print(pg.diff(experiment, experiment.sym_clone(override={
    'training.batch_size': 128,
    'training.optimizer.learning_rate': 0.2
})))

print(pg.diff(experiment, experiment.sym_clone(override={ 'training.batch_size': 128, 'training.optimizer.learning_rate': 0.2 })))

Experiment(
  training = Dict(
    batch_size = Diff(
      left = 256,
      right = 128
    ),
    optimizer = RMSProp(
      learning_rate = Diff(
        left = 0.01,
        right = 0.2
      )
    )
  )
)

Querying parts of an experiment¶

The ML experiment is now a symbolic tree, which can be queried by nodes' locations or values.

In [14]:

# Query by key path.
pg.query(experiment, '.*input')

# Query by key path. pg.query(experiment, '.*input')

Out[14]:

{'training.input': mnist(training=True),
 'evaluation.input': mnist(training=False)}

In [15]:

# Query by values.
pg.query(experiment, where=lambda v: isinstance(v, bool))

# Query by values. pg.query(experiment, where=lambda v: isinstance(v, bool))

Out[15]:

{'model.activation.inplace': False,
 'training.input.training': True,
 'evaluation.input.training': False}

Meta-programming by traversing the experiment¶

In [16]:

def list_keys(key, value, parent):
  print(key)

pg.traverse(experiment, list_keys)

def list_keys(key, value, parent): print(key) pg.traverse(experiment, list_keys)

model
model.activation
model.activation.inplace
training
training.input
training.input.training
training.batch_size
training.num_epochs
training.loss
training.loss.weight
training.loss.size_average
training.loss.ignore_index
training.loss.reduce
training.loss.reduction
training.loss.label_smoothing
training.optimizer
training.optimizer.learning_rate
evaluation
evaluation.input
evaluation.input.training
evaluation.batch_size

Out[16]:

True

Comparing concepts¶

When the user want to check if two components have the same symbolic representation, symbolic comparison can be applied.

In [17]:

print(pg.eq(experiment, exp1))
print(pg.eq(experiment, experiment.sym_clone(deep=True)))

print(pg.eq(experiment, exp1)) print(pg.eq(experiment, experiment.sym_clone(deep=True)))

False
True

Saving/Loading concepts¶

Objects can be serialized solely based on their symbolic representation, regardless their internal states.

In [18]:

pg.eq(experiment, pg.from_json_str(experiment.to_json_str()))

pg.eq(experiment, pg.from_json_str(experiment.to_json_str()))

Out[18]:

True

Preventing an object from further modification¶

In [19]:

experiment.sym_seal()
try:
  experiment.sym_rebind({
      'training.batch_size': 5
  })
except pg.WritePermissionError as e:
  print(e)
experiment.sym_seal(False)

experiment.sym_seal() try: experiment.sym_rebind({ 'training.batch_size': 5 }) except pg.WritePermissionError as e: print(e) experiment.sym_seal(False)

Cannot rebind a sealed Experiment.

Out[19]:

Experiment(...)

Experiment( model=NewModel( activation=ReLU( inplace=False ) ), training={ 'input': mnist( training=True ), 'batch_size': 256, 'num_epochs': 2, 'loss': CrossEntropyLoss( weight=None, size_average=None, ignore_index=-100, reduce=None, reduction='mean', label_smoothing=0.0 ), 'optimizer': RMSProp( learning_rate=0.01 ) }, evaluation={ 'input': mnist( training=False ), 'batch_size': 32 } )

modelmodel

NewModel(...)

NewModel( activation=ReLU( inplace=False ) )

activationmodel.activation

ReLU(...)

ReLU( inplace=False )

inplacemodel.activation.inplace

bool

False

False

trainingtraining

Dict(...)

{ 'input': mnist( training=True ), 'batch_size': 256, 'num_epochs': 2, 'loss': CrossEntropyLoss( weight=None, size_average=None, ignore_index=-100, reduce=None, reduction='mean', label_smoothing=0.0 ), 'optimizer': RMSProp( learning_rate=0.01 ) }

inputtraining.input

mnist(...)

mnist( training=True )

trainingtraining.input.training

bool

True

True

batch_sizetraining.batch_size

int

256

256

num_epochstraining.num_epochs

int

2

2

losstraining.loss

CrossEntropyLoss(...)

CrossEntropyLoss( weight=None, size_average=None, ignore_index=-100, reduce=None, reduction='mean', label_smoothing=0.0 )

weighttraining.loss.weight

NoneType(...)

None

None

size_averagetraining.loss.size_average

NoneType(...)

None

None

ignore_indextraining.loss.ignore_index

int

-100

-100

reducetraining.loss.reduce

NoneType(...)

None

None

reductiontraining.loss.reduction

str

'mean'

'mean'

label_smoothingtraining.loss.label_smoothing

float

0.0

0.0

optimizertraining.optimizer

RMSProp(...)

RMSProp( learning_rate=0.01 )

learning_ratetraining.optimizer.learning_rate

float

0.01

0.01

evaluationevaluation

Dict(...)

{ 'input': mnist( training=False ), 'batch_size': 32 }

inputevaluation.input

mnist(...)

mnist( training=False )

trainingevaluation.input.training

bool

False

False

batch_sizeevaluation.batch_size

int

32

32

Riding the Power of Symbolic Manipulation¶

Symbolic manipulation allows modification of the parts of an experiment after its creation. Moreover, it provides programming interfaces to meta-program the experiment by rules or algorithms.

Encapsulating ideas and making them reusable¶

For common software systems, code reuse happens at type definition level; for traditional ML systems, code reuse happens at instance level (e.g. reusing an experiment). For symbolic ML systems, code reuse can takes place at an even higher level - the procedure (meta-program) that transforms object A to object B. Essentially, such meta-programs encapsulates ideas that make ML effective, such as linear scaling of learning rate when batch size increaes, replacing ReLU layers with ELu layers, and etc.

In [20]:

@pg.patcher([
  ('batch_size', pg.typing.Int(min_value=1))
])
def scale_lr_with_batch_size(exp, batch_size):
  # A patcher returns a dict of location to updated values
  # within the patching target.
  return {
      'training.batch_size': batch_size,
      'training.optimizer.learning_rate': (
          exp.training.optimizer.learning_rate * (
              batch_size / exp.training.batch_size))
  }

new_experiment = experiment.sym_clone(deep=True)
pg.patch(new_experiment, scale_lr_with_batch_size(batch_size=1024))
print(pg.diff(experiment, new_experiment))

@pg.patcher([ ('batch_size', pg.typing.Int(min_value=1)) ]) def scale_lr_with_batch_size(exp, batch_size): # A patcher returns a dict of location to updated values # within the patching target. return { 'training.batch_size': batch_size, 'training.optimizer.learning_rate': ( exp.training.optimizer.learning_rate * ( batch_size / exp.training.batch_size)) } new_experiment = experiment.sym_clone(deep=True) pg.patch(new_experiment, scale_lr_with_batch_size(batch_size=1024)) print(pg.diff(experiment, new_experiment))

Experiment(
  training = Dict(
    batch_size = Diff(
      left = 256,
      right = 1024
    ),
    optimizer = RMSProp(
      learning_rate = Diff(
        left = 0.01,
        right = 0.04
      )
    )
  )
)

Patcher can also by defined as transforms that is based on patterns in the symbolic tree. For example, swapping all ReLU activations to ELU can be expressed as

In [21]:

ELU = pg.symbolize(nn.ELU)

@pg.patcher()
def relu_to_elu(unused_exp):
  def _swap_activation(k, v, p):
    """Transform fn for each node in the symbolic tree in bottom-up a manner.

    Args:
      k: a `pg.KeyPath` object representing the location of current node.
      v: the value of current node.
      p: the parent node.

    Returns:
      Transformed value.
    """
    if isinstance(v, ReLU):
      return ELU()
    return v
  return _swap_activation

new_experiment = experiment.sym_clone(deep=True)
pg.patch(new_experiment, relu_to_elu())
print(pg.diff(experiment, new_experiment))

ELU = pg.symbolize(nn.ELU) @pg.patcher() def relu_to_elu(unused_exp): def _swap_activation(k, v, p): """Transform fn for each node in the symbolic tree in bottom-up a manner. Args: k: a `pg.KeyPath` object representing the location of current node. v: the value of current node. p: the parent node. Returns: Transformed value. """ if isinstance(v, ReLU): return ELU() return v return _swap_activation new_experiment = experiment.sym_clone(deep=True) pg.patch(new_experiment, relu_to_elu()) print(pg.diff(experiment, new_experiment))

Experiment(
  model = NewModel(
    activation = Diff(
      left = ReLU(
        inplace = False
      ),
      right = ELU(
        alpha = 1.0,
        inplace = False
      )
    )
  )
)

Combining ideas¶

Ideas can be easily combined by chaining the patchers together. For example, we can apply both techiniques introduced above to a new experiment by:

In [22]:

new_experiment = experiment.sym_clone(deep=True)
# Combining ideas by chaining the patchers.
pg.patch(new_experiment, [
    scale_lr_with_batch_size(batch_size=1024),
    relu_to_elu(),
])
print(pg.diff(experiment, new_experiment))

new_experiment = experiment.sym_clone(deep=True) # Combining ideas by chaining the patchers. pg.patch(new_experiment, [ scale_lr_with_batch_size(batch_size=1024), relu_to_elu(), ]) print(pg.diff(experiment, new_experiment))

Experiment(
  model = NewModel(
    activation = Diff(
      left = ReLU(
        inplace = False
      ),
      right = ELU(
        alpha = 1.0,
        inplace = False
      )
    )
  ),
  training = Dict(
    batch_size = Diff(
      left = 256,
      right = 1024
    ),
    optimizer = RMSProp(
      learning_rate = Diff(
        left = 0.01,
        right = 0.04
      )
    )
  )
)

Patching experiment from the command line¶

Though it's easy to patch experiments via Python code. It would be better if ML practitioners can patch experiments from the command line, which requires no rebuild/repackage of the source code.

To address this requirement, patcher can also be invoked as a URL-like string. Therefore, the strings can be passed in as command-line arguments.

In [23]:

print(pg.diff(experiment, pg.patch(experiment.sym_clone(deep=True), [
    'scale_lr_with_batch_size?batch_size=1024',
    'relu_to_elu'
])))

print(pg.diff(experiment, pg.patch(experiment.sym_clone(deep=True), [ 'scale_lr_with_batch_size?batch_size=1024', 'relu_to_elu' ])))

Experiment(
  model = NewModel(
    activation = Diff(
      left = ReLU(
        inplace = False
      ),
      right = ELU(
        alpha = 1.0,
        inplace = False
      )
    )
  ),
  training = Dict(
    batch_size = Diff(
      left = 256,
      right = 1024
    ),
    optimizer = RMSProp(
      learning_rate = Diff(
        left = 0.01,
        right = 0.04
      )
    )
  )
)

Summary: A better expression of ML experiment¶

Putting things together, a new experiment can be expressed as an existing experiment applying new ML ideas, with each patcher represents an ML idea. Such expression is more high-level than hyperparameter-level differences, and better reasons their differencs. Moreover, the ML ideas can be applied to other experiments too.

See also:

• Symbolic Neural Modeling

Embracing AutoML as a Part of the ML Experimentation Process¶

Machine learning is a process of trials and errors. Today, most of such trials are manually done with human-conducted evolution, a repeated and laboring process. On the other hand, automatic hyperparameter tuning is not a stranger, it's often supported for large scale ML systems. More complex automatic explorations, like neural architecture search, however, are still considered advanced technologies that most ML practioners do not have access to.

AutoML should be an integral part of ML experimentation process, and it should be straightforward for every ML practioner. PyGX enables this with a few lines of code changes. This means, an arbitrary part of an ML experiment can be added to the search space easily, and to be optimized by state-of-the-art search algorithms.

Defining what to explore¶

In [24]:

# Let's take another look of the experiment.
print(experiment)

# Let's take another look of the experiment. print(experiment)

Experiment(
  model = NewModel(
    activation = ReLU(
      inplace = False
    )
  ),
  training = {
    input = mnist(
      training = True
    ),
    batch_size = 256,
    num_epochs = 2,
    loss = CrossEntropyLoss(
      weight = None,
      size_average = None,
      ignore_index = -100,
      reduce = None,
      reduction = 'mean',
      label_smoothing = 0.0
    ),
    optimizer = RMSProp(
      learning_rate = 0.01
    )
  },
  evaluation = {
    input = mnist(
      training = False
    ),
    batch_size = 32
  }
)

Now assume that we have three ideas:

• Try different activations.
• Try different optimizers.
• Try different batch size.

We can make a search space with jointly optimizing these three apsects:

In [25]:

class SGD(Optimizer):

  def make(self, params) -> optim.Optimizer:
    return optim.SGD(params, lr=self.learning_rate)


PReLU = pg.symbolize(nn.PReLU)

learning_rate = 0.001

experiment_space = experiment.sym_clone(override={
    'model.activation': pg.oneof([ReLU(), ELU(), PReLU()]),
    'training.optimizer': pg.oneof([
        RMSProp(learning_rate=learning_rate),
        SGD(learning_rate=learning_rate),
        Adam(learning_rate=learning_rate),
    ]),
    'training.batch_size': pg.oneof([32, 64, 128])
})

# Let's inspect the decision points from this space:
print(pg.dna_spec(experiment_space))

class SGD(Optimizer): def make(self, params) -> optim.Optimizer: return optim.SGD(params, lr=self.learning_rate) PReLU = pg.symbolize(nn.PReLU) learning_rate = 0.001 experiment_space = experiment.sym_clone(override={ 'model.activation': pg.oneof([ReLU(), ELU(), PReLU()]), 'training.optimizer': pg.oneof([ RMSProp(learning_rate=learning_rate), SGD(learning_rate=learning_rate), Adam(learning_rate=learning_rate), ]), 'training.batch_size': pg.oneof([32, 64, 128]) }) # Let's inspect the decision points from this space: print(pg.dna_spec(experiment_space))

Space({
  0 = 'model.activation': Choices(num_choices=1, [
    (0): ReLU()
    (1): ELU()
    (2): PReLU()
  ])
  1 = 'training.batch_size': Choices(num_choices=1, [
    (0): 32
    (1): 64
    (2): 128
  ])
  2 = 'training.optimizer': Choices(num_choices=1, [
    (0): RMSProp()
    (1): SGD()
    (2): Adam()
  ])
})

In [26]:

# Check the size of the space.
print(pg.dna_spec(experiment_space).space_size)

# Check the size of the space. print(pg.dna_spec(experiment_space).space_size)

27

Automatic exploration with regularized evolution¶

With an experiment space, we can specify how to optimize based on:

• Search algorithm: we use regularized evolution here.
• Reward function: we use accuracy as the reward.

In [27]:

best_exp, best_accuracy = None, None
history = []
for exp, feedback in pg.sample(experiment_space,
                               pg.algo.evolution.regularized_evolution(
                                   population_size=5, tournament_size=3),
                               num_examples=20):
  print(f'Trial {feedback.id}: {feedback.dna}')
  accuracy = exp.run()
  if best_accuracy is None or best_accuracy < accuracy:
    best_accuracy, best_exp = accuracy, exp
  history.append(accuracy)
  feedback(accuracy)

print(f'Best acurracy: {best_accuracy}')
print(f'Best experiment: {best_exp}')

best_exp, best_accuracy = None, None history = [] for exp, feedback in pg.sample(experiment_space, pg.algo.evolution.regularized_evolution( population_size=5, tournament_size=3), num_examples=20): print(f'Trial {feedback.id}: {feedback.dna}') accuracy = exp.run() if best_accuracy is None or best_accuracy < accuracy: best_accuracy, best_exp = accuracy, exp history.append(accuracy) feedback(accuracy) print(f'Best acurracy: {best_accuracy}') print(f'Best experiment: {best_exp}')

Trial 1: DNA([2, 2, 2])

Trial 2: DNA([2, 0, 0])

Trial 3: DNA([2, 2, 1])

Trial 4: DNA([1, 2, 1])

Trial 5: DNA([1, 2, 0])

Trial 6: DNA([0, 0, 0])

Trial 7: DNA([0, 2, 0])

Trial 8: DNA([0, 0, 0])

Trial 9: DNA([0, 0, 2])

Trial 10: DNA([1, 0, 0])

Trial 11: DNA([0, 0, 0])

Trial 12: DNA([0, 1, 0])

Trial 13: DNA([1, 0, 0])

Trial 14: DNA([1, 0, 2])

Trial 15: DNA([0, 2, 0])
Trial 16: DNA([0, 1, 0])

Trial 17: DNA([0, 1, 0])
Trial 18: DNA([0, 1, 1])

Trial 19: DNA([2, 1, 0])
Trial 20: DNA([1, 1, 0])

Best acurracy: 0.90234375
Best experiment: Experiment(
  model = NewModel(
    activation = ReLU(
      inplace = False
    )
  ),
  training = {
    input = mnist(
      training = True
    ),
    batch_size = 64,
    num_epochs = 2,
    loss = CrossEntropyLoss(
      weight = None,
      size_average = None,
      ignore_index = -100,
      reduce = None,
      reduction = 'mean',
      label_smoothing = 0.0
    ),
    optimizer = RMSProp(
      learning_rate = 0.001
    )
  },
  evaluation = {
    input = mnist(
      training = False
    ),
    batch_size = 32
  }
)

In [28]:

import matplotlib.pyplot as plt
plt.plot(list(range(len(history))), [a - 0.9 for a in history])
plt.show()

# It turns out that SGD with the given learning rate doesn't work well.

import matplotlib.pyplot as plt plt.plot(list(range(len(history))), [a - 0.9 for a in history]) plt.show() # It turns out that SGD with the given learning rate doesn't work well.

See also:

• NAS with MNIST
• NAS-Bench-101 with PyGX
• Evolving Reinforcement Learning Algorithms