Initial commit

{\rtf1\ansi\ansicpg1252\cocoartf1038\cocoasubrtf350

{\fonttbl\f0\fswiss\fcharset0 Helvetica;}

{\colortbl;\red255\green255\blue255;}

\margl1440\margr1440\vieww9000\viewh8400\viewkind0

\pard\tx720\tx1440\tx2160\tx2880\tx3600\tx4320\tx5040\tx5760\tx6480\tx7200\tx7920\tx8640\ql\qnatural\pardirnatural

\f0\fs24 \cf0 No work files for this lesson...}

 No newline at end of file

%% Cell type:markdown id: tags:

# `Variables` in TensorFlow-Slim

*by Marvin Bertin*

<img src="../images/tensorflow.png" width="400">

%% Cell type:markdown id: tags:

### TensorFlow Computational Graph

<img src="../images/tensors_flowing.gif" width="300" align=right>

* **Directed graph**, where nodes are connected by directed edges.

* **Nodes** represent **mathematical operations** (computations).

* **Edges** represent the **multidimensional data arrays** (tensors) *flowing* between the nodes.

### Variables

In TensorFlow, a `Variable`:

- is a mutable node in the computational graph

- is a Tensor of data

- has a shape

- has an initial value

- has a name

### Variable Useage

`Variables` are most often used to hold the trainable paramters in a Deep Learning model. They can be saved to disk and reloaded later for model analysis or inference. However, there exist other types of `Variables`.

### TensorFlow Variable Types

- **Regular Variable** - A variable that do not necessarily represent the model. They are used during learning or evaluation but are not required for actually performing inference.

- **Model Variable** - A type of trainable variable that represent the Deep Learning model parameters. These variables are updated during training and loaded from checkpoints during evaluation or inference.

- **Local Variable** - A transient variable that only exist for the duration of the session and is not saved to disk.

- **Global Variable** - A variable that persists across every training iterations and is globally accessible.

### TensorFlow-Slim - `Variables`

TF-Slim provides a set of thin wrapper functions, that makes variable creation and manipulation simpler. <br>

All variable related functions are accesssible in [variable.py](https://github.com/tensorflow/tensorflow/blob/r0.12/tensorflow/contrib/framework/python/ops/variables.py). <br>

<br>

The following methods will be covered in this notebook:

```

slim.variable

slim.model_variable

slim.local_variable

slim.get_or_create_global_step

slim.get_variables

slim.get_model_variables

slim.get_local_variables

slim.get_global_step

```

%% Cell type:markdown id: tags:

## Import TensorFlow

%% Cell type:code id: tags:

``` python

from __future__ import absolute_import

from __future__ import division

from __future__ import print_function

import tensorflow as tf

slim = tf.contrib.slim

```

%% Cell type:markdown id: tags:

## Import Helper Functions

%% Cell type:code id: tags:

``` python

import sys

sys.path.append("../")

from utils.pretty_printer import inspect_variables, inspect_layers

```

%% Cell type:markdown id: tags:

## Defining Variables with Native TensorFlow

Example of defining different variable types with regular Tensorflow

%% Cell type:code id: tags:

``` python

# Initialize graph

g = tf.Graph()

with g.as_default():

    # Global variable

    global_step = tf.Variable(tf.constant(value=1, shape=[], dtype=tf.int32),

                              trainable=False,

                              name="global_step",

                              collections=["global_variables"])

    # Start variable scope

    with tf.variable_scope("TF-Native"):

        # Pin variables to CPU.

        with tf.device("/cpu:0"):

            # Model variable

            weights = tf.Variable(trainable=True,

                                  name="weights",

                                  initial_value=tf.truncated_normal([3, 3, 64, 128],

                                                                    stddev=0.1,

                                                                    dtype=tf.float32),

                                  collections=["model_variables"])

            # Model variable

            biases = tf.Variable(tf.constant(0.0, shape=[128], dtype=tf.float32),

                                 trainable=True,

                                 name="biases",

                                 collections=["model_variables"])

            # regularization loss on model parameters

            regularizers = (tf.nn.l2_loss(weights) + tf.nn.l2_loss(biases))

            # Trainable regular variable

            predictions = tf.Variable(tf.constant(0.0, shape=[32, 10], dtype=tf.float32),

                                      trainable=True,

                                      name="predictions",

                                      collections=["trainable_variables"])

            # Local variable

            accuracy = tf.Variable(tf.constant(0.0, shape=[], dtype=tf.float32),

                                   trainable=False,

                                   name="accuracy",

                                   collections=["local_variables"])

```

%% Cell type:markdown id: tags:

## Inspect Name Space

%% Cell type:code id: tags:

``` python

with g.as_default():

    print("Variables:")

    inspect_variables([global_step, weights, biases, predictions, accuracy])

```

%% Output

    Variables:

    name = global_step:0                                           shape = ()

    name = TF-Native/weights:0                                     shape = (3, 3, 64, 128)

    name = TF-Native/biases:0                                      shape = (128,)

    name = TF-Native/predictions:0                                 shape = (32, 10)

    name = TF-Native/accuracy:0                                    shape = ()

%% Cell type:markdown id: tags:

## Defining Variables with TensorFlow-Slim

TF-Slim provides mutiple wrappers functions that makes initializing and manipulating variables easier.

%% Cell type:markdown id: tags:

    => slim.get_or_create_global_step(graph=None)

    Docstring:

    Returns and create (if necessary) the global step variable.

    Args:

      graph: The graph in which to create the global step. If missing, use default

          graph.

    Returns:

      the tensor representing the global step variable.

---

    => slim.model_variable(*args, **kwargs)

    Docstring:

    Gets an existing model variable with these parameters or creates a new one.

    Args:

      name: the name of the new or existing variable.

      shape: shape of the new or existing variable.

      dtype: type of the new or existing variable (defaults to `DT_FLOAT`).

      initializer: initializer for the variable if one is created.

      regularizer: a (Tensor -> Tensor or None) function; the result of

          applying it on a newly created variable will be added to the collection

          GraphKeys.REGULARIZATION_LOSSES and can be used for regularization.

      trainable: If `True` also add the variable to the graph collection

        `GraphKeys.TRAINABLE_VARIABLES` (see tf.Variable).

      device: Optional device to place the variable. It can be an string or a

        function that is called to get the device for the variable.

      ...

    Returns:

      The created or existing variable.

---

    => slim.variable(*args, **kwargs)

    Docstring:

    Gets an existing variable with these parameters or creates a new one.

    Args:

      name: the name of the new or existing variable.

      shape: shape of the new or existing variable.

      dtype: type of the new or existing variable (defaults to `DT_FLOAT`).

      initializer: initializer for the variable if one is created.

      regularizer: a (Tensor -> Tensor or None) function; the result of

          applying it on a newly created variable will be added to the collection

          GraphKeys.REGULARIZATION_LOSSES and can be used for regularization.

      trainable: If `True` also add the variable to the graph collection

        `GraphKeys.TRAINABLE_VARIABLES` (see tf.Variable).

      device: Optional device to place the variable. It can be an string or a

        function that is called to get the device for the variable.

      ...

    Returns:

      The created or existing variable.

---

    => slim.local_variable(initial_value, validate_shape=True, name=None)

    Docstring:

    Create variable and add it to `GraphKeys.LOCAL_VARIABLES` collection.

    Args:

      initial_value: A `Tensor`, or Python object convertible to a `Tensor`,

          which is the initial value for the Variable.

      name: Optional name for the variable. Defaults to `'Variable'` and gets

          uniquified automatically.

      ...

    Returns:

      New variable.

%% Cell type:code id: tags:

``` python

# Initialize graph

g = tf.Graph()

with g.as_default():

    # Global variable

    global_step = slim.get_or_create_global_step()

    with tf.variable_scope("TF-Slim-weights"):

        # Model variable

        weights = slim.model_variable('weights',

                                      shape=[3, 3, 64, 128],

                                      initializer=tf.truncated_normal_initializer(stddev=0.01),

                                      regularizer=slim.l2_regularizer(0.05),

                                      device='/cpu:0')

        # Model variable

        biases = slim.model_variable('biases',

                                      shape=[128],

                                      initializer=tf.zeros_initializer(),

                                      regularizer=slim.l2_regularizer(0.05),

                                      device='/cpu:0')

    # Trainable variable

    probabilities = slim.variable('probabilities',

                                  shape=[32, 10],

                                  initializer=tf.zeros_initializer())

    # Local variable

    accuracy = slim.local_variable(initial_value=0.0,name="accuracy")

```

%% Cell type:markdown id: tags:

## Inspect Name Space

TF-Slim makes is possible to collect all graph variables filtered by *scope* or *variable type*, with the function:

    => slim.get_variables(scope=None, suffix=None, collection='variables')

    Docstring:

    Gets the list of variables, filtered by scope and/or suffix.

    Args:

      scope: an optional scope for filtering the variables to return.

      suffix: an optional suffix for filtering the variables to return.

      collection: in which collection search for. Defaults to GraphKeys.VARIABLES.

    Returns:

      a list of variables in collection with scope and suffix.

%% Cell type:markdown id: tags:

## Scope Variables

%% Cell type:code id: tags:

``` python

with g.as_default():

    print("TF-Slim-weights:")

    inspect_variables(slim.get_variables(scope="TF-Slim-weights"))

```

%% Output

    TF-Slim-weights:

    name = TF-Slim-weights/weights:0                               shape = (3, 3, 64, 128)

    name = TF-Slim-weights/biases:0                                shape = (128,)

%% Cell type:markdown id: tags:

## Model Variables

%% Cell type:code id: tags:

``` python

with g.as_default():

    print("model_variables:")

    inspect_variables(slim.get_variables(collection="model_variables"))

```

%% Output

    model_variables:

    name = TF-Slim-weights/weights:0                               shape = (3, 3, 64, 128)

    name = TF-Slim-weights/biases:0                                shape = (128,)

%% Cell type:markdown id: tags:

## Trainable Variables

%% Cell type:code id: tags:

``` python

with g.as_default():

    print("trainable_variables:")

    inspect_variables(slim.get_variables(collection="trainable_variables"))

```

%% Output

    trainable_variables:

    name = TF-Slim-weights/weights:0                               shape = (3, 3, 64, 128)

    name = TF-Slim-weights/biases:0                                shape = (128,)

    name = probabilities:0                                         shape = (32, 10)

%% Cell type:markdown id: tags:

## Global Variables

%% Cell type:code id: tags:

``` python

with g.as_default():

    print("global_variables:")

    inspect_variables([slim.get_global_step()])

```

%% Output

    global_variables:

    name = global_step:0                                           shape = ()

%% Cell type:markdown id: tags:

## Local Variables

%% Cell type:code id: tags:

``` python

with g.as_default():

    print("local_variables:")

    inspect_variables(slim.get_variables(collection="local_variables"))

```

%% Output

    local_variables:

    name = accuracy:0                                              shape = ()

%% Cell type:markdown id: tags:

## Regularization Losses

%% Cell type:code id: tags:

``` python

with g.as_default():

    print("regularization_losses:")

    inspect_variables(slim.get_variables(collection="regularization_losses"))

```

%% Output

    regularization_losses:

    name = TF-Slim-weights/weights/Regularizer/l2_regularizer:0    shape = ()

    name = TF-Slim-weights/biases/Regularizer/l2_regularizer:0     shape = ()

%% Cell type:markdown id: tags:

## Next Lesson

### `Layers` in TensorFlow-Slim

- Combining primitive TF operations into abstract higher level layers.

<img src="../images/divider.png" width="100">