So variable_scope will be replaced by name_scope, right? Here url of variable_create_scope is linked to a blank line, could you give more details about the function (say, some examples)?

Fixed the link. The documentation has examples of how it's used.

1. Could you explain why we choose tf.Variable(*args, **kwargs), rather than tf.get_variable, to construct variables?

   The tf.Variable class will be an abstract base class which defines a tf.Variable interface.

   If tf.Variable will be an abstract base class, how to call tf.Variable(*args, **kwargs)?

2. Could you explain what is scoped constructor functions?

1. tf.get_variable was created to handle silent sharing of variables in the graph. This behavior is being removed.

2. See the link I updated about variable_creator_scope

1. Will it be possible to recover tf.Variable objects only from a graph or graph_def, just like it's now possible to do with tf.Variable.from_proto? We work a lot with managing models restored purely from graph def files, without necessarily having all the code that produced the original graph. The ability to restore basic TF objects such as tf.Variables directly from graph def data only is a must for us.

2. How is the above affected by tf.Variable types written by users?

3. Will it be possible to explicitly recreate or recover tf.Variable objects from other non-python-object pieces of data like in some way?

1. tf.get_variable was created to handle silent sharing of variables in the graph. This behavior is being removed.
2. See the link I updated about variable_creator_scope

a related question: instead of tf.Variable why not calling factory function directly since it is supposed to call a factory function.

Hi @alextp.
Could you please show an example about how to create a PartitionedVariable via API tf.Variable(*args, **kwargs) ? My question is whether user should pass an indicator to show what kinds of concrete Variable to create ? Does it mean the parameters *args and **kwargs are exposed to users without any limit?

1. Will it be possible to recover tf.Variable objects only from a graph or graph_def, just like it's now possible to do with tf.Variable.from_proto? We work a lot with managing models restored purely from graph def files, without necessarily having all the code that produced the original graph. The ability to restore basic TF objects such as tf.Variables directly from graph def data only is a must for us.
2. How is the above affected by tf.Variable types written by users?
3. Will it be possible to explicitly recreate or recover tf.Variable objects from other non-python-object pieces of data like in some way?

+1 on this. It is crucial for us to restore them from serialized graphdefs. Currently we use a RestoredVariable class inheriting from tf.Variable, but RefVariable changes in TF1.11 are breaking this inheritance. See issues #23591, #22648.

So tf.global_variables_initializer will be deprecated as well, right?

Can we let variable take care of the initialization by itself? I find that it's awkward to force user to call sess.run(tf.global_variables_initializer) before training. When a variable is read, it knows whether its status is initialized or not in fact.

global_variables_initializer will be deprecated, yes. I agree there could be a better solution to initialization but it's not in scope for this change.

Note that if eager is turned on by default and variables are created from eager then they're already automatically initialized even if most code runs inside graph functions, so most people in tf 2 will hopefully not be affected by this.

I have trouble understanding the sentence, do you mean that:

v = tf.Variable(xxxxx)
v_read = v.read_variable_op()

v is mutable, while v_read not?

yes, exactly

If you'll allow a slight digression, what is the role of tf.experimental module? Would it become a next tf.contrib? cc @martinwicke

Initially this interface will have enough abstract methods such that the user-visible API of tf.Variable does not change.

I'm not sure this makes sense: did it mean to read "enough concrete methods"? Adding many abstract methods doesn't change the user-visible tf.Variable API (for those using the existing/TensorFlow 1.x tf.Variable API)

This change has already been implemented. If you look at tf.Variable now it's a class with no implementations of methods, and most concrete instances are instances of subclasses (RefVariable for the old ones and ResourceVariable for the new ones).

It would be good to include a justification for why the client API should be calling the constructor to an abstract base class instead of having users explicitly call the type of variable they want. This document just says "it will do this complicated thing" without saying what the rationale is.

The goal is that the user should not have to know what type they want. For example, code called under distribution strategies might create MirroredVariables when the user calls tf.Variable. Think of tf.Variable as a factory function for which isinstance also works.

Could we add an example of what that canonical approach for sharing variables will be? There are a large number of models that relied on tf.get_variable() (as it was pushed to be the standard way to create/access variables), so demonstrating what the new uses would look like would be beneficial.

The canonical approach to sharing variables is by sharing their objects, as in Keras layers and Keras models, tf.make_template, and other ways of doing that.

Ok, I'm glad you brought up tf.make_template, as it's my preferred way of sharing weights between training/eval/inference in a single graph, but I'm wondering what the plan is to support tf.make_template given that it heavily relies on the existing variable_scope and naming semantics in order to work. There's a comment at the bottom which mentions it potentially being in scope, but I wonder what the mechanisms would like like without collections or special naming semantics.

Link to the API for reference.

https://www.tensorflow.org/api_docs/python/tf/contrib/checkpoint/Checkpointable (added to the document too)

Going to request that this is a requirement instead of an optional.

Alternatively, expand layer-based APIs to make it easier to reuse existing variables imperatively.

Also want to request make_template in v2.
One question: right now in make_template, one can use get_variable to create reused variable and tf.Variable(trainable=False) to create local (unshared) variables. After get_variable is deprecated I wonder what should be the alternative.

@alextp could you comment on how make_template will be supported and how a user should create shared and unshared variables inside make_template ?

So how would this look in the following scenario?
There is a non-trainable floating point Variable in the model that affects the calculation of the loss function (e.g the discount factor in reinforcement learning). This Variable should be saved to the checkpoint, but obviously should not be considered a parameter in the optimizer.

Nit: please escape _ in _UnreadVariable, markdown thinks you are trying to put stuff in italics (I think _UnreadVariable works).

Please make item assignment possible:

>>> import tensorflow as tf
>>> a = tf.Variable([1, 2, 3])
>>> a[1] = 5
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'Variable' object does not support item assignment

• It's already possible via other methods (tf.scatter*) but really cumbersome.
• Since it's already possible, I assume it can't be that hard to implement (but maybe I'm missing something)
• It would make teaching TensorFlow easier ("it's just like NumPy")
• It's one of those little things that makes some people prefer PyTorch: they can say "PyTorch is just like NumPy", but it's harder to say this about TensorFlow when something as fundamental to NumPy is missing.
• I have run into real-life use cases where I really needed it (porting a library from NumPy to TensorFlow to make it run on a GPU).
  Please, pretty please with sugar on top? ;-)
  Edit: Alex pointed out that it will be possible in TF 1.11 with a[1].assign(5).

1. Will it be possible to recover tf.Variable objects only from a graph or graph_def, just like it's now possible to do with tf.Variable.from_proto? We work a lot with managing models restored purely from graph def files, without necessarily having all the code that produced the original graph. The ability to restore basic TF objects such as tf.Variables directly from graph def data only is a must for us.

2. How is the above affected by tf.Variable types written by users?

3. Will it be possible to explicitly recreate or recover tf.Variable objects from other non-python-object pieces of data like in some way?

I see this is a better generalization of custom_getters for creating variables, but it seems to miss another use case that worked for custom_getter: transparently intercepting variable reads.

Here are three example cases where this can be very useful.

1. Applying spectral normalization over arbitrary models. You could simply define a getter that applies it to the scope of a model and returns the normalized variable result, without having to change the code to explicitly do it everywhere. This becomes particularly important in more complex models, or in third party models that you are simply reusing and cannot change their code.

2. Automatically make models "mode-adaptive". This is a powerful technique that basically consists on creating K separate networks (by replicating or batching their variables) and using a soft-attention mechanism to create one combined network at weight level that then you use. That paper uses it with FCs, so it's easy, but consider how tricky that would become if it used RNNs, convnets or other more complex layers. By having a way to intercept variable access (and creation too in this case) we can simply add a K batch dimension and automatically apply soft-attention when reading. For all the network using it knows, it always had the shape it expected.

3. You can similarly implement differentiable plasticity in a transparent way if you store the extra information in the creator scope and apply their moving averages when trying to read it. Again, this would help reusing models and scaling to more complex ones easily.

Perhaps a better way to support this in this new model is to have a separate variable_reader_scope API? Combined with the proposed one it should allow doing both these examples.

instead of relying on a scope, can we ask the variable factory explicitly?

Why would you? tf.Variable is the only public API symbol for constructing variables, and it'll eventually bottom out to one or more calls to the base variable class ResourceVariable (RefVariable exists only in tf.compat.v1).

If you want to control what variables a piece of code can construct you can add a creator to the stack around that piece of code, but you should not have access to the lower bits because that would allow you to break the behavior of things like distribution strategies or make_template.

Can you clarify what are you trying to do?

IMHO, this is a chance to eliminate magic as much as possible. For example if DistributionStrategy needs to control how variables are created, they can provide it. Something like (speculating):

ds = MirroredStrategy(...)
model = tf.keras.Sequential(variable_factory=ds.variable_creator())
model.add(Dense....)

Hi @alextp.
Could you please show an example about how to create a PartitionedVariable via API tf.Variable(*args, **kwargs) ? My question is whether user should pass an indicator to show what kinds of concrete Variable to create ? Does it mean the parameters *args and **kwargs are exposed to users without any limit?