Embedding Layer trainable bug?

[Fricker95]

In the class Embedding(Layer) keras/src/layers/core/embedding.py

    def build(self, input_shape=None):
        if self.built:
            return
        if self.quantization_mode is not None:
            self.quantized_build(input_shape, mode=self.quantization_mode)
        if self.quantization_mode != "int8":
            self._embeddings = self.add_weight(
                shape=(self.input_dim, self.output_dim),
                initializer=self.embeddings_initializer,
                name="embeddings",
                regularizer=self.embeddings_regularizer,
                constraint=self.embeddings_constraint,
                trainable=True,
            )
        self.built = True
        if self.lora_rank:
            self.enable_lora(self.lora_rank)

Here the embeddings weight is built using trainable=True, this doesn't reflect the keras.layers.Layer parent class attribute _trainable. Therefore if we build the Embedding class using trainable=False as a parameter the underlying self._embeddings weight will still be trainable.

Is this set to true for a specific reason or is it simply a bug?
Thought I would point this out.

[JyotinderSingh]

The existing layer behavior works as expected.

Note that whether a layer is affected by backprop or not is controlled by the _trainable argument at the base Layer level.

1. When we instantiate the Embedding layer and pass a value for trainable (either True or False), this value is captured by the **kwargs in the Embedding's __init__ method.
2. Inside the Embedding's __init__ method, the line super().__init__(**kwargs) calls the __init__ method of its parent class, which is Layer. This passes along the trainable we provide.
   

   keras/keras/src/layers/core/embedding.py

   Line 100 in 37eacb0

   super().__init__(**kwargs)

3. The Layer's __init__ method then retrieves the trainable value from the kwargs (or uses the default True if it's not provided) and sets the self._trainable attribute accordingly.
   

   keras/keras/src/layers/layer.py

   Line 303 in 37eacb0

   self._trainable = trainable

This is then made available for reading and mutation by this property decorator. You can also see how the setter works (and dynamically controls the training value in case of callbacks)

keras/keras/src/layers/layer.py

Lines 586 to 609 in 37eacb0

	@property
	def trainable(self):
	"""Settable boolean, whether this layer should be trainable or not."""
	return self._trainable
	@trainable.setter
	def trainable(self, value):
	"""Sets trainable attribute for the layer and its sublayers.
	When this value is changed during training (e.g. with a
	`Callback`) you need to call the parent
	`Model.make_train_function` with `force=True` in order to
	recompile the training graph.
	Args:
	value: Boolean with the desired state for the layer's trainable
	attribute.
	"""
	value = bool(value)
	self._trainable = value
	for v in self._trainable_variables:
	v.trainable = value
	for layer in self._layers:
	layer.trainable = value

The following API is used to determine whether a layer is affected by backprop:

keras/keras/src/layers/layer.py

Lines 638 to 646 in 37eacb0

	@property
	def trainable_variables(self):
	"""List of all trainable layer state.
	This is equivalent to `layer.trainable_weights`.
	"""
	if not self.trainable:
	return []
	return [v for v in self.variables if v.trainable]

You can see in the trainer code here that we retrieve the weights we want to optimize using the above API:

keras/keras/src/backend/tensorflow/trainer.py

Lines 75 to 80 in 37eacb0

	if self.trainable_weights:
	trainable_weights = self.trainable_weights
	gradients = tape.gradient(loss, trainable_weights)
	# Update weights
	self.optimizer.apply_gradients(zip(gradients, trainable_weights))