0

I have created a custom loss function called

def customLoss(true, pred) //do_stuff //print(variables) return loss

Now I'm calling compile as model.compile(optimizer='Adamax', loss = customLoss)

EDIT: I tried tf.Print and this is my result.

    def customLoss(params):

        def lossFunc(true, pred):
            true = tf.Print(true, [true.shape],'loss-func') #obviously this won't work because the tensors aren't the same shape; however, this is what I want to do.
            #stuff
            return loss
        return lossFunc

    model = Model(inputs=[inputs], outputs=[outputs])
    parallel_model = multi_gpu_model(model, gpus=8)
    parallel_model.compile(opimizer='Adam', loss = customLoss(params), metrics = [mean_iou)
    history = parallel_model.fit(X_train, Y_train, validation_split=0.25, batch_size = 32, verbose=1)

and the output is 

Epoch 1/10
1159/1159 [==============================] - 75s 65ms/step - loss: 0.1051 - mean_iou: 0.4942 - val_loss: 0.0924 - val_mean_iou: 0.6933
Epoch 2/10
1152/1159 [============================>.] - ETA: 0s - loss: 0.0408 - mean_iou: 0.7608

The print statements still aren't printing. Am I missing something - are my inputs into tf.Print not proper?

Jonathan
  • 1,876
  • 2
  • 20
  • 56
  • You need to call the print statement through Keras' framework. Keras uses TensorFlow for its backend so maybe try `tf.Print()` as mentioned here: https://stackoverflow.com/questions/49819438/keras-print-inside-loss-function-does-not-work – Dascienz Aug 23 '18 at 20:17

1 Answers1

3

It's not because Keras dumps buffers or does magic, it simply doesn't call them! The loss function is called once to construct the computation graph and then the symbolic tensor that represents the loss value is returned. Tensorflow uses that to compute the loss, gradients etc.

You might instead be interested tf.Print that is null operation with a side effect that prints the arguments passed. Since tf.Print is part of the computation graph it will be run when training as well. From the documentation:

Prints a list of tensors. This is an identity op (behaves like tf.identity) with the side effect of printing data when evaluating.

nuric
  • 11,027
  • 3
  • 27
  • 42
  • Because it is still not part of the computation graph, you need something like `ptensor = tf.Print(ptensor, [ptensor]...)` and chain ptensor further down the computation. The print acts like identity. – nuric Aug 23 '18 at 20:59
  • I've added an exact line of code to the loss function, but it still won't work. – Jonathan Aug 23 '18 at 21:15
  • You are passing `weightedLoss` instead of `customLoss`? Maybe that's problem, without the full code it would be difficult to debug. – nuric Aug 23 '18 at 21:35
  • oops, sorry about that. it's fixed. All I want to be able to do is print things like shape of the tensors in the loss function. But I don't know how I'd be able to splice the computation graph to also take in shape. – Jonathan Aug 23 '18 at 21:47
  • Oh, if the shape is determined at compile time you can just `print(tensor)` and it will show it's known shape at compile time. If any dimension is not known it will show `None`, ex. `(None, 64)` where the first dimension is batch size (not known at compile time) and 64 feature size. – nuric Aug 23 '18 at 22:09