How to properly set steps_per_epoch and validation_steps in Keras?

Question

I've trained several models in Keras. I have 39, 592 samples in my training set, and 9, 899 in my validation set. I used a batch size of 2.

As I was examining my code, it occurred to me that my generators may have been missing some batches of data.

This is the code for my generator:

train_datagen = ImageDataGenerator(
            rescale=1. / 255,
            shear_range=0.2,
            zoom_range=0.2,
            horizontal_flip=True)

val_datagen = ImageDataGenerator(rescale=1. / 255)

train_generator = train_datagen.flow_from_directory(
            train_dir,
            target_size=(224, 224)
            batch_size=batch_size,
            class_mode='categorical')

validation_generator = val_datagen.flow_from_directory(
            val_dir,
            target_size=(224, 224),
            batch_size=batch_size,
            class_mode='categorical')

I searched around to see how my generators behave, and found this answer: what if steps_per_epoch does not fit into numbers of samples?

I calculated my steps_per_epoch and validation_steps this way:

steps_per_epoch = int(number_of_train_samples / batch_size)
val_steps = int(number_of_val_samples / batch_size)

Using the code in this link with my own batch size and number of samples, I got these results: "missing the last batch" for train_generator and "weird behavior" for val_generator.

I'm afraid that I have to retrain my models again. What values should I choose for steps_per_epoch and validation_steps? Is there a way to use exact values for these variables(Other than setting batch_size to 1 or removing some of the samples)? I have several other models with different number of samples, and I think they've all been missing some batches. Any help would be much appreciated.

Two related question:

1- Regarding the models I already trained, are they reliable and properly trained?

2- What would happen if I set these variables using following values:

steps_per_epoch = np.ceil(number_of_train_samples / batch_size)
val_steps = np.ceil(number_of_val_samples / batch_size)

will my model see some of the images more than once in each epoch during training and validation? or Is this the solution to my question?!

In the link, if you look at the line `remainingSamples = total_samples % batch_size #confirm that this is greater than 0` - his answer depends on the number of samples not being divisible by the batch-size, which yours is for the train_set (39592 % 2 == 0). So from what you have posted, I see no reason why batches should be missing, or individual data-points not being able to fill a batch being missing. For your validation-set, you may have a single image not being used. — Jeppe, May 28 '19 at 13:31

score 1 · Answer 1 · answered Jun 24 '19 at 14:43

Since Keras data generator is meant to loop infinitely, steps_per_epoch indicates how many times you will fetch a new batch from generator during single epoch. Therefore, if you simply take steps_per_epoch = int(number_of_train_samples / batch_size), your last batch would have less than batch_size items and would be discarded. However, in your case, it's not a big deal to lose 1 image per training epoch. The same is for validation step. To sum up: your models are trained [almost :) ] correctly, because the quantity of lost elements is minor.

Corresponding to implementation ImageDataGenerator https://keras.io/preprocessing/image/#imagedatagenerator-class if your number of steps would be larger than expected, after reaching the maximum number of samples you will receive new batches from the beginning, because your data is looped over. In your case, if steps_per_epoch = np.ceil(number_of_train_samples / batch_size) you would receive one additional batch per each epoch which would contains repeated image.

score 0 · Answer 2 · answered Jun 24 '21 at 11:37

In addition to Greeser's answer, To avoid losing some training samples, you could calculate your steps with this function:

def cal_steps(num_images, batch_size):
   # calculates steps for generator
   steps = num_images // batch_size

   # adds 1 to the generator steps if the steps multiplied by
   # the batch size is less than the total training samples
   return steps + 1 if (steps * batch_size) < num_images else steps

How to properly set steps_per_epoch and validation_steps in Keras?

2 Answers2