Spring-batch exiting with Exit Status : COMPLETED before actual job is finished?

Question

In my Spring Batch application I have written a CustomItemWriter which internally writes item to DynamoDB using DynamoDBAsyncClient, this client returns Future object. I have a input file with millions of record. Since CustomItemWriter returns future object immediately my batch job exiting within 5 sec with status as COMPLETED, but in actual it is taking 3-4 minutes to write all item to the DB, I want that batch job finishes only after all item written to DataBase. How can i do that?

job is defined as below

    <bean id="report" class="com.solution.model.Report" scope="prototype" />
        <batch:job id="job" restartable="true">
            <batch:step id="step1">
                <batch:tasklet>
                    <batch:chunk reader="cvsFileItemReader"  processor="filterReportProcessor" writer="customItemWriter"
                        commit-interval="20">
                    </batch:chunk>
                </batch:tasklet>
            </batch:step>
        </batch:job>
<bean id="customItemWriter" class="com.solution.writer.CustomeWriter"></bean>

CustomeItemWriter is defined as below

public class CustomeWriter implements ItemWriter<Report>{
    public void write(List<? extends Report> item) throws Exception {
    List<Future<PutItemResult>> list = new LinkedList();
    AmazonDynamoDBAsyncClient client = new AmazonDynamoDBAsyncClient();
        for(Report report : item) {
            PutItemRequest req = new PutItemRequest();
            req.setTableName("MyTable");
            req.setReturnValue(ReturnValue.ALL_ODD);
            req.addItemEntry("customerId",new 
            AttributeValue(item.getCustomeId()));
            Future<PutItemResult> res = client.putItemAsync(req);
            list.add(res);
            }
    }

}

Main class contains

JobExecution execution = jobLauncher.run(job, new JobParameters());
System.out.println("Exit Status : " + execution.getStatus());

Since in ItemWriter its returning future object it doesn't waits to complete the opration. And from the main since all item is submitted for writing Batch Status is showing COMPLETED and job terminates. I want that this job should terminate only after actual write is performed in the DynamoDB. Can we have some other step well to wait on this or some Listener is available?

Get the results from the `Future` that should block the writer until it is actually completed. — M. Deinum, May 17 '17 at 09:36

pvpkiran · Accepted Answer · 2017-05-17T09:49:00.667

Here is one approach. Since ItemWriter::write doesn't return anything you can make use of listener feature.

@Component
@JobScope
public class YourWriteListener implements ItemWriteListener<WhatEverYourTypeIs> {


  @Value("#{jobExecution.executionContext}")
  private ExecutionContext executionContext;


  @Override
  public void afterWrite(final List<? extends WhatEverYourTypeIs> paramList) {
     Future future = this.executionContext.readAndValidate("FutureKey", Future.class);
     //wait till the job is done using future object
  }

  @Override
  public void beforeWrite(final List<? extends WhatEverYourTypeIs> paramList) {

  }

  @Override
  public void onWriteError(final Exception paramException, final List<? extends WhatEverYourTypeIs> paramList) {

  }
}

In your writer class, everything remains same except addind the future object to ExecutionContext.

public class YourItemWriter extends ItemWriter<WhatEverYourTypeIs> {

  @Value("#{jobExecution.executionContext}")
  private ExecutionContext executionContext;

  @Override
  protected void doWrite(final List<? extends WhatEverYourTypeIs> youritems) 

     //write to DynamoDb and get Future object
    executionContext.put("FutureKey", future);
    }

  }

}

And you can register the listener in your configuration. Here is a java code, you need to do the same in your xml

@Bean
  public Step initStep() {

    return this.stepBuilders.get("someStepName").<YourTypeX, YourTypeY>chunk(10)
        .reader(yourReader).processor(yourProcessor)
        .writer(yourWriter).listener(YourWriteListener)
        .build();
  }

Spring-batch exiting with Exit Status : COMPLETED before actual job is finished?

1 Answers1

Linked