1

I am migrating existing applciation which using elasticsearch-spark 7.6.0 version to latest i.e elasticsearch-spark-30_2.12:7.15.0. I am loading ES data with date mapping as below

"my_partition_key": {
                    "format": "yyyy-MM-dd",
                    "type": "date"
                }

However I am getting error

*org.elasticsearch.hadoop.rest.EsHadoopRemoteException: mapper_parsing_exception: failed to parse field [my_partition_key] of type [date] in document with id 'asdaszzdfdf-Ptn'. Preview of field's value: '1634601600000';org.elasticsearch.hadoop.rest.EsHadoopRemoteException: illegal_argument_exception: failed to parse date field [1634601600000] with format [yyyy-MM-dd];org.elasticsearch.hadoop.rest.EsHadoopRemoteException: date_time_parse_exception: Text '1634601600000' could not be parsed at index 0
{"index":{}}*

I am unable to figure out though value is in long and printing data set with dataset.show() displays valid date, why unable to write to ES? Is any new configuration I need to pass for date fields with latest ES? any help appreciated.

Oli
  • 9,766
  • 5
  • 25
  • 46
Naresh G
  • 83
  • 5
  • It would seem that your input data is not in the format you expect, but stored as timestamps. – Oli Nov 03 '21 at 06:29
  • Actually I am reading a partition where partition key is a date. Looks like spark is automatically treating as date based on schema. Based on my research looks like I need to explicitly convert into String to avoid storing as ephoch milliseconds. – Naresh G Nov 08 '21 at 21:07

0 Answers0