1

I did a paste special of date column as values in excel. I want to convert the resulted long values back to dates in spark (using scala api).

Example: converting 41088.96389 to date in excel results in 6/29/16 23:08 same when did through cast(DataTypes.TimestampType) in spark, it gives 01 Jan 1970 11:24:48 GMT

Any links to how excel handles long type when converting to date will be appreciated.

Omley
  • 426
  • 6
  • 17

1 Answers1

2

Excel's number is the amount of days elapsed since 1st of January, 1900. DateTime or Date usually accepts a UNIX epoch timestamp, which is the number of milliseconds (or sometimes seconds, depends on the implementation) elapsed since the 1st of January, 1970.

Converting between the two is not so easy, since you have to count for leap years, and maybe even time zones.

You can find multiple implementations for this in different languages, maybe you can port one of those:

https://gist.github.com/peter216/6361201

https://gist.github.com/christopherscott/2782634

Or maybe just try to normalize the Excel output if you have access to the sheet: https://stackoverflow.com/a/11140924/1395437

Community
  • 1
  • 1
Daniel Zolnai
  • 16,487
  • 7
  • 59
  • 71