We are ingesting parquet files from a source and need to add a new column to the Glue table we have set up against this data. The new column is a "value" column that is sometimes a double and is sometimes a bigint (ex of different values for different rows. 4134, 4134.0, 12569901, 1431509.4354199, etc.).
If we set up the table to use "bigint" it complains about the double values of the column in some partitions. If we set up the table to use "double" it complains about the "bigint" values.
Is there anyway to just tell it to use "double" and handle "bigint" values as doubles?
I've tried some options here How to create AWS Glue table where partitions have different columns? ('HIVE_PARTITION_SCHEMA_MISMATCH'), but I don't think these solve the issue because we regularly expect both types of values to be coming in in the parquet files.
Any suggestions would be appreciated.
