Problem Description
For datetime columns, the metadata specification allows me to input a datetime_format string that specifies exactly how my datetimes are to be formatted. For example, '21 Jul 2026' vs '21-07-2026'.
If the data is represented as a dtype object (containing string values), then SDV can explicitly validate that the provided format matches the data.
However if for some reason the data is represented as a datetime64, then there is no format really associated with those values. So in this case, SDV does not explicitly validate whether the data matches any explicitly provided format string. But as a user, I don't know that this validation is skipped.
Expected behavior
The metadata.validate_data functions should raise a warning if:
- The metadata has a specified
datetime_format string for a particular column AND
- The data is represented as a dtype
object (or any other dtype where the format cannot be checked)
The warning should specifically indicate that the datetime format cannot be verified and will be skipped.
Warning: The datetime format for column 'checkin_date' (table 'guests') could not be verified because the data is represented as dtype 'datetime64[ns]'.
Please omit the datetime format string from the metadata or cast the data to strings with the right format.
Problem Description
For datetime columns, the metadata specification allows me to input a
datetime_formatstring that specifies exactly how my datetimes are to be formatted. For example,'21 Jul 2026'vs'21-07-2026'.If the data is represented as a dtype
object(containing string values), then SDV can explicitly validate that the provided format matches the data.However if for some reason the data is represented as a
datetime64, then there is no format really associated with those values. So in this case, SDV does not explicitly validate whether the data matches any explicitly provided format string. But as a user, I don't know that this validation is skipped.Expected behavior
The
metadata.validate_datafunctions should raise a warning if:datetime_formatstring for a particular column ANDobject(or any other dtype where the format cannot be checked)The warning should specifically indicate that the datetime format cannot be verified and will be skipped.