Error in parsing CSV file

[PradipKumarDas]

Hello!

I noticed that Orange 3.26.0 is failing to import content from this attached CSV file (link below) and shows error message as "mpi_roof_2009b.csv was not loaded due to a text encoding error. The file might be saved in an unknown or invalid encoding, or it might be a binary file.". It is also strange because there is no issues when the same file gets loaded from Pandas running in Jupyter Notebook. Hence, I thought to check in this discussion if anyone knows any reason behind this behaviour. By the way, this CSV is a file that contains weather data at Jena and is from Max Planck Institute for Biogeochemistry (https://www.bgc-jena.mpg.de/index.php/Main/HomePage). No modifications were made to the original CSV file.

By the way, I have raised an issue on this with ID #5436

Regards,
Pradip Kumar Das
Bengaluru, Karnataka, India

mpi_roof_2009a.csv

[ajdapretnar]

μ and subscript (squared) don't get recognized properly. It works in the File widget, though. Column names do get garbled when loading with pandas (v1.2.3, datareader 0.9.0), but the import works. Is there something special about CSV File Reader, since it is based on pandas anyway? @ales-erjavec

[PradipKumarDas]

Thanks @ajdapretnar.
I had tried just with CSV File Reader, but with your hint I tested the same CSV with File Reader, too, and found that the data gets loaded without any issues. So, it is confirmed that there are some issue (as @ajdapretnar rightly mentioned, the issue is in parsing the few column labels) only with CSV File Reader. Can it be fixed around the same way as File Reader and Pandas.Read_CSV method handle loading this CSV?

[ales-erjavec]

@PradipKumarDas In the csv import options change the encoding to "Western Europe (ISO Latin 1)"

[ajdapretnar]

While changing the encoding does solve the issue, is it not strange that the file is imported normally with the File widget and pandas? Don't they use 'utf-8' by default as well?

[ales-erjavec]

Difference is

pd.read_csv("./mpi_roof_2009a.csv")
# or 
pd.read_csv(open("./mpi_roof_2009a.csv", encoding="utf-8", errors="replace"))

vs

pd.read_csv(open("./mpi_roof_2009a.csv", encoding="utf-8"))