How to determine if a pandas column type can be reduced from int64 to int32 or from float64 to float32? [duplicate]

131
April 03, 2022, at 10:20 AM
This question already has an answer here:
How to downcast numeric columns in Pandas? (1 answer)
Closed last month.

I have a dataframe which is huge(8 gb). I am trying to find if i will loose any information if i downsize the columns from int64 to int32 or from float64 to float32.

How to find out if the information in dataframe will be lost or not in conversion?

Answer 1

I have a dataframe which is huge(8 gb). I am trying to find if i will loose any information if i downsize the columns from int64 to int32 ...

The simplest way to cast integers to a smaller type and make sure that you are not losing information is to use

df['col'] = pd.to_numeric(df['col'], downcast='integer')

This will both do the conversion, and check that the conversion didn't lose data. You'll need to do that for each integer column in your dataframe.

... or from float64 to float32.

Casting a number to a smaller floating point number always loses some information, unless you are dealing with an exact binary fraction. In practice, you can use 32-bit float if you need around 7 digits or fewer of precision.

Answer 2

You can easily check integers, e.g.:

df.abs().ge(2 ** 32).to_numpy().any()

If it's False, you're safe. Otherwise you might have to check column-wise and handle accordingly.

Like @anon01 mentioned, when it comes to floats it really depends on your use case.

Rent Charter Buses Company
READ ALSO
Folium popup does not appear python

Folium popup does not appear python

I have a geodataframe with Multistring GeometryI wanted to plot an interactive map, on which when I click on each line, the name and id would appear! I can see that mouse form would change but nothing will appear

88
How can I refactor my code to return a collection of dictionaries?

How can I refactor my code to return a collection of dictionaries?

Basically I am able to return a single dictionary value upon a iteration here in this exampleI have tried printing it as well and others

112
What could be causing install errors with pip, conda and setup.py install

What could be causing install errors with pip, conda and setup.py install

Whenever I try to install a package with pip or pip3 I get an error that starts with:

104
Snapcraft a conda environment

Snapcraft a conda environment

I have a project that uses a conda environment for dependencies managementFor now, the user can activate the environment and use it (it's exposed to the command line through an entry_point)

159