How to use incremental PCA on dask dataframe?

168
November 27, 2019, at 09:40 AM

I am using a dask dataframe which can not be loaded directly into the memory because of the size of it. I want to perform dimentionality reduction of top of using incremental PCA.

My dataframe is sparse in nature, so the question is can I perform it and if yes then how to do so.

image_features_df.head(3)
   feat1   feat2   feat3  ...  feat25087    feat25088   fid   selling_price
0   0.0    0.0     0.0    ...   0.0          0.0         2       269.00
4   0.3    0.1     0.0    ...   0.0          0.8         26      1720.00    
6   0.8    0.0     0.0    ...   0.0          0.1         50      18145.25

The above is a view of my dataframe. I want the output to have 95% cumulative varience. How to do so?

My dataframe has 100,000 rows and 25088 columns so please tell a solution which is memory efficient.

Rent Charter Buses Company
READ ALSO
Merging pandas dataframes

Merging pandas dataframes

I am fairly new to pandas

157
TensorFlow 2.0+: Saved trained model with custom python object (overright error)

TensorFlow 2.0+: Saved trained model with custom python object (overright error)

I trained a custom keras modelIt ran fine and finished

119
GoogleBidManager RunQuery on Saved Query Error

GoogleBidManager RunQuery on Saved Query Error

I'm creating a python function that will run on Google Cloud Functions in order to pull data from DBM (Google Bid Manager / DisplayVideo 360)

132
I'am building online shop with python django

I'am building online shop with python django

I'm working with Django to build online store and i built a cart system and i make a mistake i delete all product from data but when i deleted i found old product still on cart page and cart didn't work and give me this error

111