PySpark: how to groupby, resample and forward-fill null values?

219
August 16, 2019, at 7:20 PM

Considering the below dataset in Spark, I would like to resample the dates with a specific frequency (eg. 5 minutes).

START_DATE = dt.datetime(2019,8,15,20,33,0)
test_df = pd.DataFrame({
    'school_id': ['remote','remote','remote','remote','onsite','onsite','onsite','onsite','remote','remote'],
    'class_id': ['green', 'green', 'red', 'red', 'green', 'green', 'green', 'green', 'red', 'green'],
    'user_id': [15,15,16,16,15,17,17,17,16,17],
    'status': [0,1,1,1,0,1,0,1,1,0],
    'start': pd.date_range(start=START_DATE, periods=10, freq='2min')
})
test_df.groupby(['school_id', 'class_id', 'user_id', 'start']).min()

However I also want the resample to happen between two specific date ranges: 2019-08-15 20:30:00 and 2019-08-15 21:00:00. So every group of school_id, class_id and user_id will have 6 entries, one every 5 min bucket between the two date ranges. The null entries generated by the resample should be populated by forward-fill.

I've used Pandas for the sample dataset, but the actual dataframe will be pulled in Spark, so the approach I'm looking for should be done in Spark as well.

I guess the approach might be similar to this one PySpark: how to resample frequencies but I'm not getting it to work in this scenario.

Thanks for your help

Rent Charter Buses Company
READ ALSO
Android Data binding create two Binding Adapters with same parameters

Android Data binding create two Binding Adapters with same parameters

I'm following MVVM Pattern + Data binding in android, I moved my whole UI logic to view (binding adapters or modify directly xml file using data binding) but I have a problem with binding adapters

146
how to set application theme in code rather than in manifest

how to set application theme in code rather than in manifest

I want my application to apply Leanback-theme on Android-TV, and AppCompat-theme on Android phones

122
Duplicate class found in modules with R8 enabled

Duplicate class found in modules with R8 enabled

I created two aar-libraries: comexample:test1:1

161
How to decrypt signal protocol message sent from node in android - java

How to decrypt signal protocol message sent from node in android - java

I'm setting up a connection between a laptop and an android phone with socketIOI wanted the messages to be encrypted with signal protocol, and so far I've been able to start a session and send a message from the phone to the laptop and decrypt it

131