Unexpected performance degradation in unpickled set in python

43
October 13, 2021, at 3:50 PM

Set operations of a separately pickled objects are significantly slower than set operations of normal objects, or jointly pickled object.

import random
import timeit
import pickle
data = set(tuple([n]*100) for n in range(2000))
subset = set(random.sample(data, k=1000))
data2 = pickle.loads(pickle.dumps(data))
subset2 = pickle.loads(pickle.dumps(subset))
data3, subset3 = pickle.loads(pickle.dumps([data, subset]))
print(timeit.timeit(lambda: data & subset, number=1000))
print(timeit.timeit(lambda: data2 & subset2, number=1000))
print(timeit.timeit(lambda: data3 & subset3, number=1000))

stdout:

0.037282757000184574
1.339586072999964
0.038100944000007075

Almost 3000% difference. What could be the reason?

Answer 1

The culprit was that separately pickled objects have different memory address, while in other 2 cases the objects have same memory address.

Credits to @hexnewbie from LiberaChat #python channel.

len({id(x) for x in data} & {id(x) for x in subset})  # 1000
len({id(x) for x in data2} & {id(x) for x in subset2})  # 0
len({id(x) for x in data3} & {id(x) for x in subset3})  # 1000
READ ALSO
How to set class name to child element of a wrapper element?

How to set class name to child element of a wrapper element?

Above is the block where an image component is being called

12
mysql Trigger update table

mysql Trigger update table

Trigger Triggername afhentet Tabel scan After Insert

58
How to do a mirror scroll effect on half of the page

How to do a mirror scroll effect on half of the page

I'm trying to recreate this scrolling mirror effect: http://mirrorschoolsdesigning

50
How to pass state from component to file app.js in React

How to pass state from component to file app.js in React

I have to pass my state from Authenticationjs to App

41