How to capture log level Error and Warning both in Pyspark?

34
October 22, 2019, at 10:00 PM

I have tried to set log level as error or warn, but curious how to capture both ?

I have tried in pyspark

sc.setLogLevel("ERROR")
sc.setLogLevel("WARN")

In Debug I am getting lot of extra information and so is with info or default, would appreciate if anyone can suggest a better solution for getting errors and warning only.

Answer 1

If you set it to WARN it will capture the ERROR logs as well.

It's a hierarchical log level structure.

  • ERROR level is above WARN in the hierarchy, so WARN logs will be ignored if LogLevel set to ERROR. But since WARN is below ERROR in the hierarchy, and you set the LogLevel to WARN, it will show you ERROR logs as well.
READ ALSO
Inconsistent plotting behaviour with radial bar chart in matplotlib

Inconsistent plotting behaviour with radial bar chart in matplotlib

I'm trying to create a circular bar chart in Python/matplotlibI have the code working fine on my desktop (using Spyder), but the output of the program is completely different on different computers even in the same version of Python (3

49
Python built in function - Input

Python built in function - Input

I run this code as apy file which will bring up a command prompt from the built in Python function input

50
Why python has different types of bytes

Why python has different types of bytes

I have two variables, one is b_d, the other is b_test_d

44