Extract the extra fields in logging call in log formatter

100
December 05, 2019, at 4:40 PM

so I can add additional fields to my log message like so

logging.info("My log Message", extra={"someContext":1, "someOtherContext":2})

which is nice

but unclear how to extract all the extra fields in my log formatter

def format(self, record):
  record_dict = record.__dict__.copy()
  print(record_dict)

in the above i can see all my extra fields in the output dict but they are flattened in to a dict with load of other junk i dont want

{'name': 'root', 'msg': 'My log Message', 'args': (), 'levelname': 'INFO', 'levelno': 20, 'pathname': '.\\handler.py', 'filename': 'handler.py', 'module': 'handler', 'exc_info': None, 'exc_text': None, 'stack_info': None, 'lineno': 27, 'funcName': 'getPlan', 'created': 1575461352.0664868, 'msecs': 66.48683547973633, 'relativeCreated': 1253.0038356781006, 'thread': 15096, 'threadName': 'MainThread', 'processName': 'MainProcess', 'process': 23740, 'someContext': 1, 'someOtherContext':2}

is there any way of getting all my extra keys without having to know them all upfront,

i am writing a json formatter and want to create a dict a la

justMyExtra = ?????
to_log = {
"message" record_dict["message"], 
**justMyExtra
}
Answer 1

If you read the source code of the logging.Logger.makeRecord method, which returns a LogRecord object with the given logging information, you'll find that it merges the extra dict with the __dict__ attribute of the returing LogRecord object, so you cannot retrieve the original extra dict afterwards in the formatter.

Instead, you can patch the logging.Logger.makeRecord method with a wrapper function that stores the given extra dict as the _extra attribute of the returning LogRecord object:

def make_record_with_extra(self, name, level, fn, lno, msg, args, exc_info, func=None, extra=None, sinfo=None):
    record = original_makeRecord(self, name, level, fn, lno, msg, args, exc_info, func, extra, sinfo)
    record._extra = extra
    return record
original_makeRecord = logging.Logger.makeRecord
logging.Logger.makeRecord = make_record_with_extra

so that:

class myFormatter(logging.Formatter):
    def format(self, record):
        print('Got extra:', record._extra) # or do whatever you want with _extra
        return super().format(record)
logger = logging.getLogger(__name__)
handler = logging.StreamHandler()
handler.setFormatter(myFormatter('%(name)s - %(levelname)s - %(message)s - %(foo)s'))
logger.addHandler(handler)
logger.warning('test', extra={'foo': 'bar'})

outputs:

Got extra: {'foo': 'bar'}
__main__ - WARNING - test - bar

Demo: https://repl.it/@blhsing/WorthyTotalLivedistro

Answer 2

Two approaches come to mind:

  1. Dump all of the extra fields into a dictionary within your main dictionary. Call the key "additionalContext" and get all the extra entries.
  2. Create a copy of the original dictionary and delete all of your known keys: 'name','msg','args', etc. until you only have justYourExtra
READ ALSO
Opening file sometimes results in json.decoder.JSONDecodeError

Opening file sometimes results in json.decoder.JSONDecodeError

I have a file that is updated with new content and saved every hour:

100
Cropping an image based on its angles with PyQt

Cropping an image based on its angles with PyQt

I want to make a python script where the program shows an image that contains a rectangle or a square and then the End-user should specify four angles of this shape by drag some pointers on its anglesThese pointers will be presented by the program

215
How to randomly select items from a numpy matrix (in a vectorized way)?

How to randomly select items from a numpy matrix (in a vectorized way)?

I have numpy matrix of fares by products, something like

115