Parsing

Intro

Today, I want to discuss a topic that, when I discovered it, significantly improved my workflow. It is the standard library module unicodedata. This module proves exceptionally valuable when parsing data. Let’s delve into why.

Normalization

The unicodedata.normalize(form, unistr, /) function serves to convert Unicode strings into their standardized form.

There are four possible values for form, two for composing and two for decomposing:

• NFC: Normalization Form Composed. In this form, characters are composed into their canonical equivalents.
• NFKC: Normalization Form Compatibility Composed. Similar to NFC, but it also applies compatibility equivalents.
• NFD: Normalization Form Decomposed. In this form, characters are decomposed into their canonical equivalents.
• NFKD: Normalization Form Compatibility Decomposed. Similar to NFD, but it also applies compatibility transformations.

Firstly, let’s check some stuff in a Python shell: