Hands-on Entity Resolution with Python-v3.png

dockerfile.tar.gz

conda-main

A typical business stores data across multiple systems, including ERPs for operations, a CRM for marketing, files, notebooks, and BI apps for other purposes. Records of the same customer (entity) exist in multiple places, likely not in sync across nor unique within sources. This inconsistent situation generates an opportunity for us to drive business value by cross-referencing and deduplicating records with entity resolution.

This course covers business acumen and hands-on coding. It starts with several business cases and a quick introduction to entity resolution in Python. Then, it explores semantic-preserving preprocessing, similarity feature engineering, graph clustering, weak supervision, confident learning, and integration.

As a developer, you’ll increase your company’s business value by developing and deploying entity resolution pipelines. As a decision-maker, you’ll know which solution best suits your business cases and how to negotiate the best value for your money.

An Introduction to Entity Resolution in Python

The Damerau-Levenshtein distance is $d=1$.

What is the edit distance $d$ between strings "1996-01-13" and "1996-01-31"?

What is the edit distance <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>d</mi></mrow><annotation encoding="application/x-tex">d</annotation></semantics></math>d between strings “1996-01-13” and “1996-01-31”?

A similarity $s(x,y)=1$ means $x$ and $y$ must be the same.

The cosine similarity $s(x,y)$ is the mathematical dot product of two numeric vectors divided by the Euclidean norm of both vectors.

Let $d$ be a mathematical distance, then $s=T_a(d)=\frac{1}{(1+d)^a}$ for any nonzero number $a$ is a similarity function with $d(x_1,y_1)\leq d(x_2,y_2)$ if and only if $s(x_1,y_1)\geq s(x_2,y_2)$.

The cosine similarity between scalars (one-dimensional vectors) must be always equal to one due to the normalization.

Which statement about similarity functions is true?

Which statement about similarity functions is true?

The term "hello" will have a higher TF-IDF score than the term "python" across the documents.

The cosine similarity between the vectorized texts "python world" and "super python world" is less than 1.

The implementation will throw an error because it requires at least four texts to fit a model.

The cosine similarity between the vectorized texts "pythonic world" and "everything pythonic" is zero.

We fit a TF-IDF vectorizer using `TfidfVectorizer` from the `sklearn` package on texts ('hello world,' 'hello python,' and 'everything python') with the default configuration parameters. Which statement is true?

We fit a TF-IDF vectorizer using <code>TfidfVectorizer</code> from the <code>sklearn</code> package on texts (‘hello world,’ ‘hello python,’ and ‘everything python’) with the default configuration parameters. Which statement is true?

The Jaro similarity function of two texts is always smaller than the Jaro-Winkler similarity for the same pair of texts.

The Damerau-Levenshtein similarity between two texts cannot be smaller than the Levenshtein similarity of the same pair of texts.

Computing scores with Jaro and Jaro-Winkler in the same entity resolution pipeline is always a waste of computing resources.

Edit similarities cannot be applied to numeric attributes.

Which statement about edit-based similarity functions is true?

Which statement about edit-based similarity functions is true?

We can use pretrained LLMs but not pretrained TF-IDF models for embeddings.

TF-IDF embeds the whole text, whereas LLMs can only embed single words.

LLMs must be trained on massive datasets. We cannot rely on them for a specific task without access to a massive training dataset.

LLMs learn the meanings of words in the context of texts, whereas TF-IDF embeddings only work with lexical representations of tokens.

What is a valid argument when comparing LLM and TF-IDF based embeddings?

What is a valid argument when comparing LLM and TF-IDF based embeddings?

Evaluate your learnings about similarity feature engineering.

Introduction to Entity Resolution and Applications

A Quickstart Guide Using the RecordLinkage Package

Preprocessing

Indexing

Feature Engineering

Pairwise Matching

Clustering

Integration

Entity Resolution Fundamentals

Matching Products Across Two Online Shops

Conclusion

Appendix

Quiz on Similarity Features