Hands-on Entity Resolution with Python-v3.png

dockerfile.tar.gz

conda-main

A typical business stores data across multiple systems, including ERPs for operations, a CRM for marketing, files, notebooks, and BI apps for other purposes. Records of the same customer (entity) exist in multiple places, likely not in sync across nor unique within sources. This inconsistent situation generates an opportunity for us to drive business value by cross-referencing and deduplicating records with entity resolution.

This course covers business acumen and hands-on coding. It starts with several business cases and a quick introduction to entity resolution in Python. Then, it explores semantic-preserving preprocessing, similarity feature engineering, graph clustering, weak supervision, confident learning, and integration.

As a developer, you’ll increase your company’s business value by developing and deploying entity resolution pipelines. As a decision-maker, you’ll know which solution best suits your business cases and how to negotiate the best value for your money.

An Introduction to Entity Resolution in Python

The index can contain pairs with records having similar but not the same keys.

Computing the index costs fewer resources because it uses an efficient sort algorithm.

We can use multiple attributes as keys, whereas, in SB, we can use only one.

RecordLinkage's SN implementation has the following advantage compared to SB on the same key:

RecordLinkage’s SN implementation has the following advantage compared to SB on the same key:

Using any combination of univariate attributes as the block key

Using any combination of univariate string attributes as the block key

Using a string attribute as the block key

SB always means equality-based indexing with:

SB always means equality-based indexing with:

A union reduces the redundancy if a pair is present in both indexes.

We can split an index into several sets to parallelize processing.

We can create a sophisticated index by combining logical rules with the AND and OR operators.

An intersection creates a less restrictive index than each in separation.

What is not a reason to apply set operations on two indexes before running the comparisons?

What is not a reason to apply set operations on two indexes before running the comparisons?

We apply SB. The size of the index is $k=24$.

We apply standard SN (not the `recordlinkage` implementation!) with window size $w=3$. The size of the index is $k=n\cdot w=30$.

We use the implementation of `recordlinkage` of SN with window size $w=3$. We end up with a full index of size $k=n*(n-1)/2=45$.

We use extended SN with window size $w=3$. We end up with a full index of size $k=n*(n-1)/2=45$.

We index a dataset with $n=10$ records using the $x$ attribute as the block or sort key. $b_1=7$ records have $x=val_1$, and the other $b_2=3$ have $x=val_2$. Which statement about the size of an index is wrong?

We index a dataset with <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>n</mi><mo>=</mo><mn>10</mn></mrow><annotation encoding="application/x-tex">n=10</annotation></semantics></math>n=10 records using the <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>x</mi></mrow><annotation encoding="application/x-tex">x</annotation></semantics></math>x attribute as the block or sort key. <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>b</mi><mn>1</mn></msub><mo>=</mo><mn>7</mn></mrow><annotation encoding="application/x-tex">b_1=7</annotation></semantics></math>b1​=7 records have <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>x</mi><mo>=</mo><mi>v</mi><mi>a</mi><msub><mi>l</mi><mn>1</mn></msub></mrow><annotation encoding="application/x-tex">x=val_1</annotation></semantics></math>x=val1​, and the other <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>b</mi><mn>2</mn></msub><mo>=</mo><mn>3</mn></mrow><annotation encoding="application/x-tex">b_2=3</annotation></semantics></math>b2​=3 have <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>x</mi><mo>=</mo><mi>v</mi><mi>a</mi><msub><mi>l</mi><mn>2</mn></msub></mrow><annotation encoding="application/x-tex">x=val_2</annotation></semantics></math>x=val2​. Which statement about the size of an index is wrong?

Indexing is a binary classification of pairs into match and no-match with a cheap algorithm.

Indexing is relevant only in budget-aware resolution tasks.

The name "indexing" is borrowed from the search literature, where we add match keys to every record used for fast retrieval of results by comparing those keys with a search query.

Indexing is about segmenting records into disjoint blocks with the same match keys within and different match keys across blocks.

Which statement about indexing is wrong?

The Lance format is designed to be fast in vector search.

The LanceDB database uses the Lance format to store data. We can create an index for ANN search, which typically is magnitudes faster than KNN. The quality of ANN and KNN remains the same.

The index blocking is conducted by looping over all examples, pairing them with each of its KNN, and including these into the index. Our aim is high recall at a relatively small K. Precision is irrelevant.

The Lance format works with any vector embedding model—for example, we can use fastText for lexical search, a pretrained LLM for semantic search, or even classic TF-IDF vectors.

We can use the Lance columnar data format and embeddings from a pretrained large language model for index blocking. Which statement is not true?

We can use the Lance columnar data format and embeddings from a pretrained large language model for index blocking. Which statement is not true?

Evaluate your learnings about indexing for entity resolution.

Introduction to Entity Resolution and Applications

A Quickstart Guide Using the RecordLinkage Package

Preprocessing

Indexing

Feature Engineering

Pairwise Matching

Clustering

Integration

Entity Resolution Fundamentals

Matching Products Across Two Online Shops

Conclusion

Appendix

Quiz on Indexing